How CrowdStrike’s Failure Highlights the Power of Shift-Left Testing

The Risks of Cutting Corners in Testing

The CrowdStrike software update failure serves as a critical lesson in the importance of early, automated, and comprehensive software testing. As my colleague, Miroslaw Zielinski, emphasized in Software Testing Insights From the CrowdStrike Incident, cutting corners to save time and money in testing can lead to disastrous outcomes. Testing is not just a box to be checked. It’s a vital part of ensuring that the software you deliver is reliable, secure, and capable of performing as expected in the real world.

While there are multiple factors behind this failure, one common challenge that I frequently hear from organizations is the relentless pressure to release frequent updates. This pressure often makes optimizing the cost and time of testing an appealing target. However, the key to avoiding incidents like the CrowdStrike failure lies in adopting a comprehensive, shift-left approach to testing.

Parasoft’s software testing, particularly static analysis, is designed to facilitate this shift-left approach. But before we delve into this solution, let’s first understand the power of shift-left testing.

The Power of Shift-Left Testing

Shift-left testing is one of the most powerful concepts in modern software development, enabling early error detection and improving overall code quality. Traditionally, testing and debugging were conducted towards the end of the development cycle—after the code was written and integrated—and often after the initial testing phases. This approach usually led to the discovery of critical bugs and issues late in the process when they were much more difficult and time-consuming to fix.

By implementing static analysis during the implementation process, engineers can address coding issues or compliance violations as they write the code. This immediate feedback allows developers to correct problems on the spot, significantly enhancing code quality from the outset. The result is a substantial reduction in defects that would otherwise be discovered later in the development life cycle, where fixing them becomes increasingly complex and costly.

For instance, when a defect is identified early through static analysis, it can be addressed quickly and efficiently. However, if the same defect is not caught until later by quality assurance testing, the resolution process becomes more complicated. More team members are involved and the steps to log, report, reproduce, and fix the issue take longer, leading to increased efforts and delays.

The stakes become even higher if the defect reaches production. Addressing a defect in the field is particularly challenging, as it directly impacts customers and may require urgent patches, recalls, or even damage control to protect the product’s reputation. The development team must act swiftly to fix the bug, while the QA team needs to retest the solution, adding more layers of complexity to the process.

Reflecting on the CrowdStrike incident, it’s clear how vital static analysis is in identifying and addressing defects early in the development process, preventing costly and damaging issues from reaching production.

Static Analysis Is More Than Just Syntax Checking

Static analysis is an advanced technique for examining source code to identify potential errors such as the use of uninitialized variables, NULL pointer dereferencing, buffer overflows, and many other coding defects, without executing the code.

For instance, Parasoft’s static analysis engine goes beyond simple syntax checks. It not only performs both control flow and data flow analysis to catch a wide range of bugs and coding issues, but it also leaps ahead of the competition with a unique application of AI and machine learning. Our AI-driven solution reviews new static analysis findings in the context of historical interactions with the codebase and previous static analysis results, predicting relevance and prioritizing new findings to help organizations adopt static analysis more effectively.

Types of Issues Static Analysis Detects

Here are just a few types of bugs or issues that static analysis can detect.

• Null pointer dereferencing. Automatically detect attempts to dereference NULL pointers.
• Memory leaks. Identify instances where memory is allocated but not properly released.
• Buffer overflows. Critical for security, detect when more data is written to a buffer than it can hold.
• Uninitialized variables. Flag variables that are used before being initialized.
• Infinite loops. Detect loops that may never terminate, causing the program to hang.
• Dead code. Find code that is never executed and can be safely removed.
• Divide by zero. Identify divisions where the denominator could be zero, leading to runtime errors.
• Syntax and type errors. Flag mistakes in the code that prevent it from compiling.
• Logic errors. Catch errors in the logic of the code that could lead to incorrect behavior.
• Security vulnerabilities. Detect the use of known vulnerable libraries and other security issues.
• Coding standard violations. Ensure adherence to coding standards, crucial for obtaining certifications in safety-critical environments.

These types of coding bugs are aggressively sought out and mitigated in safety-critical development, where lives could be lost if the system crashes or fails. Although safety concepts are not a requirement for CrowdStrike, software quality is, and they should consider adopting coding standards like MISRA, CERT, CWE, or others.

Coding standards like MISRA are developed by experts with years of experience, and Parasoft, as a contributing member of the MISRA C and C++ 2023 coding standards, offers a robust static analysis solution built on this expertise.

Despite these clear benefits, it’s surprising how many development teams still do not use static analysis. Adopting static analysis is essential in cutting cost in software testing, while increasing code quality.

By incorporating static analysis into their continuous integration/continuous deployment (CI/CD) pipeline, CrowdStrike could have automatically identified memory access issues that led to their software update failing.

Integrating Static Analysis Into the CI/CD Pipeline

In a modern CI/CD environment, developers write and commit code, triggering an automated build process. When a build successfully completes, automated testing, including static analysis, runs. Parasoft C and C++ testing solutions like C/C++test easily integrate into CI/CD pipelines, working seamlessly with tools like Jenkins, GitLab, Bamboo, VS Code, Eclipse, and many others.

This integration ensures that code quality is continuously monitored and improved with developers receiving immediate feedback on any issues that static analysis identifies. By automating these quality checks, organizations can:

• Prevent technical debt from accumulating.
• Reduce testing costs by finding issues earlier in development.
• Achieve high standards of code quality throughout the development process.

Lessons From the CrowdStrike Incident

The CrowdStrike incident offers valuable lessons for development teams.

1. Resist the temptation to cut corners. The pressure to deliver quickly can lead to compromises in quality. It’s essential to resist this temptation and prioritize comprehensive testing to prevent costly failures.
2. Understand the criticality of your systems. Even seemingly noncritical systems can have significant impacts when they fail. Understanding the true risks associated with your software is key to making informed decisions about testing and quality assurance.
3. Learn from safety-critical industries. Industries like automotive and aerospace, where software failure can have life-or-death consequences, have developed rigorous testing standards. Adopting some of these best practices can help improve the reliability of your software, even if it’s not safety-critical.

Conclusion

The CrowdStrike software update failure highlights the importance of comprehensive, automated testing in today’s software development environment. Organizations can significantly reduce the risk of such incidents by leveraging Parasoft’s solutions for static analysis along with unit testing, code coverage, and other testing methods. Our tools help teams maintain high standards for software quality while managing business risks, ensuring that their software is reliable, secure, and ready to perform in the real world.

“MISRA”, “MISRA C” and the triangle logo are registered trademarks of The MISRA Consortium Limited. ©The MISRA Consortium Limited, 2021. All rights reserved.