Shift Left Your Safety-Critical Software Testing With Test Automation

We’re in a cost crisis with safety-critical software, which means that the increased functionality required has grown beyond the ability to pay for its development. The Boeing 787 program, for example, required 6.5 million lines of code, which cost $4 billion to design, develop, and test. The cost numbers for Boeing’s 777X program were not publicly disclosed and Boeing’s 737 MAX was originally estimated at $3.6 billion but it rose to $6.3 billion, which does not include the billions in failure costs.

The trend in major safety-critical projects shows exponential growth in total costs with software being a larger portion of the total development budget year-over-year. The next big aerospace program will likely be unaffordable using the same techniques used in previous programs. So, what can we do?

Safety certification and the required test and verification required is a large portion of the software development budget. Shift-left testing of software—testing earlier in the SDLC—while leveraging automation, pays big dividends in cost, risk, and schedule. The image below shows the cost in millions of dollars per thousand lines of code for commercial airline software development, which clearly shows the exponential increase.

Understanding When Bugs Are Introduced & Detected

Unsurprisingly, most defects are introduced into a project at the beginning, even before the first line of code is written. Most bugs are found and fixed during testing. But a good percentage, as much as 20%, are discovered during operation after the product has been sold and shipped.

In certified systems, this either means an extremely expensive fix-test-recertify cycle or operator workarounds for the problem. The next graph shows the relative percentage of bugs introduced and detected over each phase of the software development life cycle.

Defects are cheapest to fix early in the life cycle. They become exponentially more expensive to find and fix over the length of the project. In operation, after a product has shipped into customers’ hands, it’s the most expensive to fix. The post-deployment defect fix costs are conservative and don’t include damage to your brand and liability from safety incidents in the field.

The following graph shows the relative cost of fixing a defect at each stage of the life cycle. Clearly, the goal is to move the defects detected and fixed to earlier in the life cycle. In other words, shift left. Additionally, it’s desirable to reduce the number of defects making it to the customer, a reality in every domain.

Shift-Left Testing Through Test Automation

The safety-critical software industry recognizes the need to change the way things are done. Too many projects are reinventing the wheel, and certifying brand new software is time-consuming and expensive. The growth in connectivity and functionality of new products means methods need to change. In this post, we won’t cover all of the techniques being proposed, but rather concentrate on the role that test automation plays in shifting left the reduction, detection, and fixing of defects and security vulnerabilities.

Advantages of Test Automation

A huge part of any safety-critical project is testing. Automation is absolutely necessary to achieve safety, security, and quality goals. Following are examples of the ways in which testing automation tools support modern software development methods and increase testing and documentation productivity.

Supporting Agile & CI/CD Pipelines

The problems with the waterfall method are understood and many teams are using more modern development methods to improve quality and safety. Test automation is an important part of any iterative development method since test suites are run on each new iteration of a module, component, and so on.

Test automation supports these methods with repeatable automated tests, providing reporting at various levels for each test but also accumulative results over time. These tools are also designed to work with CI processes by integrating with work item tracking, requirements management, build systems, and source repository services. Dynamic analysis tools are critical for detecting runtime errors that are difficult to detect. Static analysis plays an important role in detecting defects before testing starts.

Supporting Software Inspections

One of the best practices to remove defects early in the development life cycle is inspections. Inspections mean reviewing everything, not just source code. For example, inspecting requirements and design is critical in preventing the leading source of bugs in the system. See Figure 2. Many bugs are literally designed into the system. Tools play a lesser role at this stage but do enhance the effectiveness of code reviews.

Automated unit testing, dynamic error detection, and static analysis provide greatly improved error detection in the early coding stages of a project. The results of automated testing can be presented in code reviews, thus decreasing the dependence on manual error detection and allowing more time for detecting incorrect requirements and design decisions.

Increasing Testing Productivity

Manual testing is tedious and less repeatable. Results collection can be ad hoc, and an error can be missed despite “correct” results. Achieving required code coverage, which varies based on safety standards and the criticality of the project, is difficult to track.

Test automation not only makes testing much less tedious and repeatable, but the reporting capabilities of advanced testing tools create important management information on the state of the project. Adding dynamic analysis, which analyzes the code when it’s running to detect tricky runtime errors, and static analysis, which analyzes the code before it’s run, greatly increases the bug detection capability of test tools.

Automating Coding Standard Compliance

Many safety-critical projects require source code standards. MISRA, for example, is common in automotive software but has gained acceptance in other industries. Some standards require that code comply with a company standard that meets certain goals. In each case, manually enforcing coding compliance is tedious and error-prone. Static analysis tools are ideal for enforcing compliance and advanced tools go further by detecting errors that are beyond format violations.

Automating Certification Documentation

A large portion of the workload in achieving software safety certifications is in documenting processes, validation, and verification. Test automation decreases the cost of documenting test results and coverage analysis significantly.

Accelerating Reuse of Legacy Software

A key strategy for increasing productivity is to reuse software. Ideally, already-certified components can be used to reduce the development costs for these subunits. Automating the evaluation of legacy software with static and dynamic analysis tools decreases the risk of using these components.

Improving Quality, Safety, & Security

Even strict testing regimens can miss critical errors. Code coverage alone is not enough to ensure proper behavior in the case of security attacks or multithreaded code, for example. Static analysis tools can detect errors in source code without running a specific test and can find bugs such as security vulnerabilities that are hard to discover in unit or system testing.

Dynamic analysis tools can detect errors in running code during tests that might be reflected in test results, like a slow memory leak. Fuzzing and penetration testing during system testing can find bugs missed during normal operating conditions. In totality, the additional defects and security vulnerabilities found by state-of-the-art tools help reduce cost, risk, and many of the 20% or so bugs that make it into production.

What Is the Impact of Shift-Left Testing?

It’s clear that something must be done to solve the issues clearly shown in Figure 2. Too many defects are being introduced and staying undetected at the beginning of the life cycle. And too many are being left in the product when it is manufactured and in customers’ hands, or in planes or cars as the case may be. Adopting modern development methods, reusing components, leveraging COTS and open source, and tool automation are all key steps to improving development productivity.

Assuming a development process using state-of-the-art tools where testing is shifted left and more defects are detected and fixed earlier in the life cycle, unit testing is extremely effective, helping fewer bugs make it through to production. In Figure 4, a hypothetical example shows the change in defect detection over the life cycle, where the bulk of defect detection and fixing is shifted to the left earlier in the life cycle.

We know from Figure 3 above that costs rise significantly at each phase of development. Figure 5 below shows the comparison of costs to fix defects in the traditional method versus the modern method shown in Figure 4. Shifting the finding and fixing bugs to the left, costs less than fixing them later. In the situation presented here, the overall cost difference is about 40% in favor of the shift-left approach.

The Importance of Certified Tool Chains & Qualification Assistance

Using automated tools in safety-critical projects requires trust in the tools themselves. The onus is on the manufacturer of the product to have confidence that the processes and tools used to create the software meet the requirements of the standard. Tool vendors can help with this by having tools certified by safety standards bodies before sale to the manufacturers, or, in cases where no such pre-certification is possible, provide qualification assistance. They can then use the tool vendor’s certification evidence in their own submission for certification and reduce the effort needed. For example, Parasoft C/C++test has been certified by TÜV SÜD to be qualified for safety-related software development according to IEC 61508 and ISO 26262 standards.

In some software safety standards, such as DO-178C, certification is done on a system level, and individual tools and software aren’t certified independently. In these cases, the tool’s vendor provides qualification kits and assistance in terms of documentation and professional services, greatly reducing the cost and effort needed to qualify the tools for use on the project.

Navigating Safety-Critical Software Costs With a Shift-Left Strategy

Safety-critical software is certainly in a cost crisis. New, large safety-critical projects are becoming too expensive to develop, to the point that they may not be profitable. Modern methods are needed to develop safety-critical software, and that effort must reduce the number of errors found late in the software development life cycle. Shifting left, the detection and fixing of defects and security vulnerabilities as early in the life cycle as possible reduces costs significantly. Test automation plays a key role in improving test efficiency and outcomes. It’s an important part of the modern approach to safety-critical software development.

“MISRA”, “MISRA C” and the triangle logo are registered trademarks of The MISRA Consortium Limited. ©The MISRA Consortium Limited, 2021. All rights reserved.