DO-178C Software Compliance for Aerospace and Defense

Collecting and analyzing code coverage metrics is an important aspect of safety-critical software development. Code coverage measures the completion of test cases and executed tests. It provides evidence that verification is complete, at least as specified by the software design. The objectives for test coverage analysis include achieving the following test coverage targets:

How this translates to types and amounts of coverage is somewhat open to interpretation. However, in airborne software development, the onus is on the manufacturer to plan for code coverage, adhere to the plan, document, and complete it.

Types of Code Coverage

Following are the different types of code coverage.

Advanced unit test automation tools, such as Parasoft C/C++test, provide all these code coverage metrics and more. C/C++test CT also automates this data collection on host and target testing and accumulates test coverage history over time. This code coverage history can span unit, integration, and system testing to ensure coverage is complete and traceable at all levels of testing.

Coverage From System Testing

Obtaining code coverage through system testing is an excellent method to determine if enough testing has been performed. The approach is to run all your system tests, and then examine what parts of the code have not been exercised.

The unexecuted code implies that there may be need for new test cases to exercise the untouched code where a defect may be lurking and helps answer the question: Have I done enough testing?

When teams perform system testing, the average resulting metric is 60% coverage. Much of the 40% unexecuted code is due to defensive code in your application. Defensive code only executes upon the system triggering a fault or entering a problematic state that may be difficult to produce. Conditions like memory leakage or other types of faults caused by hardware failure may take weeks, months, or years to encounter.

There’s also defensive code mandated by your coding guidelines where system test cases can never get you to execute. For these reasons, system testing cannot take you to 100% structural code coverage. You’ll need to employ other testing methods like manual and/or unit testing to reach 100%.

Coverage From Unit Testing

As mentioned, unit testing can be used asa complementary approach to system testing to obtain 100% coverage. 0btaining code coverage through unit testing is one of the more popular methods used, but it doesn’t expose whether you have done enough testing of the system because the focus is at the unit level (function/procedure).

The goal here is to create a set of unit test cases that exercise the entire unit at the required coverage need (statement, branch, and MC/DC) in order to reach 100% coverage for that single unit. This is repeated for every unit until the entire code base is covered. However, to get the most out of unit testing, do not solely focus on obtaining code coverage. That can generally be accomplished through sunny day scenario test cases.

Truly exercise the unit through sunny and rainy-day scenarios to ensure robustness, safety, security, and low-level requirements traceability. Let code coverage bea biproduct of your test cases and fill in coverage where needed.

To help expedite code coverage through unit testing, configurable and automated test case generation capabilities exist in Parasoft C/C++test. Test cases can be automatically generated to test for use of null pointers, min-mid-max ranges, boundary values, and much more. This automation can get you far. In minutes, you’ll obtain a substantial amount of code coverage.

Additionally, C/C++test CT extends development workflows with code coverage by integrating with proprietary unit testing frameworks and IDEs. Tightly integrate code coverage line, statement, simple condition, decision, branch, function, call and MC/DC with proprietary unit testing frameworks like GoogleTest and CppUnit and IDEs like VS Code.

However, as in system testing, obtaining 100% code coverage is elusive due to the use of defensive code or formal language semantics. At the granular level of a unit, defensive code may come in the form of a default statement in a switch. If every possible case in a switch is captured, this leaves the default statement unreachable. In the example below, the return 0; will never get executed because the while (1) is infinite.

This coverage performed through manual/visual inspection and reports can be used to supplement the coverage captured through unit testing. The addition of both coverage reports can be used to prove 100% structural code coverage.

The goal of obtaining code coverage is an added means to help ensure code safety, security, and reliability.

Code Instrumentation

Code coverage is more often than not identified by having the code instrumented. Instrumented refers to having the user code adorned with additional code to ascertain during execution if that statement, branch, or MC/CD has been executed.

Based on the target or system under test, the coverage data can be stored in the file system, written to memory, or sent out through various communication channels, such as the serial port, TCP/IP port, USB, and even JTAG.

Partial Instrumentation

Be aware that code instrumentation causes code bloat. The increase in code size may impact the ability to load the code onto memory-constrained target hardware for testing.

The workaround is to instrument part of the code by following these steps:

1. Run your tests and capture the coverage.
2. Instrument the other part of the code.
3. Run your tests again.
4. Capture the coverage.
5. Merge the coverage from the previous test execution.

Coverage Advisor

Parasoft C/C++test resolves coverage gaps in test suites. Parasoft discovered how to use advanced static code analysis (data and control flow analysis) to find values for the input parameters required to execute specific lines of uncovered code.

In complex code, there are always those elusive code statements for which it is exceedingly difficult to obtain coverage. It’s likely there are multiple input values with various permutations and possible paths that make it mind twisting and time consuming to decipher. But only one combination can get you the coverage you need. Parasoft makes it easy to obtain coverage of those difficult to reach lines of code.

When you select the line of code you want to cover, the Coverage Advisor will tell you what input values, global variables, and external calls you need to stimulate the code and obtain coverage.

The figure on the right shows an analysis report providing the user with a solution. The Preconditions field expresses:

• The range and input values for mainSensorSignal and coSensorSignal
• The expected outputs from the external calls

Upon creating the unit test case with these set parameter values and stubs for external calls, you get coverage of the line selected, plus the additional lines expressed in the Expected Coverage field.

Object Code Coverage

For the most stringent safety-critical applications, DO-178C Level A, 0bject Code Coverage is required. Therefore, assembly level coverage must be performed. Imagine the rigor and labor cost of having to perform this task. Fortunately, Parasoft ASMTools provides an automated solution for obtaining object code coverage.