Everything You Need to Know About Structural Code Coverage for C & C++

Structural coverage is the identification of code that has been executed and logged. There are several reasons why it’s important to perform this activity for embedded safety- and security-critical systems. One is to determine if the software has been adequately tested. Another is to satisfy compliance and certification requirements. You might also want to ensure there’s no dead code in your software.

This presentation dives deep into structural code coverage for C and C++ development, focusing on its importance for safety-critical systems and exploring various measurement criteria like statement, branch, and MC/DC coverage. We’ll also cover practical methods for achieving and automating code coverage.

We’ll show you structural code coverage for statement, branch, and MC/DC along with automated reporting and metrics for code coverage and code complexity.

Understanding Code Coverage

Code coverage is all about identifying which parts of your code have actually been executed during testing. It’s a key metric to determine if your software has been tested thoroughly enough. For safety- and security-critical embedded systems, this is vital for compliance and certification. Plus, it helps you find and eliminate dead code – code that’s never run.

Essentially, code coverage answers the question: “Have I tested enough?” It also helps uncover bugs lurking in untested sections. The big question is, do you want to risk those untested areas? In many regulated industries, you might not have a choice – you may need to achieve 100% coverage.

Key Takeaways

• Purpose: Identify executed code, ensure adequate testing, satisfy compliance, and find dead code.
• Mechanism: Typically achieved through code instrumentation, where extra code is added to log execution.
• Metrics: Common types include statement, branch (decision), and Modified Condition/Decision Coverage (MC/DC).
• Standards: Industries like automotive (ISO 26262), aerospace (DO-178C), and medical (IEC 62304) have specific coverage requirements based on safety integrity levels.
• Automation: Tools can automate test case generation and coverage analysis, integrating into IDEs and CI/CD pipelines.
• Challenges: Achieving 100% coverage can be difficult due to defensive code or unreachable code, often requiring visual inspection (debugger) or specific test case design.

Demonstration Highlights

During the demonstration, key capabilities are showcased:

• IDE Integration: Visualizing coverage metrics directly within IDEs like Eclipse and VS Code, showing line, statement, branch, and MC/DC coverage.
• Automated Test Generation: Generating unit tests that contribute to coverage goals.
• Application Coverage (Command Line): Using tools like C/C++test to instrument, build, and run tests on applications outside the IDE, generating coverage logs that can then be imported back into an IDE for analysis.
• Target Hardware Testing: Support for collecting coverage data on embedded targets, with options to optimize for size or speed.
• Coverage Dashboard: Centralized reporting and analysis of coverage data using platforms like DTP.

How Code Coverage Works

The most common way to get code coverage is through code instrumentation. This involves adding small pieces of code to your original source code. These additions track whether a statement, decision, or branch has been executed. The instrumentation then logs this information, allowing tools to calculate a coverage percentage and visualize which parts of the code were hit.

You might see your code highlighted in green (tested) with some red parts (untested). Some tools show partially covered code, often in yellow. This might happen with an if statement that has multiple conditions, where not all possible outcomes of those conditions were tested.

Types of Structural Coverage

When we talk about structural coverage, we’re looking at different ways to measure how thoroughly the code’s structure has been tested. While terms can sometimes be overloaded or interpreted differently across industries, the primary types for safety-critical C and C++ applications are:

• Statement Coverage: Ensures every executable statement in the code has been run at least once.
• Branch Coverage (or Decision Coverage): Guarantees that every possible branch (like the true and false outcomes of an if statement) has been executed.
• Modified Condition/Decision Coverage (MC/DC): This is a more rigorous standard, often required for high-assurance systems. It ensures that each condition within a decision has been independently shown to affect the decision’s outcome. For a decision like (A && B), MC/DC requires tests that show A changing the outcome while B is fixed, and B changing the outcome while A is fixed, in addition to testing all branches.

Other types exist, like condition coverage (testing individual conditions) and line coverage (ensuring each line is executed, which can differ from statement coverage if multiple statements are on one line). However, statement, branch, and MC/DC are the most frequently cited in safety standards.

Automating Code Coverage

Manually figuring out the exact test cases needed to achieve specific coverage goals can be incredibly time-consuming. Fortunately, tools can help significantly.

Coverage Advisor features can analyze your code and suggest specific parameter values or pre-conditions for your unit tests to hit particular lines or branches. This can drastically speed up test case creation.

Automated Unit Test Case Generation is another powerful capability. Tools can automatically generate a suite of unit tests designed not just to check functionality but also to meet your structural coverage requirements. These generated tests often include various types of checks, like null pointer tests, boundary value tests, and mid-max tests, to uncover potential bugs and improve coverage.

Combining Coverage from Different Methods

It’s rare that a single testing method will get you to 100% coverage, especially if your target is high. Here’s how different methods can be combined:

• System Testing: Running your existing system tests against instrumented code can cover a large portion of your application with minimal extra effort. However, system tests often miss defensive code paths that only trigger under fault conditions (like memory corruption or hardware failures), typically resulting in coverage in the 60% range.
• Unit Testing: Creating targeted unit tests is crucial for exercising these defensive code paths or other specific scenarios that system tests miss. By designing unit tests to hit specific conditions, you can significantly increase your overall coverage.
• Manual Testing: Even manual tests can contribute to coverage data when run against instrumented code.

Modern tools allow you to merge coverage results from these different testing methods. This gives you a consolidated view of your total code coverage, satisfying compliance requirements that permit combining metrics.

Handling Challenges

• Unreachable Code: Sometimes, code might be structured in a way that makes it impossible to reach through normal testing (e.g., an infinite loop followed by a return statement). In such cases, standards often allow for coverage by inspection. This means you can visually confirm the code is unreachable or use debugging tools to step through it, documenting why it’s considered covered.
• Code Bloat: Instrumentation can sometimes increase the size of your executable, potentially preventing it from fitting on resource-constrained target hardware. A common workaround is to instrument half the code, run tests, capture coverage, then un-instrument the first half, instrument the second half, run tests and merge coverage results.
• Assembly Code Coverage: For the highest safety levels (like DO-178C DAL A), coverage analysis might be required at the assembly or object code level. This is because compilers and linkers can generate code not directly traceable to the source. Specialized tools can automate the process of gathering structural coverage from executable object code.

Integrating Coverage into CI/CD

For modern development, integrating code coverage into your Continuous Integration/Continuous Delivery (CI/CD) pipeline is key. Tools can instrument your application as part of the build process, collect raw coverage data during automated tests, and then report on this data within your IDE or a central dashboard (like Parasoft’s DTP – Development Testing Platform).

This allows for continuous feedback on code quality and risk, enabling better decision-making throughout the development lifecycle. Integrations with popular CI/CD tools like Jenkins, GitLab, and Azure DevOps are common.

Conclusion

Structural code coverage is a critical practice for ensuring the quality, safety, and reliability of software, especially in regulated industries. By understanding the different coverage types, leveraging automation, and integrating coverage analysis into your development workflow, you can effectively meet compliance requirements and build more robust software.