The Essential Guide to Automated Test Processes for Embedded Software Systems

An essential start to the journey of delivering high-quality, real-time embedded software systems is incorporating static analysis into the development workflow. With static analysis, software engineers can enhance the reliability, performance, and security of their software. At the same time, they can reduce the cost and time associated with identifying and fixing defects later in the development cycle.

Static analysis identifies errors in the code at an early stage in the development process, long before the software runs on the target hardware. This includes detecting syntax errors, logic errors, and potential runtime errors. Security is also paramount in embedded systems, especially those that are part of a critical infrastructure or the Internet of things (IoT) devices. Static analysis can uncover vulnerabilities such as buffer overflows, input validation flaws, and other security weaknesses that could be exploited by attackers.

In addition, most development teams will agree that unit testing is also essential to embedded software development despite the effort and costs. Unit testing helps developers truly understand the code they’re developing and provides a solid foundation to a verification and validation regimen needed to satisfy safety and security goals for a product. Building on this foundation of unit tests enables teams to accelerate agile development while mitigating risk of defects slipping into later stages of the pipeline.

Static Analysis

Automating static analysis into the software development workflow provides significant benefits.

• Integrates into developer IDEs and CI/CD pipelines. Integrating static analysis into the build process for the entire development organization is a best practice. With automation, teams can integrate static analysis into developers’ IDEs for quick analysis as they write the code. When teams integrate static analysis into CI/CD pipelines, code is automatically analyzed each time they submit code changes.

This ensures that issues are detected and addressed immediately, maintaining code quality throughout the development process, which is essential for large teams or projects with rapid development cycles. It will enforce coding standards across the entire code base, eliminating human error and bias in code reviews. This proactive approach will also ensure security by identifying potential vulnerabilities to be mitigated before they can be exploited.

• Ensures continuous compliance. Automated static analysis ensures continuous compliance with relevant standards like MISRA and CERT for industries for embedded software testing such as automotive, medical devices, and aerospace with strict regulatory requirements. This is crucial for passing audits and certifications that require static analysis.
• Eliminates repetitive, mundane tasks. Developers don’t have to perform repetitive and mundane tasks of reviewing code and assigning identified coding violations to fix. Parasoft applies patented AI and ML solutions to the static analysis workflow to prioritize rule violation findings and streamline remediation steps, immediately reducing team effort in adopting static analysis. Developers can focus on more complex and creative aspects of software development, which leads to increased job satisfaction and productivity.

Unit Testing

While static analysis provides valuable insights into code quality and potential issues without execution, unit testing verifies the actual behavior and correctness of the code in execution. Using both techniques together ensures a more robust and reliable software development process, as they complement each other and cover a wider range of potential problems.

However, despite the benefits, development teams often struggle with performing sufficient unit testing. The constraints on the amount of testing are due to multiple factors such as the pressure to rapidly deliver increased functionality and the complexity and time-consuming nature of creating valuable unit tests.

Common reasons developers cite that limit the efficiency of unit testing as a core development practice include the following.

• It’s difficult to understand, initialize, and/or isolate the dependencies of the unit under test.
• Determining what to validate and defining appropriate assertions is time consuming and often requires intelligent guess work.
• There’s a lot of manual coding involved, often even more than was required to implement a specific feature or enhancement.
• It’s just not that interesting. Developers don’t want to feel like testers. They want to spend time delivering more functionality.

Unit test automation tools universally support some sort of test framework, which provides the harness infrastructure to execute units in isolation while satisfying dependencies via stubs. This includes the automated generation of test harnesses and the executable components needed for host and target-based testing.

Test data generation and management, however, is the biggest challenge in unit testing and test generation. Test cases need to cover a gamut of validation roles such as ensuring functional requirements, detecting unpredictable behavior, and assuring security, and safety requirements. All while satisfying test coverage criteria.

Automated test generation decreases the inefficiencies of unit testing by removing the difficulties with initialization, isolation, and managing dependencies. It also removes much of the manual coding required while helping to manage the test data needed to drive verification and validation.

Static code analysis provides a variety of metrics that help assess different aspects of embedded code safety, security, reliability, performance, and maintainability. They each offer valuable information about various aspects of the code’s health. One of these metrics is cyclomatic complexity, which measures the number of linearly independent paths through a program’s source code.

Higher complexity indicates higher potential for bugs and harder to read, maintain, and test. Other values that static analysis can provide that aren’t quality measures by themselves, but give a sense of maintainability, include:

• Total number of lines of code
• Comment density
• Class coupling or cohesion
• Function length

Interesting metrics that static analysis can provide but aren’t generally applied include code coverage and performance metrics. Though code coverage is commonly captured by unit tests, static analysis can provide this metric and even find dead code. The performance metric is measured by identifying code that may lead to performance issues, such as inefficient loops or recursive calls. There are many other metrics around resource usage, duplication, null pointer, and divide by zero.

By taking advantage of these provided metrics, static code analysis helps developers identify potential issues early in the development cycle, improve code quality and maintainability, and ensure adherence to coding standards and best practices.

Software verification and validation is an inherent part of embedded software development, and testing is a key way to demonstrate correct software behavior. Unit testing is the verification of module design. It ensures that each software unit does what it’s required to do.

In addition, safety and security requirements may require that software units don’t behave in unexpected ways and are not susceptible to manipulation with unexpected data inputs.

In terms of the classic V model of development, unit test execution is a validation practice to ensure module design is correct. Many safety specific development standards have guidelines for what needs to be tested for unit testing. For example, ISO 61502 and related standards, have specific guidelines for testing in accordance with safety integrity level where requirements based testing and interface testing are highly recommended for all levels. Fault injection and resource usage tests are recommended at lower integrity levels and highly recommended at the highest SIL (Safety Integrity Levels) levels. Similarly, the method of driving test cases is also specified with recommended practices.

Test automation provides large benefits to embedded software. Moving away from test suites that require a lot of manual intervention means that testing can be done quicker, easier, and more often. Offloading this manual testing effort frees up time for better test coverage and other safety and quality objectives. An important requirement for automated test suite execution is being able to run these tests on both host and target environments.

For practical purposes, automated tools should generate test cases in existing well-known formats like CppUnit. By default, one test suite per source/header file makes sense, but tools should support one test suite per function or one test suite per source file if needed.

Another important consideration is the automatic stub definitions to replace “dangerous” functions, which includes system I/O routines such as rmdir(), remove(), rename(), and so on. In addition, stubs can be automatically generated for missing function and variable definitions. User-defined stubs can be added as needed.

Requirements Based Test Case Generation

Although test automation tools can’t derive requirements tests from documentation, they can help make the creation of test cases, stubs, and mocks easier and more efficient. In addition, automation greatly improves test case data management and tool support for parameterized tests also reduces manual effort.

Particularly important is traceability from requirements to code to tests and test results. Manually managing traceability is nearly impossible and automation makes two traceability a reality.

While requirements are being decomposed, traceability must be maintained throughout the phases of development as customer requirements decompose into system, high-level, and low-level requirements. The coding or implementation phase realizes the low-level requirements. Consider the typical V diagram of software.

Each phase drives the subsequent phase. In turn, the work items or refined requirements in each phase must satisfy the requirements from the previous phase. Architectural requirements that have been created or decomposed from system design must satisfy the system design/requirements, and so on.

Developers write code that implements or realizes each requirement and for safety-critical applications, links for traceability to test cases and down to the code are established. Therefore, if a customer requirement changes or is removed, the team knows what it impacts down the line, all the way to the code and tests that validate the requirements.

Industry standards like DO-178B/C, ISO 26262, IEC 62304, IEC 61508, EN 50128, and others require the construction of a traceability matrix for identification of any gaps in the design and verification of requirements. This helps achieve the ultimate goal of building the right product. More than that, it’s to ensure the product has the quality, safety, and security to ensure it remains the right product.

Code Coverage Based Test Case Generation

The creation of productive unit tests has always been a challenge. Functional safety standards compliance demands high-quality software, which drives a need for test suites that affect and produce high code coverage statistics. Teams require unit test cases that help them achieve 100% code coverage. This is easier said than done. Analyzing branches in the code and trying to find reasons why certain code sections are not covered continues to steal cycles from development teams.

Unit test automation tools can be used to fill in the coverage gaps in test suites. For example, advanced static code analysis (data and control flow analysis) is used to find values for the input parameters required to execute specific lines of uncovered code.

It’s also valuable if you have automated tools that not only measure code coverage but also keep track of how much modified code is being covered by tests, because this can provide visibility into whether enough tests are being written along with changes in production code. See the following example code coverage report.

As part of most software development processes, regression testing is done after changes are made to software. These tests determine if the new changes had an impact on the existing operation of the software. Managing and executing regression tests are a large part of the effort and cost in testing. Even with automated test generation, test execution, gathering results, and re-running tests is very time consuming. Regression testing encompasses test case maintenance, code coverage improvements and traceability.

Regression tests are necessary, but they only indicate that recent code changes have not caused tests to fail. There’s no assurance that these changes will work. In addition, the nature of the changes that motivate the need to do regression testing can go beyond the current application and include changes in hardware, operating system, and operating environment.

In fact, all previously created test cases may need to be executed to ensure that no regressions exist and that a new dependable software version release is constructed. This is critical because each new software system or subsystem release is built or developed upon. If you don’t have a solid foundation the whole thing can collapse.

To prevent this, it’s important to create regression testing baselines that are an organized collection of tests and will automatically verify all outcomes. These tests are run automatically on a regular basis to verify whether code modifications change or break the functionality captured in the regression tests. If any changes are introduced, these test cases will fail to alert the team to the problem. During subsequent tests, Parasoft C++test will report tasks if it detects changes to the behavior captured in the initial test.

How to Decide What to Test

The key challenge with regression testing is determining what parts of an application to test. It’s common to default to executing all regression tests when there’s doubt on what impacts recent code changes have had—the all or nothing approach.

For large software projects, this becomes a huge undertaking and drags down the productivity of the team. This inability to focus testing hinders much of the benefits of iterative and continuous processes, potentially exacerbated in embedded software where test targets are a limited resource.

A couple of tasks are required here.

1. Identify which tests need to be re-executed.
2. Focus the testing efforts (unit testing, automated functional testing, and manual testing) on validating the features and related code impacted by the most recent changes.

Test Impact Analysis

Test Impact Analysis (TIA) uses data collected during test runs and changes in code between builds to determine which files have changed and which specific tests touched those files. Parasoft’s analysis engine can analyze the delta between two builds and identify the subset of regression tests that need to be executed. It also understands the dependencies on the units modified to determine what ripple effect the changes have made on other units.

Focus on the Risk

Due to the complexity of today’s codebases, every code change, however innocuous, can subtly impact application stability and ultimately “break the system.” These unintended consequences are impossible to discover through manual inspection, so testing is critical to mitigate the risk they represent. Unless it’s understood what needs to be re-rested, efficient testing practice can’t be achieved. If there is too much testing in each sprint or iteration, the efficiency brought by test automation is reduced. Testing too little is not an option.

The best approach is to identify which tests need to be re-executed and focus the testing efforts (unit testing, automated functional testing, and manual testing) on validating the features and related code that are impacted by the most recent changes.

This is discovered with TIA and planning testing based on a data-driven approach called change-based testing.

TIA needs a repository of already-completed tests that are already executed against each build, either as part of a fully automated test process (such as a CI-driven build step) or while testing the new functionality. This analysis provides insight into where in the code the changes occurred, how the existing tests correlate to those changes, and where testing resources need to focus. Following is an example of a TIA.

From here, the regression test plan is augmented to address failed and incomplete test cases with the highest priority and using the retest recommendations to focus scheduling of additional automated runs and prioritizing manual testing efforts.

Testing is a major bottleneck for embedded software development with too many defects being identified at the end of the release cycle due to not enough or misdirected testing. To yield the best results, focus testing efforts on the impact of the changes being made to unlock the efficiency that test automation delivers.

Testing is essential to embedded software development. It fosters true understanding of the code being developed and provides a solid foundation to a verification and validation regimen needed to satisfy safety and security goals for a product.

Static analysis is a crucial technique in software development that involves examining source code without executing it to identify potential issues and improve overall code quality. When teams use it from the get-go, static analysis:

• Provides valuable insights into code complexity, maintainability, security vulnerabilities, and adherence to coding standards.
• Helps prevent bugs.
• Optimizes performance.
• Ensures compliance with best practices and regulatory requirements.

Integrating static analysis into the development workflow leads to a more robust and maintainable codebase. Doing so must be done in conjunction with unit testing so as not to be seen as a damper on productivity.

The constraints on testing productivity are due to multiple factors such as the pressure and time it takes to deliver increased functionality, and the complexity and time-consuming nature of creating valuable tests.

Test data generation and management is by far the biggest challenge in unit testing and test generation. Test cases are particularly important in safety-critical software development because they must ensure functional requirements and test for unpredictable behavior, security, and safety requirements. All while satisfying test coverage criteria.

Automated test generation decreases the inefficiencies of unit testing by removing the difficulties with initialization and isolation and managing dependencies. It also removes much of the manual coding required while helping to manage the test data needed to drive verification and validation. This improves quality, safety, and security. It also reduces test time, costs, and time to market.