Regression Testing of Embedded Systems

Development teams perform regression testing to verify that the code changes (fixing a bug or adding new functionality) in a software application don’t result in introducing another bug or breaking any of the existing system’s functionality.

For many, if not most embedded systems, teams perform regression testing at the end of the lifecycle to determine the stability of each software version release. This is an iterative process that continues until the project reaches the end of development or end of maintenance.

In other workflows, regression testing is an activity that is the day-to-day reality of developers. In fact, one can argue that in iterative and Agile processes, most testing is regression testing. Before we go any further, let’s look at what regression testing is and why it’s such a big part of software development practices.

What Is Regression Testing?

A regression is “a trend or shift toward a lower or less than perfect state,” which is what we’re all trying to avoid as we develop software. Regression testing helps find defects that creep into our software as we add new features, fix bugs, and when making changes to the test cases themselves.

As part of most software development processes, developers perform regression testing after making changes to the software. These tests determine if the new changes had an impact on the existing operation of the software.

Regression tests are necessary, of course, but they only indicate that recent code changes have not caused tests to fail. There’s no assurance that the changes work themselves. In addition, the nature of the changes that motivate the need to do regression testing can go beyond the current application and include changes in hardware, operating system, and operating environment.

Why Regression Testing in Embedded Systems Is Important

Since embedded systems tend to have safety- and security-critical constraints, dev teams test the system or subsystem for regressions. The testing arguably happens at the end of each iterative lifecycle of the system or subsystem. This means that all the unit test cases that were defined for a previous software system or subsystem release, plus the new unit test cases that have been added to verify new functionality, must be executed in order to expose regressions in the software.

If a previous unit test case passed and now fails, a potential regression may have been identified. New functionality could cause the failure. If this is the scenario, then the test case may need to be updated by taking into consideration changes in input and output values.

It’s also crucial to understand that regression testing does not just imply unit test cases. Regression testing of embedded systems also includes the execution of integration test cases, system test cases, performance test cases, stress test cases, and more. In fact, developers should execute all previously created test cases to ensure:

• No regressions exist.
• Construction of a new, high-quality software version release.

Both are critical because each new software system or subsystem release is built or developed upon its previous release. If you don’t have a solid foundation the whole thing can collapse.

Parasoft C/C++test can automatically generate test cases. Combining them with manually created test cases results in a collective set: a test suite. Developers and testers execute the test suite to identify application defects.

Parasoft C/C++test captures the test results and identifies failures to analyze and fix. After this initial cycle, teams can reuse the existing test cases or test suite for regression testing.

Regression Testing in Agile Development

It’s probably obvious that updates and changes to software need to be tested. In that sense, regression testing is an obvious step in the development process. In modern software development with Agile processes and continuous integration and deployment, regression testing becomes a critical step in each cycle.

As software grows, so does the suite of regression tests. As the suite grows, so does execution and debugging time and often the bottleneck in the pipeline. It’s difficult to have Agile development and a streamlined DevOps pipeline without focused and streamlined regression testing.

Regression Testing for Embedded Devices

Working on embedded systems adds another dimension to testing because it’s often preferable or required to perform testing on target hardware. Regression testing can be more challenging due to the complexity of initiating and observing tests on embedded targets. Add to that the limited access to target hardware that software teams have due to the high cost of these target systems.

So putting in place a reusable and configurable test automation solution that can seamlessly transition in a continuous way from execution on a host and/or a virtual system — and for verification and validation on the target system — provides significant savings in resources, time, and cost.

Development teams can use Parasoft to execute unit tests on the host platform, target processor simulator, or the embedded target. Optimized to take minimal additional overhead for the binary footprint or process cycles, Parasoft C/C++test’s test harness comes in the form of source code. That means teams can customize for required platform-specific modifications. This is necessary because test results and code coverage data collection from the target system is essential for safety- and security-critical systems and required by the process standards like DO-178B/C, ISO 26262, IEC 62304, and more.

Additionally, Parasoft C/C++test offers dedicated integrations with embedded IDEs and debuggers that make the process of executing test cases smooth and automated. Supported IDE environments include Eclipse, VS Code, Green Hills Multi, Wind River Workbench, IAR EW, ARM MDK, ARM DS-5, TI CCS, Visual Studio, and many others. See all Parasoft C/C++test Technical Specifications.

Regression Testing Best Practices & Techniques

At the core of regression testing is the legacy of test cases from the previous software version tested. Regression testing is about reusing existing test cases to identify a relapse in code quality between each software update and new release. Therefore, best practices consist of applying automation to manage the large collection of regression test cases.

To ensure a solid test suite for each regression test undertaken, teams should perform the following tasks.

• Identify test cases that need updating.
• Update test cases to test code changes.
• Create new test cases to test new code.
• Delete irrelevant test cases.

Because there can be thousands of test cases, QA teams will want to automate the management of regression tests. Organizations will want to adopt a solution that automatically identifies the test cases that need to be updated due to software changes. Change base testing for Java, C#, and VB.Net testing solutions are explained below. Solutions like Parasoft expedite changes to existing test case logic, including stubbing and managing variations of input and output test values.

For newly added code, new test cases will need to be created. Parasoft enables automated unit test case generation. In addition, a graphical interface helps accelerate manual test case creation. For deleted code, don’t forget to delete all corresponding irrelevant tests.

How to Decide What to Regression Test?

The key challenge with regression testing is determining what parts of an application to test. It’s not uncommon to default to running all regression tests when there’s doubt about the impacts of recent code changes. It’s that “all or nothing” approach. However, for large software projects, this becomes a huge undertaking and drags down the productivity of the team. This inability to focus testing hinders many of the benefits of iterative and continuous processes — potentially exacerbated in embedded software where test targets are a limited resource.

What is needed is a way to identify which tests need to be re-executed and focus the testing efforts (unit testing, automated functional testing, and manual testing) on validating the features and related code that are impacted by the most recent changes. Using a combination of Parasoft’s proprietary coverage analysis engines (C/C++test for C & C++, Jtest for Java, and dotTEST for C#) and the Process Intelligence Engine (PIE) within Parasoft DTP, developers, and testers can understand the changes in the code base between builds, and with this improved efficiency, achieve the promise of Agile. This form of smart test execution is called test impact analysis, sometimes referred to as change based testing.

Understand the Impact of Code Changes on Testing With Test Impact Analysis

Test impact analysis uses data collected during test runs and changes in code between builds to determine which files changed and which specific tests touched those files. Parasoft’s analysis engine can analyze the delta between two builds and identify the subset of regression tests to execute. It also understands the dependencies on the units modified to determine what ripple effect the changes made on other units.

Parasoft Jtest and dotTest provide insights into the impact of software changes and offer recommendations for where to add tests and where to run more regression tests. See the example change based testing report below.

Regression Test Earlier and More Often

Streamlining regression testing with test impact analysis lowers the overall burden of testing considerably. It enables developers and testers to focus solely on the impacted code and tests. The results?

• More testing to increase code coverage.
• More cycles to converge on a better product.
• Or a combination of the two.

Regression testing is no longer a burden but a key step in improving product quality and security. It generates metrics to help measure quality, security, and overall progress.

With state-of-the-art tool support and target-based testing, it’s also possible to start regression testing as soon as developers create code. Each new unit test becomes a regression test over time and developers can leverage change-based testing right away.

Doing regression testing early (shifting it to the left of the development timeline) means earlier detection of defects. Shifting left saves time and money immediately. It pays off even more later in the software lifecycle.

Don’t underestimate this payoff. It can save significant downstream costs. Finding defects earlier in the SDLC cuts the cost of fixing them and reduces the impact on downstream activities. Check out the diagram below.

Summary

Regression testing is a core activity in embedded software testing. Each newly added feature, bug fix, or configuration change requires tests to confirm what already works and continues to work. However, as a project grows and software complexity grows, so does the number of regression tests.

Eventually, test automation becomes necessary to help deal with the testing burden. New technology such as smart test execution makes regression testing easier. It drives developers and testers to focus on tests that specifically test the changed and impacted code.

Combining process improvements like Agile, CI/CD, and DevOps with software test automation makes it possible to start regression testing almost as soon as the code is written. It effectively shifts testing to the left and detects bugs earlier when they are cheaper and easier to fix.