See what API testing solution came out on top in the GigaOm Radar Report. Get your free analyst report >>
Regression Testing
As part of most software development processes, regression testing is done after changes are made to software. These tests determine if the new changes had an impact on the existing operation of the software.
DO-178C doesn’t explicitly mention regression testing, but it is a good engineering practice and is widely employed in the aerospace industry to verify the stability and correctness of the software throughout its development lifecycle. Requirements around the software and hardware integration process imply the need to maintain up-to-date verification and validation after any changes.
Requirements-Based Hardware/Software Integration Testing, Section 6.4.3
Integration testing is a level of testing in DO-178C that verifies the interactions between different software units. When changes are made to software components or units, regression testing is necessary to verify that the modifications have not adversely affected the integrated system.
Integration Process, Section 5.4
Focuses on the integration of software components and emphasizes that the integration process should be planned and controlled. Integrating new or modified software units requires regression testing to ensure that the system’s overall behavior remains correct and that no unintended side effects have been introduced.
Software Verification Results, Section 11.14
Covers the documentation and recording of verification results, including the results of testing activities. If regression testing is performed, the results should be documented to demonstrate that the changes did not negatively impact the system.
Regression tests are necessary, but they only indicate that recent code changes have not caused tests to fail. There’s no assurance that these changes will work. In addition, the nature of the changes that motivate the need to do regression testing can go beyond the current application and include changes in hardware, operating system, and operating environment.
Software Regression Testing in Airborne Systems
In safety-critical software development, validation is critical in proving correct functionality, safety, and security. Tests are needed for two primary reasons.
- Confirm any changes to the application to ensure functionality.
- Verify that there aren’t any unforeseen impacts on the rest of the system.
If a test case previously passed but now fails, a potential regression has been identified. The failure could be caused by new functionality, in which the test case may need to be updated so that it takes into consideration changes in input and output values.
Regression testing of embedded systems also includes the execution of the following types of test cases:
- Unit
- Integration
- System
- Performance
- Stress and more
In fact, all previously created test cases need to be executed to ensure that no regressions exist and that a new dependable software version release is constructed. This is critical because each new software system or subsystem release is built upon it. If you don’t have a solid foundation the whole thing can collapse.
Parasoft C/C++test supports the creation of regression testing baselines as an organized collection of tests and automatically verifies all outcomes. These tests are run automatically on a regular basis to verify whether code modifications change or break the functionality captured in the regression tests. If any changes are introduced, these test cases will fail to alert the team to the problem. During subsequent tests, C/C++test will report tasks if it detects changes to the behavior captured in the initial test.
How To Decide What to Regression Test
The key challenge with regression testing is determining what parts of an application to test. It is common to default to executing all regression tests when there’s doubt on what impacts recent code changes have had-the all or nothing approach.
For large software projects, this becomes a huge undertaking and drags down the productivity of the team. This inability to focus testing hinders much of the benefits of iterative and continuous processes, potentially exacerbated in embedded software where test targets are a limited resource.
A couple of tasks are required here.
- Identify which tests need to be re-executed.
- Focus the testing efforts (unit testing, automated functional testing, and manual testing) on validating the features and related code that are impacted by the most recent changes.
Developers and testers can get a clear understanding of the changes in the codebase between builds using the Process Intelligence Engine (PIE) within Parasoft DTP (Development Testing Platform) combined with Parasoft’s proprietary coverage analysis engines:
With this combination, teams can improve efficiency and achieve the promise of Agile. This form of smart test execution is called test impact analysis. It’s sometimes referred to as change-based testing.
Understand the Impact of Code Changes on Testing With Test Impact Analysis
Test impact analysis uses data collected during test runs and changes in code between builds to determine which files have changed and which specific tests touched those files. Parasoft’s analysis engine can:
- Analyze the delta between two builds.
- Identify the subset of regression tests that need to be executed.
- Understand the dependencies on the units modified to determine what ripple effect the changes have made on other units.
Parasoft Jtest and dotTEST provide insight into the impact of software changes. Each solution recommends where to add tests and where further regression testing is needed.
