The Value of Using a Unified C/C++ Testing Tool

A unified testing solution with deep integration into the developer’s IDE provides the most productive environment for development testing. A unified tool, such as Parasoft C/C++test, enables teams to focus their testing on high-risk and most recently-modified code.

Software verification and validation is an inherent part of software development. The effort and budget put into particular V&V projects depends on many factors, such as functional safety objectives of a project, the level of business risk, or the quality culture of an organization. Regardless of what drives an organization to implement quality initiatives and processes, it takes more than determination to produce safe and high-quality software products.

Choosing appropriate testing methodologies is a challenging task for many reasons. Technology is evolving at a rapid pace, and companies have to choose which software testing tools to adopt. In many cases, choosing between open source and commercial products is also difficult.

This post explains how automated testing techniques, such as advanced static analysis, runtime memory monitoring, automated unit testing, and flow analysis, can be combined to improve quality assurance processes, and identifies the benefits of implementing a unified testing solution. The concepts discussed here are generic and can be applied to any programming language, but the examples here have C and C++ programming languages in mind, created using Parasoft C/C++test.

Defect Classes & Tool Automation

When thinking about possible high-level software malfunctions, several classes of software errors can be distinguished: errors resulting from missing requirements, errors caused by incorrectly specified requirements, and errors that occur because requirements have been coded incorrectly. The first two classes of software defects fall into the category of requirements engineering, and won’t be discussed here. I’m focusing here on errors caused by incorrect implementation which includes a wide range of potential software problems that many teams struggle with.

So what does it mean that a requirement has been coded incorrectly? It can be many things. Take, for example, an incorrectly implemented requirement — unit tests designed to check the correct implementation fail and test automation tools detect and report the defect. In another example, a runtime analysis tool may detect a critical memory access error during a unit test that is undetectable from the unit test results alone. Different tools are better at detecting certain classes of errors, as illustrated below.

Figure 1. Software defects and a detection strategy landscape.

Obviously, not all software projects need to use all available technologies for testing and improving software quality, and organizations face the dilemma of how to balance budget and quality. Projects that are related to functional safety are likely to select all available technologies to ensure uncompromised quality, as well as compliance with software safety standards, such as ISO 26262. Other teams may decide to choose only static analysis and unit testing, as they seem to cover a significant portion of software defects, while some teams may find an open source unit testing framework sufficient.

Unwillingness to implement a wide range of testing technologies often stems from a concern that using multiple techniques imposes significant overhead on the pace of development, not to mention the budget. This is more likely a real concern if teams decide to select unintegrated tools. The cost of multiple separate tools, the learning curve, and the necessity to switch between different use models and interfaces can be problematic. As a result, developers may avoid using tools and automation, as it redirects their attention from writing code to tool usage, decreasing their productivity.

The Value of a Unified Testing Solution

It is important to specify what is expected from a unified testing solution before discussing its value. Ideally, a solution should:

• Support multiple testing technologies
• Be easy to use
• Detect functional problems and regressions
• Provide traceability from requirements to tests
• Measure the complexity, portability, and maintainability of code
• Educate developers by providing instant feedback as they are writing the code
• Provide information about development progress
• Combine results from different techniques for advanced analytics

A unified and integrated testing solution helps avoid a number of problems:

• Multiple learning curves and usability issues with using detached tools with different interfaces
• Distracting developers from writing code
• Preventing the exchange of information between different elements of a toolchain

Detect as Many Defects as Possible

Software defects fall into different categories, so it cannot be expected that all of them are identified by one testing technique. For example, manual system-level testing. To provide a more concrete example, observe the following example code snippet:

The few lines of code above contain several issues. Specifically, in line 16, the developer is trying to initialize global buffer using the index value computed in the calculateIdx() function, but they fail to validate whether it is within the allowed range. Even if dozens of manual testing sessions are run, they may not reveal this issue, as writing an integer value to a random memory location rarely produces a spectacular effect immediately.

But one day, most likely after the release, the memory layout may change, and the operation from line 16 crashes the application. Interestingly, static analysis may also fail to flag this problem because of the loop that is used to compute the index. For performance reasons, data and control flow analysis used in static analysis tools have to apply heuristics and simplifications to finish code analysis within a reasonable time, and it is common to apply them in loops meaning some errors may be missed.

A memory monitoring tool, which instruments source code by injecting special checks, and reports improper memory operations, is certain to detect this error. Runtime analysis tools detect software problems only on paths that are actually executed, without making any guesses — a big advantage over static analysis, as the accuracy of reported problems is very high.

In this example, Parasoft’s memory error detection easily identified the problem:

A reverse situation is equally possible – static analysis may detect issues that runtime memory monitoring cannot identify. For example, there is the possibility of a null pointer dereference at line 27/28 in the code snippet below. Calling the storePersonToFile function with the person pointer argument being null causes an error, but this occurs only if the retrivePersonFromDB function returns null. Such a condition is unlikely during system testing sessions, because the database connection is most likely functioning as expected, so runtime memory monitoring tools remain silent. However, flow analysis within the static analysis tool easily detects the null pointer to be returned from this function and reports a potential null pointer dereferencing problem.

Figure 4. Static analysis results example.

Detect Functional Problems and Regressions

What about code that always executes flawlessly, but does not comply with requirements? Static and dynamic analysis aren’t useful in identifying these kinds of issues. Detecting discrepancies between the expected result and the actual result requires a comparison of computation results with predefined values. There are many popular ways of implementing such checks, including manual system-level testing, integration testing, and unit testing.

There are many areas where a unified testing solution can facilitate the unit testing process. The list of benefits includes:

• Increases developer productivity while creating test cases, for example, by providing graphical wizards for creating, editing, and configuring test cases.
• Integrates with Test Double frameworks, allowing easy mocking and stubbing to simulate complex test scenarios, without involving large code bases.
• Correlates test cases with code coverage reports to assess the completeness of testing.
• Optimizes testing sessions by automatically computing the minimal set of test cases required to verify the code delta.
• Merges code coverage reports from other testing techniques, such as manual/system level testing or integration testing to identify untested code.
• Traces results back to requirements for a better understanding of the impact of failing tests.

Test creation with graphical wizards or editors can increase the productivity of an entire organization by engaging the QA teams in the process of writing unit tests, thus making them actively contribute to the development process. It is easier for QA team members to use graphic wizards than write appropriate test code in code editors, since configuring values using entry forms (as shown below) does not require advanced coding skills and is less time-consuming, especially if team members lack experience. For example, the screenshot below shows an example of Parasoft C/C++test’s unit testing capability for C/C++, a wizard that helps with unit test creation.

Another benefit of using a unified testing solution is the feedback it provides about the completeness of testing and the health status of critical business requirements. Low numbers in the MC/DC coverage report may indicate insufficient branch coverage, even if the statement coverage report shows high values. This kind of analysis requires a tool chain that supports multiple coverage metrics, allowing teams to start with something simple, such as line or statement coverage, and proceed towards more thorough code coverage as they are progressing with improving their test cases.

Requirements to Test Traceability

Unit testing and system testing coverage reports are great sources of information about the testing process, especially when combined. But if test results aren’t correlated with requirements, teams lack several pieces of critical information. By viewing a tests-to-requirements traceability report, you can quickly determine the status of requirements coverage. An example of such a report is shown below:

The ability to correlate requirements with results from different types of testing is a great benefit of using a unified testing solution. Unit testing, system testing, integration testing results, as well as coding standards, and code metrics results can be correlated to provide feedback about the health of critical and non-critical requirements.

Combine Results From Different Techniques for Advanced Analytics

When you combine data from the various testing techniques used in a project, you can attain second-level metrics and more sophisticated analytics. Using a unified testing solution enables teams to look at code from a completely new perspective. A failed unit test case may have a different meaning to a developer, depending on whether it occurs in code rated high or low risk by code metrics analysis. If this information is further combined with statistics from source control and correlated to requirements, teams can make better decisions about when and how the code should be corrected.

With the Parasoft tool suite, you can leverage change-based testing to improve team productivity. Change-based testing technology captures the relationship between source code, test cases, and code coverage results to compute the optimal set of test cases for verification and validation of a specific code delta. In effect, teams can limit their testing sessions by running only a small subset of tests instead of full regression suites. This focus on only testing what is absolutely required translates to significant savings, especially for middle or large size projects.

Putting It All Together: Parasoft C/C++test & DTP

All of the testing technologies discussed above are available in Parasoft C/C++test and Parasoft DTP. Parasoft C/C++test is a unified testing solution for C and C++ projects. C/C++test is available as a plugin for popular IDEs, such as Eclipse and Visual Studio. Close integration with the IDE prevents the problems discussed above.

Developers can instantly run coding standards compliance checks or execute unit tests at the time of writing code. QA team members can perform manual test scenarios while the application is monitored for code coverage and runtime errors. Offline analyses, such as flow analysis provided by static analysis, can be performed in the continuous integration phase.

Server sessions, supported with a convenient command line interface, report analysis results to Parasoft DTP, which aggregates information from the development environment. Parasoft DTP provides extensive reporting capabilities, including the ability to integrate data from third-party tools, send test results to developers’ IDEs, and compute advanced analytics. Continuous feedback provided in each phase of the workflow accelerates agile development and reduces the cost of achieving compliance with safety standards.

It should also be noted that the approach to quality assurance processes and architecture discussed here is not limited to C/C++ projects. Parasoft provides similar solutions for Java and C#/.NET.

Conclusion

Software quality improvement is not a homogeneous endeavor, and it requires different technologies to eradicate different types of software defects. The key challenge is how to do it in an effective and efficient way that avoids devastating project budgets and undermining the morale of developers.

A unified testing solution with deep integration into the developer’s IDE provides the most productive environment for development testing. A unified tool, such as Parasoft C/C++test, with its ability to unify metrics and results from all aspects of testing, provides an additional benefit by allowing teams to focus their testing on high-risk and most recently modified code.