Functional Regression Testing With SOAtest as Part of a Continuous Integration Process

What Is Functional Regression Testing?

A regression is defined as “a trend or shift toward a lower or less than perfect state,” which is what we’re all trying to avoid when developing software. Regression testing discovers defects that creep into our software as we add new features, fix bugs, or otherwise change the application.

Regression testing is a common method that most software development processes use to double-check the functionality of software after making modifications to it. These tests determine whether the new modifications affect the software’s current functionality. Of course, new tests are needed for new functionality. Regression tests make sure what is already tested remains functional.

Regression testing plays a vital role in continuous integration (CI) in software development by ensuring that the newly integrated code does not break existing functionality or introduce regressions. These tests provide the rapid feedback essential in CI by detecting defects in newly integrated code so they can be quickly identified and resolved before they impact the stability of the software.

Functional Regression Testing vs. Functional Testing

Regression testing includes functional testing. In practice, most functional tests become regression tests over time. The big difference is the timing and objective of these tests.

Functional testing is typically performed during development when new features are being developed, implemented, and tested. It focuses on specific functions, features, or use cases of the software to verify if they behave as expected. It may involve various testing techniques like unit testing, API testing, UI testing, and others.

Regression testing, on the other hand, is performed during development as part of the CI pipeline and/or after important development milestones, such as at the end of a sprint or a release cycle. Regression tests are automated as much as possible so that they can be repeatable and efficient, and they include both functional and nonfunctional tests. More on that below.

Functional Regression Testing Tools

Regression testing benefits significantly from automation, and thus tools designed to support the practice also support the automation of all types of testing. These tools help automate the execution of test cases that verify software functionality, collate and analyze results, and report on any regression issues. Here are some commonly used classes of software automation tools for functional regression testing.

• User interface testing. These tools automate the testing of user interfaces based on a record and playback paradigm. Testers record use cases or user workflows and the tools play back the same actions in the user interface. Selenium, for example, is a popular open source automation tool widely used for web application testing. It supports multiple programming languages and provides a framework for automating browser interactions.
• API testing. Test automation at the API level requires test cases and harnesses to run them that interact directly with the underlying application APIs. Testing at this level has advantages over manual and automated UI testing because it’s more resilient to UI changes and tests the business logic more directly.
• Unit testing. Unit testing is done at the lowest software component, which is usually a method in C++ or Java or a function in C. Unit tests work by individually testing code functions and/or procedures in a source file for safety, security, and robustness. As unit tests are executed, output values from the functions are collected and inspected for correctness, and reports may be stored for audit or compliance purposes. Many development teams also measure code coverage to expose code that has not been tested. Knowing that each individual unit of code has been tested and is sound eliminates risk and helps ensure the delivery of a quality application.
• Test data management. Appropriate test data must be provided to applications under test during the testing process so that the test cases can validate application functionality, including common or critical user paths and corner cases. When realistic test data is used, it also helps to replicate error conditions and defects. Test data management (TDM) provides a way to create and manage safe and appropriate datasets that your company can use across multiple teams for validating an application’s functionality and performance.
• Reporting and analytics. After executing regression tests, results are analyzed to provide both status and actionable findings that can be viewed on the web, desktop, or in static HTML reports. Ideally, analytics are presented in easy-to-use, customizable dashboards that are made up of highly flexible widgets that clearly display the risks within your codebase and provide a comprehensive view of project status and quality.

These types of tools mentioned are a subset of the many different tools that may be needed to fully validate an application. It’s critical these tools work together seamlessly in order to facilitate the CI/CD pipeline.

Achieve Velocity While Safeguarding Your Application Against Regressions

CI is a well-understood, well-adopted practice and a necessary first step to significantly improve the velocity of application delivery. It allows developers to push their changes directly into a master branch of source code. A single developer can potentially push many changes to the master branch throughout a single day. To ensure that the master branch is pristine, buildable, and production-grade, it’s critical to have appropriate test cases for every change, since it serves as the golden copy of source code for the application.

Let’s dive into how to configure Parasoft SOAtest to execute functional testing and regression testing as part of a continuous integration process. We’ll go over the steps to configure Parasoft SOAtest with Jenkins, a popular automation platform using the open source Parabank demo application and deploy it using Docker to keep things simple.

How Will This Work?

The diagram below shows the workflow described in this article. Follow the steps from left to right.

1. A developer or a tester will check their test cases into a source control repository. In this example we will use GitHub.
2. Jenkins will check out the repository from GitHub, which contains the SOAtest project called “Parabank” that contains our functional test cases. Jenkins will also pull a Docker image called parasoft/parabank from Docker Hub. This image contains not only Parabank, but also the application server and dependencies needed to run the application.
3. Jenkins will spin up a Parabank image (called a “container”) and configure the application under test (AUT) to the specifics of the test environment.
4. Jenkins will then trigger Parasoft SOAtest to run functional tests pulled from GitHub so that we can validate our Parabank application and implement any needed bug fixes.

Admittedly, this isn’t quite in the spirit of true continuous integration since we’re starting with a pre-built Docker image. But this saves us the trouble of building Parabank with Maven and installing and configuring Tomcat/Java.

The diagram below shows a more realistic, real-world diagram of CI. As soon as a developer checks source code into GitHub, we want to run necessary regression tests and any new functional test cases to verify that the application is still in a good state even after the developer’s changes.

A source code change in GitHub automatically triggers a build in Jenkins, and Jenkins kicks off a Maven build stage, which executes any static code analysis and JUnit tests and builds our application. If all unit tests pass, then the packaged application, parabank.war, is deployed to an application server—in this case, Tomcat—as a part of the Deploy stage. SOAtest then kicks off the Test stage and runs regression tests and functional tests.

Only after the unit tests and black box test cases pass during SOAtest execution are the developer’s original changes considered good.

Let’s go over the prerequisites and steps to configure the process in the first diagram.

Configuring SOAtest & Jenkins

Prerequisites

• A machine capable of running Docker. This tutorial is using Windows so some of the commands in this article may have different syntax. This machine must be connected to the Internet.
• Create a localsettings.properties file somewhere on your machine. Copy the contents of this sample file into it. If you have a node locked license, add it to line 5. If you use a license server, set line 6 to true and add your license server hostname on line 3. We will need the path to this file later.
• Jenkins 2.387.1 or newer installed on the same machine.
• Parasoft SOAtest 2022.1 or newer installed and on the PATH to invoke soatestcli from any directory.
• Docker installed and on the PATH of the same machine.
• Git installed.

Steps

1. Log into Jenkins in your web browser. Jenkins is typically deployed at a URL like http://<JENKINS_HOST_IP>:8080/jenkins.

2. We’re going to start by installing some Jenkins plugins. Select Manage Jenkins on the left and then Manage Plugins in the new menu that appears.

3. Under the Available tab, select and install the Parasoft Findings plugin.
4. Select Install without Restart and then, on the Installation page, check the box, Restart Jenkins when installation is complete, and no jobs are running.
5. Return to the main Jenkins menu from step 1. On the left, select New Item.

6. Provide the name Parabank Deploy and Test and select Freestyle project and then OK.

7. In the configuration menu that appears, scroll down to Source Code Management and select Git. Add this URL to the Repo URL field: https://github.com/sdebrosse/soatest-automation-example.git. All other fields can be left with their default values.

8. Scroll to the bottom of the page and under Build add an Execute Windows batch command build step. If you’re using Linux, select Execute shell, instead.

9. Copy the contents of the script here and paste it into the new build step field. You need to change the values of the two variables at the top of the script to reflect the path to your own localsettings.properties file and where you want a temporary workspace to be created. SOAtest will create this workspace during the testing process. Comments in the script explain what is happening on each line.

10. That’s it. Now, we’re ready to execute our Jenkins job! Make sure to first close all open instances of SOAtest. Then select Save at the bottom of the configuration menu and click Build Now on the left.

11. You can click on the running job on the left and then view the live console output.

12. The logs will output SUCCESS status at the end of your job if everything worked correctly. This means you’ve successfully pulled a SOAtest test project from GitHub, deployed a Docker container with Parabank, and executed tests against this Parabank application. At the end of the process, we automatically killed the Parabank container and deleted our temp_workspace to clean up the build environment. But you may have noticed from looking at the logs that we had some test failures. Yes, a few of our tests against Parabank failed.

If we want our Jenkins build to fail due to SOAtest test failures, we add the -fail flag when we invoke soatestcli. Like this:

soatestcli.exe -fail -data %TEMP_WORKSPACE_PATH% -resource /Parabank -config “builtin://Demo Configuration” -localsettings %LOCALSETTINGS_PATH% 

If you open the test in the SOAtest Desktop UI, you’ll find out that this test failure is primarily a test data and test environment configuration issue. The Loan Processor Service declined a loan that it should have approved.

Summary

Implementing continuous integration into your software development process significantly increases the speed of application delivery. Automated solutions like SOAtest offer a scalable and efficient regression testing strategy with the ability to execute test cases that verify software functionality, collate and analyze results, and report any regression issues. The combination helps teams reach a greater release velocity, improve testing productivity, and ensure high test coverage.