Conquer the Challenges of Performance Testing

Software development organizations are under pressure to provide excellent user experience while balancing development and maintenance costs. For a server software application, such as a web application or an API, consistently fast and reliable responses are as important as functional correctness to ensure user satisfaction and application success.

Performance testing is the area of software quality assurance that focuses on the responsiveness and reliability of an application when is subjected to a stream of requests from multiple users. It evaluates how an application behaves under expected service request loads and analyzes the results so that bottlenecks and other inefficiencies that prevent smooth operation under load can be identified and resolved.

• Performance testing tools can help ensure that your system will handle:
• Regular load of service requests and
• Variations in request types and traffic patterns.

However, even the best tools aren’t enough if they aren’t applied in the context of the right performance testing strategy and practice. Only the combination of the two can ensure success.

Problems of the Traditional Approach

Performance testing has traditionally relied on a nearly functional application running in a production or preproduction environment. By implication, this is late in the development life cycle to find out about significant issues. Problems found this late have a big impact on cost and schedule.

In addition, the time and effort required to complete the full cycle of manual performance tests limits the frequency with which it can be done, thus either limiting the frequency of release cycles or making organizations release with partial or no performance testing.

A search for solutions that would reduce the uncertainty and risk caused by the described above problems pushed organizations to start applying performance testing to Agile and shift-left strategies already being used and proven to be effective in other areas of software testing.

Challenges of Modern Performance Testing

Shift-left performance testing states that the problems of the traditional approach come from testing too late and too infrequently. Therefore, you need to test early and often.

This sounds good in theory, but how do you implement it in practice?

For the shift-left testing promises to materialize you need to resolve the following set of problems:

• How to performance test an application that doesn’t exist yet?
• How to find the balance between agility and costs?
• How to make performance test automation pay off?
• How to reduce setup and operating costs?

Problem 1: How to Performance Test an Application That Doesn’t Exist Yet?

The requirement to start early means that performance tests should be created along with unit and functional tests, which can be long before the final application takes shape. These two major methods allow you to start performance testing a server application before it’s fully functional.

• Service virtualization
• Unit-level performance testing

Service virtualization allows testing applications early by emulating the behavior of their external dependencies—like APIs, databases, messaging systems, and more—that may not be available in the initial stages of development for one of the following reasons:

1. They’re being worked on in parallel with the application under test (AUT).
2. Access to them is limited.

A service virtualization tool should allow you to mimic the responses of external dependencies and their performance parameters, such as response delays, which will affect the performance of the AUT in a realistic way.

Unit-level performance testing allows you to evaluate third-party and in-house components that you’re planning to integrate into your application. For example, you can evaluate performance of alternative JSON parser libraries for the request sizes and load levels you expect of your target application. This will help you choose the best alternative and set realistic performance expectations for your application based on the performance of the components it uses.

Problem 2: Agility vs. Costs. How to Find Balance?

The requirement to test often may suggest that performance tests should be run as often as unit or functional tests in a continuous integration (CI) build process triggered by source code check-ins.

While performance testing must be an integral part of continuous application delivery, applying the same test frequency logic that works for unit or functional tests may not be practical because running the full suite of performance tests typically requires significantly more time and computing resources.

To solve this problem the application performance test suite should consist of different performance test types, whose execution frequency is inverse proportional to the time and resources required for running these tests. With such an approach, relatively short smoke or baseline performance tests can be run as a part of the CI build process, while more comprehensive tests are executed regularly, but less frequently.

Problem 3: How to Make Performance Test Automation Pay Off?

Automated execution of performance tests by itself will have little value without the automation of tests results analysis. A performance test can generate a lot of data. As the application development moves forward, the number of performance tests, the complexity of their scenarios and their duration will grow. Continuous execution of these tests creates a firehose of data that needs to be reduced to a pass/fail answer. This answer typically comes as a result of service level agreement (SLA) checks that are automatically applied to the performance test data collected at the completion of the test.

Creating a comprehensive set of stable SLA checks for every performance test is key for successful performance tests automation.

To get the full benefit of continuous test execution, the performance test results should automatically be published to a reporting and analytics dashboard so you can quickly understand trending data. The shift-left approach adds developers as dashboard users, in addition to managers and testers. So the dashboard has to have the low-level details that the developers are looking for to effectively investigate and establish the causes for SLA failures or historic trends.

Test automation does not have to stop at the stage of performance test execution and pass/fail analysis. The next level is automated root cause analysis of performance tests.

4. How to Reduce Setup and Operating Costs?

The benefits of shift-left automated performance testing come at a cost of building the test automation infrastructure, creating or modifying tests, and potentially acquiring and learning new tools and cultural changes.

In this journey, you need to make sure the performance testing tools you use fit the new principles that you’re putting into practice. Below is a list of performance test tool features that will save you time in setting up and maintaining automated tests.

• Offers an extensive command line interface for automated performance test execution.
• Allows reuse of testing resources, such as functional test assets, performance monitors, SLA checks, and so on.
• Can be used in cloud and closed computing environments.
• Provides functionality for bulk updates of performance test projects.
• Supports automated test case generation.
• Supports automated failure root cause analysis.
• Makes common things easy and advanced things possible.

In practice, this means that while offering GUI for common performance testing tasks the tool provides an option to extend its functionality with scripting in all major functional areas. A GUI interface for common tasks will provide productivity and fast learning curve, while extensibility with scripting or programming will assure that no matter how specific your automated testing requirements are the tool should provide means to meet them.

Test stability and maintainability contribute a lot to reducing performance testing operational costs. While this subject deserves a separate article, we shall mention one area that is often missed if one looks at performance testing of web applications from a traditional perspective. Modern web applications use API calls to a great extent, the trend is to exclusively use API calls to fetch dynamic content from servers.

This growing reliance of web applications on API calls drives a qualitative shift in modern performance testing. When a web application’s static content is served by highly available content delivery networks (CDNs), most of the web page loading time comes from the API calls. The performance of such web applications becomes a function of the performance of the APIs it depends on, which justifies the replacement of UI with API performance tests.

Such replacement brings multiple advantages.

• More stable API tests
• Significantly fewer computing resources are required to execute
• Easy to reuse from existing functional tests

Replacing UI with API performance tests for qualifying web applications can greatly contribute to your performance test stability and reduce operational costs.

Where to Go Next

Modern performance testing introduces new principles driven by test automation. The inefficiencies of the traditional testing process are discarded, but the foundations remain. And it’s important to revisit and efficiently apply new practices.