Leveraging Automation to Create High-Coverage API Test Suites

Application Programming Interfaces (APIs) are the service layer of applications. They connect the data layer and the presentation layer (UI)—and drive how users interact with applications.

If the APIs don’t respond correctly to user activity, the application is broken. It’s up to the developers and testers to ensure that connected APIs work properly with their application.

API testing is integral for identifying defects at multiple layers of your application and ensuring a seamless customer experience. Types of API testing include:

• Service tests
• Contract tests
• Component tests
• Scenario tests
• Load/performance tests
• Security tests
• Omnichannel tests

Performing API testing is challenging because it requires a technical approach to design the test cases against the various message formats and protocols. It also requires an understanding of the organization’s business rules to properly use the correct APIs together. Trying to test APIs using a manual approach is time-consuming and may cause testing to be put off until late in the development cycle.

Automated API testing gives you the power to bulletproof your applications against the unknown. It exposes errors early, evaluates the strength of the build, and leads to faster software releases. Since API tests can be automated and run continuously, you can ensure that your application is aligned to business expectations while also verifying functional accuracy.

The Pros and Cons of Automated API Testing

With the right API testing tools and processes in place, your team will reap all the benefits of automation. Without them, API testing can quickly become overwhelming.

Pros

• Provides maintainable automation of an end-to-end scenario.
• Opens communication channels between dev and test.
• Reduces time to diagnose and fix.

Cons

• Testers don’t know how to test the API or how the APIs are used.
• Requires specialized skills and tools to get comprehensive test coverage.
• Nobody wants to do it.

You can overcome these challenges by incorporating advanced testing technology into your existing testing. First, let’s look at API test creation.

The Challenge of Creating API Tests

One of the biggest challenges with establishing an API test automation practice is to create API tests. A roadblock to getting started—or expanding an existing API test suite—is coming up with appropriate tests.

Part of this issue is the required domain knowledge needed to write API tests based purely on technical knowledge or documented API specifications if they even exist. This isn’t the usual domain of testers. Most developers with the required knowledge aren’t involved in testing at this level. They have priorities elsewhere.

Consider the two complementary approaches:

• Top-down approach. Test scenarios are created from regular application usage and testing.
• Bottom-up approach. Tests are based on the intent of the design.

Combining Top-Down & Bottom-Up Approaches to API Test Creation

Most applications provide and use multiple APIs. Some are well-known. Some might even be documented. There are often a lot of moving parts under the hood that make getting started with API testing challenging.

In the top-down approach, API interactions are discovered via tool automation as part of the usual usage and integration of tests.

The other angle of attack, a bottom-up approach, is to create API tests based on design intent. It’s typically part of component testing and assisted by tools.

Combining the top-down (integration testing) and bottom-up (component testing) approaches overcomes the barrier to test creation and increases API test coverage. See the illustration below.

API Test Creation by Usage

One of the best ways to capture API test scenarios is to use the existing usage of the product both in production and test. Leveraging intelligent records and playback, it’s possible to create reusable tests and scenarios.

Using tools such as Parasoft Recorder, you can capture and model the API interactions that are happening underneath the hood of the application. Then SOAtest’s Smart API Test Generator uses artificial intelligence to organize complex traffic into patterns of API usage. This makes it possible to create a suite of API tests using web UI tests to extract, filter, and create API test scenarios.

SOAtest then uses AI to identify patterns and establish data relationships to create the API test template. You can modify and reuse the template with variations. These tests can be leveraged for security and performance testing. More importantly, once created, these tests are decoupled from the UI and the underlying application, and its dependencies can be exercised from the API tests alone.

Recording and analysis of API traffic into test scenarios extend to automated UI testing with tools such as Selenium. Any use of the application that initiates API calls, whether manual or automated can be used to create API tests. It’s also possible to link recorded scenarios to requirements for traceability.

Once recorded, these scenarios can be investigated and modified within SOAtest as shown below.

SOAtest detects the various API calls and the data interaction occurring. It’s then possible to modify the scenario and data payloads as needed. Once patterns are understood, assertions can be inserted into the scenario to verify correct values and behaviors.

The ability to create a ready-to-use suite of API tests based solely on existing testing techniques results in quick creation of tests with the ability to reuse and extend. It’s a shortcut through one of the biggest roadblocks to API testing.

API Test Creation by (Service) Design

You might be thinking, “We don’t know the design of the APIs. Most of them aren’t documented!” In most cases, this is correct. However, it’s still possible to piece together API tests from existing information to help extend API test coverage from the bottom up—by looking at the service definitions and contracts that are explicit or implicit in the application.

In terms of an API specification, most services have a definition in a Swagger document or WSDL. SOAtest can read these definitions and create an API test scenario template. It includes a test case for all the clients in the definition. You can edit and manipulate the template to create meaningful API tests.

SOAtest provides many options for creating scriptless tests from the template, including parameterized data based on random numbers or previous test results as shown in the example below.

Using simple copy and paste techniques, scriptless test creation, along with flexible test data management, SOAtest provides an easy way to build up suites of tests. It’s not meant to replace the top-down approach but rather complement it. Where manual or UI testing is missing important parts of an API (informed by coverage results), the bottom-up approach fills in these gaps. However, it’s difficult to determine what to test and what’s missing without an understanding of coverage.

Understanding Your Test Coverage

Coverage means several things depending on the context. Typically, it’s code, API, and requirements coverage. You need to understand all of these to get a clear picture of how complete your testing efforts are.

• Unit testing. The concern is code coverage. How much of the code has a test associated with its execution?
• APIs. It’s important to know which ones are tested and to what extent each API has been exercised.
• Requirements. Each requirement must have a test proving that it is implemented correctly.

These are overlapping concerns that matter at different times during the project lifecycle. For example, if API test coverage is what you’re tracking, it’s important to make sure that the testing is adequately covering the APIs and augment existing tests where it’s lacking. The best way to see this is from the service definition, as shown in the image below.

SOAtest keeps track of each test executed and what APIs are covered: fully, partially, or not at all. Knowing where tests are missing lets us direct our efforts to a bottom-up test based on the service definition—or possibly a new UI-based test, if appropriate. It also points out where tests are failing (in red), directing our attention to tests that either need updates or something is wrong in the implantation.

Parasoft DTP (Development Testing Platform) acts as a central repository and hub for results from various testing practices and provides a high-level view of where the project stands in terms of the different types of coverage as shown in the following dashboard.

This view incorporates aggregated results from static code analysis, unit testing, API, and manual and automated UI testing. It’s the complete gamut of the testing pyramid. It provides a complete picture of coverage from all aspects: API, requirements, and code coverage.

Consider requirements coverage in terms of user stories saved in JIRA. The functional testing dashboard (shown below) includes a JIRA Story Status widget, which is one way to track requirements coverage.

Looking at the JIRA status more closely, the mouse-over information indicates the number of tests passed, failed, incomplete, or missing with no tests.

You can drill down further in the detailed traceability report to find out which requirements are missing, failing their tests, and so on.

You can investigate further by drilling down into the traceability report to understand which tests are touching this story.

Additionally, you can link this coverage to APIs and code coverage as needed. You can navigate to API tests from here to determine where changes are needed and what additional tests are required. This holistic, aggregated view of coverage provides the most complete picture of testing status. Test coverage “grows” over time as developers converge on a completed application, and the new code is “covered” by API and unit tests created for associated requirements.

Summary

The importance of API testing is recognized by software development organizations, but they often struggle with adopting the practice fully. Part of the struggle is building up a set of tests because manual test creation is complex and requires intimate technical knowledge. Even teams that adopt API testing are challenged to increase coverage of their API tests because so many of them are undocumented.

Advanced functional test tools like SOAtest can help automate API test creation using a top-down approach based on analyzing in-use behavior from manual testing and regular application usage. This smart test generation helps teams create tests from their existing practices quickly and easily.

A simultaneous and complementary bottom-up approach is possible using SOAtest automation to build scriptless tests from service definitions. Along with complete coverage analysis, you can grow API test coverage over time using both top-down and bottom-up techniques. Since API tests work at a much lower level than UI tests, you can have confidence that the consistent and comprehensive tests you’re building will last for a long time to come.