What Is an “Environment-Based” Approach to Software Testing?

Testing applications and services in isolation only gets you so far, and eventually, you need to test in an operational environment. Parasoft’s “environment-based” approach to testing enables teams to test their applications in the context of all of the dependencies that exist in the real world environment.

Often testers and developers are looking at their deployment environment and its components on an individual basis, testing those pieces in isolation. This works to a certain extent but doesn’t test enough of the system. Another version of this is testing at the UI level, at a point where root causes of failures are hard to diagnose. The middle ground is where applications can be tested in as realistic an environment as possible using virtual services to mimic dependencies that aren’t available or consistent enough for testing. As more dependencies come online, you can move from virtualized services to live versions.

At Parasoft, we call this an “environment-based” approach to testing, which enables the whole team to test their application with full context of the real-world environment. Tests are deployed into a live, partially virtualized or fully virtualized environment as needed. The environments-based approach to testing provides a stabilized platform for test execution.

Environment-Based Testing

The environment-based approach to testing provides more context to better understand how dependencies impact the application during UI testing and helps the team go the next-level deep by creating API and database tests for the dependent components. By doing so, you can achieve more complete test coverage for your application and decouple testing from the UI, so both can be run together or independently. Test failures are highlighted within the environment, making pinpointing failures easier.

Some additional benefits that come with an environment-based approach to testing include:

• Shifting testing left by testing the application in the production environment as soon as possible. This is made possible with service virtualization, creating virtual services, and then transferring to live services as they become both available and stable.
• Reducing the time and effort of diagnosing test failures by pinpointing exactly where and why tests have failed.
• Easily extending testing from UI testing to API testing, which is configurable in context of the environment, as well as part of a CI/CD pipeline.
• Increasing test coverage by creating scenarios that wouldn’t be able to be implemented from UI testing alone.

These benefits reduce the overall burden on testers to set up, run, and diagnose application tests while increasing test coverage and allowing testing to be done sooner, and more comprehensively.

Types of Test Environments

As many testers know, a test environment is just a controlled structure that lets systems and software conduct tests for reliability, functionality, and compatibility. These typically simulate real-world scenarios and conditions to address potential bugs or breaches before deployment. As such, each test environment can be tailored to the type of testing and results a team wants.

A cross-browser test environment is a perfect example of a test environment for testing on different browsers and browser versions. Testing teams use the environment to check the quality and impact of the application before users interact with it.

There are several key types of testing environments including:

• Performance. Test environments for performance verify that a system performs as expected and meets performance goals. These can include response time, concurrency, stability, and more. This type of testing can be complex due to the amount of variation in systems (RAM size, numbers of users, CPU cores, and so on) and requires careful setup.
• Security. This type of environment prioritizes software security. It will test for vulnerabilities regarding authentication, authorization, confidentiality, and integrity.
• Integration. This environment tests software module integration to verify or monitor behavior. This can implement one or many modules for functional testing.
• Chaos. This targeted environment seeks to identify specific areas that pose failure threats and address them before the end users encounter them.

Test Environment Setup: Best Practices and Considerations

Creating a good test environment isn’t just about using realistic scenarios. It involves everything from considering edge cases to protecting data security to emulating versus utilizing real information. Though every team will have different goals and requirements, certain best practices can be followed to ensure the best test environment for your project.

What Is a Good Testing Environment?

A good test environment resembles the live production environment in which the software or system will function very closely. It must use the same code, operating system, data, functionality, and configuration.

The key factors in creating a good test environment include the following:

• Data. Utilize quality, realistic data that also includes edge cases to test that the software or system performs as expected.
• Design. Include a sample representative of the application or the entire application for testing.
• Automation. Leverage automation to relieve certain pressures, empower certain processes, and retrieve actionable data.
• Security. Ensure the safety of the test and its data.
• Management. Consider that continual testing must go on in the long term and verify that the test environment is set up for that.

Test Environment Best Practices

In addition to the above mechanical tenets, test environments rely on general intangible guidelines, too.

1. Follow a software testing methodology.
2. Treat testing as a part of the deployment process.
3. Integrate communication into the test environment.
4. Recycle or reuse testing resources and environments.
5. Test early, often, and continuously.
6. Adopt a feature management mindset so you can test individual features instead of the entire codebase.

Test Environment Management

Managing your test environments can be a huge headache or prove to be an unskippable part of your SDLC that provides invaluable information and data. The key is in how you approach that management. By front loading some of the work and organization, you can create an easier to maintain setup and still reap the benefits of frequent testing.

Rely on Trust and Transparency

From the most senior developer to the freshest QA tester, treat the people on development teams as important members of the project. Utilizing a single source, or a golden copy, for environments and deployments that is accessible to everyone, within reason, fosters an environment where communication is open, encouraged, and constructive.

Document and Track the Process

Managing the knowledge base—whether on Confluence, JIRA, or other tools—helps prevent miscommunication and reduces games of telephone. Team members can reference the documentation for basic information, tracking changes, seeing who worked on what, and more.

Make Information Available

In tandem with the above tenets, making this information available to stakeholders further helps maintain test environments. But this applies to things like deployment schedules, outage plans, and other aspects of development, as well.

Avoid Manual Updates

Humans make mistakes. It’s a fact. Manual updates succumb to this reality, so incorporating test environment management with the toolchain you use can help cut down on manual mistakes.

Leverage Virtualization

Being able to interact with the data your test cases generate is a must. Being able to do so in a way that is streamlined and easy is even better. Virtualization lets you take snapshots, replicate your environment end-to-end, and more. In fact, Parasoft has its own industry-leading service virtualization solution just for this purpose.

Review Data and Performance

This might not seem like it needs to be mentioned but tracking KPIs such as the number of unplanned service disruptions, environment uptime, and other variables can help you improve your test environments. Review the data after significant test cycles to see what works, what needs changes, and how resources are allocated or could be different.

Environment-Based Testing in Action With Parasoft SOAtest

So, let’s look at this in action. Parasoft SOAtest supplies an environment manager that visually displays the execution environment for the application under test. Shown below is an example of an application in its environment, connected to dependencies. Each component in the environment can be provisioned as live or virtualized as needed, depending on the stability and availability of those components.

Deployment presets configure the environment based on current test requirements. Different scenario categories may require different presets. These presets contain all the needed settings for the Environment Manager:

Once a test environment is provisioned, a set of API and UI interactions are executed against the application under test. Results are captured, recorded, and compared to expected results within the IDE and in the Parasoft web portal, where the Environment Manager utility acts as the hub for functional testing.

Failures are highlighted in the Environment Manager and it’s possible to follow error reports from there into a more detailed report.

The API test reports that are generated indicate both failures and test coverage information. Failures mean that new functionality is broken, or that there’s an issue with a test. The root cause of these failures can be traced from the failure here to the appropriate API. Test coverage information is important because it indicates missing tests, and potentially shipping untested code.

The Role of Service Virtualization

Errors also arise from missing dependencies. This is where service virtualization comes into play. It’s often not possible to have live dependencies available for testing, perhaps because they aren’t ready yet or it’s difficult and time-consuming to duplicate from the production environment. But to assure high-quality applications within schedule and budget constraints, it’s essential to have unrestrained access to a trustworthy and realistic test environment, which includes the application under test and all of its dependent components like APIs, third-party services, databases, applications, and other endpoints.

Service virtualization enables software teams to get access to a complete test environment, including all critical dependent system components, as well as alter the behavior of those dependent components in ways that would be impossible with a staged test environment, enabling you to test earlier, faster, and more completely. It also allows you to isolate different layers of the application for debugging and performance testing, but we’re not going to get into that as much here.

Individual services can be configured as live or virtualized. Deployment configurations that are combinations of live and virtualized dependencies can be saved as presets.

Integration With CI/CD Pipelines

Manual testing examples are interesting, but the real work happens in a continuous integration pipeline. Parasoft’s dynamic test environments can be deployed from Jenkins as part of a continuous integration/deployment pipeline. The following example shows “Deploy Dynamic Environment” as a Jenkins build.

Using the Parasoft Jenkins plugin, build steps are added to deploy test suites from the build. Results are captured and linked to the Environment Manager.

After build completion, test results are displayed within Jenkins. Failures link directly back to the environment manager and deployment scenario.

Summary

By considering the operational environment first and leveraging an environment-based approach to testing, software testers can make sure that everything is in place to make testing productive and efficient. Rather than spending time trying to test applications in isolation or with a pseudo-realistic environment, testers can spend more time on the actual testing itself.