A Micro-Manifesto on Continuous Testing for DevOps

Everybody talks about DevOps these days, and I think the principles behind it are fairly well understood.

What Does Continuous Testing for DevOps Mean?

The general consensus is that DevOps is “infrastructure as code” — the unification of the development and operations teams through automation.

Another way to say this is it’s a way an application can get all the way from lines of code being written by developers to a fully deployed application. The best way to do that is through automation.

Agile in DevOps

One thing we hear quite often when talking about DevOps is the concept of Agile. Agile is a development methodology where you compress cycles within the development of an application into smaller chunks so that you can more readily focus on delivering incremental value for an application.

In Agile, a project that would traditionally take 12 months to complete is broken up into a series of sprints that typically run one to two weeks.

In the sprints, development goes through the entire process of deciding a set of user stories, developing those user stories, and then delivering those user stories. Over a period of time, you will ultimately deliver the full application. But because you’re breaking it up into the smaller pieces, you can check in at every sprint to ensure you’re building the right thing. This is a critical enabler for DevOps because it gives us checkpoints for automating the delivery of those smaller chunks.

What Is Continuous Testing?

Continuous testing is the process of continuously testing the application. Sounds like a no-brainer, but it comes in two forms.

The first path is continuously running tests against the application and as new builds become available those tests verify that the application is performing as expected. The continuous nature of this testing approach enables you to get immediate feedback every time there are code changes and helps to ensure that something done in a previous build does not affect the application as a whole.

The other way to think about continuous testing is that the testing process itself is continuous. Meaning that as a part of the continuous build continuous integration and continuous delivery pipelines there is also a continuous testing track. This means that test automation is built into the actual build and release pipeline and can serve as a quality gate to ensure that applications are meeting specific specifications before they get automatically deployed via DevOps.

If you think about it, continuous testing is essential for DevOps because it serves as a mechanism to stop the conveyer belt if something is wrong with the application. If you’re taking your builds and immediately pushing them through automation to the delivery process, you could be deploying defects into production. So, this is why I think that the complete picture is DevOps and continuous testing.

What Do You Need to Enable Continuous Testing for DevOps?

There are many technologies that teams can leverage during the automated testing phase of the DevOps pipeline. Individually, each of these testing techniques and processes is quite valuable. The real bang for your buck comes from incorporating each technology and process end to end.

Automated Code Scans

In automated code scans you essentially run spell check on your code. This is often referred to as static code analysis and it happens as soon as the lines of code are written.

The idea is that the tooling will look for patterns in the code known to lead to bad practice or defects down the line. With this approach, you’re preventing, as opposed to detecting, defects before they get introduced into your applications.

Automated Test Execution

Once an application is built, the code is hidden. So the best way to then test your application is by validating user inputs and interfaces. Your targets here are UI testing, API testing, database testing, performance testing, and security testing. You want to be able to run full regression tests across the UIs and to touch each one of the exposed APIs.

These tests are known as functional tests and they will enable you to validate that the applications function as they are expected. There is a whole world of capability here and I personally think it’s a critical enabler of the DevOps process because things tend to get weird when they are integrated with each other. And so, a place to really look here is in API testing.

Requirements Management

When we build an application, we build an association with the requirements. This way we have a framework for building the application, but we also know the business expectations. Testing should follow the same paradigm. You build tests that are associated with the requirements. This helps you not only understand whether you’ve covered enough of the application to assure stakeholders that the application has a sufficient level of risk for release, but also helps you understand which tests to focus on when specific parts of the application are misbehaving. Mapping test cases to requirements becomes a critical component of prioritization and for remediation in test automation.

Change Management

“Continuous” is an effort to achieve greater speed. When we are moving so fast, if something changes and we didn’t expect it, our pipeline could break.

Test automation is an alarm bell to help us ensure that we’re delivering valuable software that’s defect free down the delivery pipeline. If the application changes, the automated software needs to know that and needs to adapt to those changes so that you don’t get false positives or, worse, false negatives. Building a change management process into the DevOps pipeline helps you ensure that the results that come from your automated testing are valid.

Test Data

Access to usable and safe test data is a critical enabler of continuous testing for DevOps. To build a scalable DevOps pipeline, you need to be able to have a combination of realistic and/or synthetically generated test data that can be injected into your test automation so that you have enough data to perform the testing that you want to achieve.

Test Environment Access and Simulation

Earlier I mentioned that functional testing kicks in when the application is built. A key piece of this is the fact that that application is built and deployed into some kind of an environment.

With the rise of service-oriented architectures and microservices, distributed systems are a collection of small applications and endpoints that need to work with each other. To run this test automation, you need to have a stable test environment with all the necessary dependencies. This is where simulation can become a critical enabler for the DevOps pipeline. Through simulation, you can create digital twins of endpoints that are unavailable or do not have the proper test data at the time that you want to run the test automation.

By combining service virtualization with DevOps, you can remove environmental constraints. This enables the organization to gain full access and control over the test environment and even deploy dynamic test environments.

In a dynamic test environment, you are deploying a containerized environment that contains mostly simulated components. Through this configuration, you can focus on the applications that you want to validate and stub out the rest.

You can spin off multiple copies of these dynamic test environments, because if you think about how the actual DevOps pipeline will take place, test runs will be executing as a function of code check-ins. There may be multiple code check-ins at any given moment, and you don’t want people queuing up waiting for a real test environment to be available. Dynamic test environments can be one of the most critical enablers that is required for continuous testing and DevOps.

Continuous Feedback

Everything that we’re doing in continuous testing for DevOps is about getting meaningful results that we can act on. The last piece of the puzzle is continuous feedback.

In continuous feedback, you are taking the results from your automated testing and combining them with other quality related metrics such as requirements coverage, API coverage, and code coverage. You can then evaluate those in an automated way to ensure that they are within defined thresholds for quality acceptance.

If the application is okay, then it is automatically deployed to the next stage. If it isn’t, you can stop the deployment process and kick it back to development associated with the original check-in. This fast and iterative feedback mechanism enables you to truly have confidence in your continuous testing for DevOps.

Summary

This is just a high-level view of what’s required to enable a scalable and reliable test automation strategy for your DevOps pipeline. There’s a lot more information that’s required as you go deeper into each one of these individual topics. The intention of this blog is to provide a condensed, easily consumable, summary.

I encourage you to look at our whitepaper on continuous testing for DevOps. We explore details around ROI, deployments, and key performance indicators to understand whether you are bringing in the right technology at the right time for your DevOps pipeline.