Continuous Testing: 4 Strategies for Enterprises

4 Continuous Testing Strategies Your Enterprise Should Start Now to Avoid a Meltdown

The holiday season is a busy and hectic time for travel and hospitality services. They experience a surge in demand and often face challenges such as bad weather and technical issues. To ensure a smooth and successful holiday season, it’s important that development teams are prepared for the unexpected and have systems in place to quickly recover from any disruptions.

When Snow Happens: Continuous Integration & Continuous Delivery

The 2022 holiday season was eventful with severe weather impacting many travel plans and affecting airline operations. Whether you were stuck in the snow, at an airport, or with family, we hope you have made it safely home. This chaos has served as a reminder of just how important it is to invest in continuous quality practices as well as operations modernization for enterprises that serve fundamental functions in our society.

In December 2022, a major U.S. airline lost a large chunk of its operations during the busiest travel time of the year. Bad weather, the pandemic, and various other factors played significant roles. But even after the weather situation cleared up, the carrier continued to cancel 50% or more of their flights each day, for several days, due to software issues.

The culprit was an automated crew scheduling system that buckled under the duress of the holiday spirit and prompted the airline to switch to manual crew scheduling. Crew management and routing is a very complex topic that has been iterated over various solutions over the last 50 years. Every year, the problem continues to evolve, as do the solutions. Here’s the general gist.

• Crews and flights get paired for a few days and return to their initial locations. Additionally, crew and flight pairings are heavily regulated worldwide.
• Between various routes, local regulations, scheduling constraints, and a variety of unknown conditions, assigning crews to flights is not an easy process. This essentially results in a large and complex computational problem that needs to be automated for efficiency.
• The industry has come a long way to address this. Enterprise crew management solutions like Sabre, Jeppesen, AIMS, and others, offer cloud-based, SaaS solutions that can scale to meet the demands of worldwide travel.

Continuously modernizing infrastructure costs money and can even be risky. But falling behind has a cost associated with it as well. In all, over 15 thousand flights were canceled in the span of a week. To put this in perspective, when this airline canceled around 2,500 flights in 2021, it was estimated to cost the company approximately $75 million.

What Can Development Teams Do to Prepare for Extreme Loads?

Only one airline was impacted severely, but the outcome is an eye-opening moment for all enterprises. Even outside the travel industry, the beginning of the year is a good time to review business and software quality strategies with the goal to prepare for peak loads and outages.

The big question is, how do you truly prepare for something to go wrong?

We’ve put together a list of four software testing strategies your enterprise can start doing now to avoid a full-scale meltdown.

1. Implement continuous load testing in your test automation.
2. Use service virtualization to solve the test data problem.
3. Implement chaos engineering in your DevOps.
4. Invest in scalability, modernization, and quality.

1. Implement Continuous Load Testing in Your Test Automation

As you plan to modernize your operations, you may want to break up the work into incremental phases and establish continuous testing practices to detect bugs in their earliest stages.

Performance testing in the form of load testing, scalability, and stress testing is already a core quality practice, but for many teams, it’s largely limited to manual ad hoc tests that get “kicked off” when the system and environment are configured for testing. The goal is to get as close to production levels as possible, but it all also means a significantly longer feedback loop.

Continuous load testing is a type of software development testing that involves subjecting a computer system, network, or web application to a continuous stream of load or traffic to determine how it performs under sustained use. The purpose of continuous load testing is to identify and fix performance issues early before they balloon and impact users or customers.

Organizations that have moved to a cloud infrastructure are less susceptible to issues in performance because they can spin up additional resources on demand. However, when a deep-rooted problem is identified in the late stages of the software development life cycle (SDLC), there are tough and expensive decisions to make with regard to fixing them. These decisions don’t always go the way of rework.

Continuous load and performance testing are often used together to continuously monitor and improve the performance and stability of a system or application. This can involve automating the testing process and integrating it into the development and deployment pipelines so that the system or application is tested at various stages of its life cycle.

It may also require test automation engineers to inherit some of the responsibility of constructing and maintaining load test scenarios. It’s important to choose tools that support the emergent needs to shift-left performance testing without fatiguing your resources. By continuously testing and monitoring the performance of a system or application, organizations can ensure that their systems are able to meet the demands of users and customers.

2. Use Service Virtualization to Solve the Test Data Problem

There’s no better test data than fresh production data. For testing, traditional test data management (TDM) solutions focus on cloning production data and massaging it to be reused for testing.

Although this is an efficient approach to obtaining realistic data, it does leave quite a few blind spots and comes with an expiration date. Additionally, you have to consider the volume, privacy, and complexity as you scale tests to full-on performance tests and implement complex test scenarios.

Service virtualization is a technique that allows you to simulate the behavior of dependent systems or components during testing. You can create virtual versions of the assets and the relevant data. This can be useful in addressing the “test data problem” because it allows you to test the performance of your system or application without having to rely on real data from external systems.

Developers and testers can use service virtualization to create a replica of the dependent systems or components on which the system or application relies. These virtual versions can mimic the behavior and response of the real systems or components, allowing you to test your system or application in a controlled environment without the need for real data.

Service virtualization can help address the test data problem in the following ways.

1. Reduce the need for large volumes of test data. Using virtual versions of dependent systems or components enables you to test your system or application with a smaller volume of test data.
2. Protect data privacy. Service virtualization allows you to test your system or application without exposing real data, protecting the privacy of users and customers.
3. Simplify data complexity. With virtual assets, you can simplify the complexity of test data and focus on testing the performance of your system or application.
4. Improve data maintenance. Quickly and easily update the behavior of your virtual systems or components so it’s easier to maintain and update the test data as needed.
5. Reduce data storage requirements. By reducing the amount of test data that you store and manage, you can minimize storage and infrastructure requirements.

Service virtualization is a flexible solution that development teams can use in all facets of DevOps and test automation. Watch Ryan Papineau of Alaska Airlines share an in-depth look at how major U.S. carriers use service virtualization with TDM solutions to test crew management systems.

3. Implement Chaos Engineering in Your DevOps

Another approach that can help enterprises prepare for unplanned issues is chaos engineering. Chaos engineering is the practice of intentionally introducing failures or disruptions into a system to test and improve its resilience and reliability. By simulating various failure scenarios, airlines can identify and fix vulnerabilities in their systems before they become major problems during the busy holiday season.

Chaos engineering has been steadily on the rise since Netflix pioneered it. It aims to continue to be a top trend in DevOps.

To implement chaos engineering, a team can develop a set of hypotheses about the potential failure modes that could occur during high-load operations. Development teams can then use chaos engineering to test their systems for the following.

• Capacity. Can the systems handle the increased volume of traffic during peak usage?
• Scalability. Can the systems scale up or down as needed to meet changing demand?
• Resilience. Can the systems recover quickly from failures or disruptions?

There are a number of tools, such as Chaos Monkey, Gremlin, and service virtualization, that simulate these failures in controlled environments like staging and testing. By monitoring the effects of the simulated failures on the system, teams can identify any issues and make adjustments to improve the system’s resilience.

4. Invest in Scalability, Modernization, & Quality

Hindsight is 20/20, but increased traffic and demand during peak periods can be difficult to predict. Being proactive may seem like a luxury, but it will go a long way to help you avoid technical debt and future-proof your software and systems. Continuous modernization and implementation of quality practices mean a significant investment in operation from a technological standpoint.

Scalability refers to the ability of a system or application to handle increased load or traffic without a decrease in performance. By investing in scalability, enterprises can ensure that their software systems can handle the increased traffic and demand that typically comes with the holiday season. This can involve things like improving the performance and capacity of their systems, as well as implementing load balancing and other techniques to distribute traffic across multiple servers or resources.

Modernization refers to the process of updating and improving the technology and infrastructure of an enterprise. By modernizing their systems and processes, enterprises can improve the efficiency, reliability, and scalability of their critical software, which can help them better handle any increased traffic and demand.

Testing is also essential for preparing for high-stress conditions. By conducting load testing, stress testing, and other types of performance testing, enterprises can identify and fix any potential issues with their systems before they impact users or customers.

Overall, investing in these areas can help enterprises prepare for unexpected situations by enabling them to handle increased traffic and demand in a reliable and efficient manner.

Conclusion

Good business and software development practices can play a crucial role in helping enterprises prepare for the holiday season and other high-load circumstances. By focusing on scalability, modernization, and testing, development teams can ensure that their systems and applications are able to handle the increased traffic and demand that may occur.
Investing and implementing these practices can help organizations improve the performance and reliability of their systems, reduce the risk of outages or downtime, and provide better customer experiences. Enterprises that ensure that they are well-prepared to handle the occasional chaos deliver a seamless experience to their users.