Test the Untestable: Alaska Airlines Solves the Test Environment Dilemma

Watch a preview of the Alaska Airlines presentation at the recent Automated Software Testing & Quality Summit. Learn how they used virtualization to efficiently scale testing for their complex flight operations application.

FULL PRESENTATION AVAILABLE HERE >>

The Challenges

At Alaska Airlines, the flight operations manager software is ultimately responsible for transporting 46 million customers to 115 global destinations via approximately 440,000 flights per year, safely and efficiently. This software coordinates a highly complex set of inputs from systems around the organization to ensure flights are on time while evaluating and managing fuel, cargo, baggage, and passenger requirements.

In addition to the previously mentioned requirements, the system considers many factors including weather, aircraft characteristics, market, and fuel costs. It’s highly complex with numerous dependencies (see Figure 1) and functions as the heart of a well-run airline.

Complex Dependencies & Ever-Changing Test Environments

The complexity and huge number of essential services as dependencies made it nearly impossible for Alaska Airlines to build repeatable tests. In addition, the test environment was shared with other teams and constantly changing. The changes impacted the flight management software and required persistent tracking and debugging. The ongoing resolution of these issues hindered their own development progress.

Multiple Databases Updating at Different Frequencies

Data management was another large issue. It entailed various databases throughout the system updating at different intervals, making test data management a huge challenge. The various services in the system were another key challenge. It was nearly impossible to simulate real world events in a running production system.

The Approach

Given the need to decouple their application under test from a very complex and constantly evolving environment, Alaska Airlines looked to service virtualization as a solution to their testing woes. Alaska Airlines initially selected the Broadcom service virtualization tool but discovered that it was difficult to use and crashed under load in their environment. They turned to Parasoft for a solution that met their needs for ease of use, reliable operation and scalability, and top-notch technical support.

Repeatable, Concise, & Precise Tests

They use Parasoft Virtualize to decouple their application under test from the myriad of dependencies in their complex test environment. Alaska Airlines started using service virtualization as a proxy between their client devices and the airport web services. They ran through various scenarios to capture the requests and responses.

These captured scenarios became the basis of repeatable, concise, and precise tests for later use. However, this was just one aspect of their system operation.

Virtualized Environments for Each Key System

The full application isolation required more virtualization and test data management. Alaska Airline’s virtualization environment included replacing all key systems with their virtualized equivalents as shown in the Figure 3.

Test data management was key to solving the issue with inconsistent databases in their real-world environment. Alaska Airlines decoupled the application from the dependent real-world databases by using Parasoft Virtualize test data management capabilities. They used IBM Optim in conjunction with custom T-SQL data transforms against data captured using Parasoft Virtualize.

Alaska Airlines captured enough data for a full day of real operations. Then deployed it within the test environment to create a “Groundhog Day” set of scenarios — repeating the same events and responses for a complete 24 hour cycle, progressing the time while keeping the scenarios consistent.

Advanced Simulations for Robust Testing

Alaska Airlines’ solution for their passenger services is a point of pride. Using service virtualization, they were able to simulate passengers going through the complete lifecycle —booking, checking in, boarding — for all types of flights in their system.

The pilot’s interaction with dispatch and fuelers is only a couple of the interactions they have with the system flight management system. It’s nearly impossible to coordinate these interactions in real time without impacting production resources and personnel.

The team simulated these complex interactions with dispatch and fuelers. Getting consistent results, and no longer needed to leverage critical personnel to operate the airline.

Results

Alaska Airlines freed themselves from the confines of testing in a complex and volatile real-world environment — to test the untestable. Once their flight management system test environment was decoupled from its many dependencies, they reaped the benefits of service virtualization.

Increased Test Capability

After adoption was complete, they became sophisticated users. Alaska Airlines increased their test capability to include cloning test data, fabricating scenarios, and creating tests where scenarios ran continually, vastly scaling the testability of their application under test.

“If you can dream it, you can build it. We essentially went from being untestable to testable to now having a thriving demand for more of these capabilities.”

—Ryan Papineau, senior software engineer at Alaska Airlines

Eliminated False Positives & Wasted Time

They have 500 automated test cases that leverage this virtualized test environment. Those test cases run as needed, improving their automated execution by magnitudes over conventional wait-and-see or hunt-and-peck testing models.

Thanks to all this simulation and automation, testers can test whatever scenario they want, whenever they want. Their tests are now 100% reliable and repeatable. They eliminated all the variables that previously caused false positives and wasted time. Enabling testers to instantly access the exact test environment configurations that their test plans call for helps Alaska Airlines ensure smooth flight operations when faced with anything from a brief air traffic control outage to the improbable July snowstorm.

Solution Benefits

Another key learning experience and benefit for Alaska Airlines was a cultural change that made the organization more open to try new things. Now, the bigger the problem to solve, the easier the adoption of new techniques and tools is for the organization. The benefits that the Parasoft service virtualization solution offered their team include the following.

Decoupling from the complex production environment, which allowed Alaska Airlines to isolate their application and generate the test cases and environment that would accelerate their testing.

Repeatable, concise, precise test scenarios were invaluable to Alaska Airlines and enabled them to scale the types and frequency of their testing. The team expanded and customized scenarios as needed and injected specific scenarios for special test cases. The repeatability of these tests sped up the development cycles and the consistency removed a serious challenge they had with the real-world operation environment.

Parallel testing environments reduced the wait time impact. The ability to clone and customize scenarios allowed Alaska Airlines to speed up their simulated test cycle. Parallelization sped up testing but also meant specific time-dependent scenarios occurred in shorter time frames.

Test data management enabled the team to decouple production databases and ensure data consistency, which plagued Alaska Airlines’ operating environment. They reused test data captured from real-world systems to create life-like test scenarios that were difficult to coordinate in real time. Parasoft Virtualize test data management allowed the team to replicate databases consistently and in minutes. Something difficult to achieve in their real environment.