Fixing Selenium in Your CI/CD Pipeline

Running automated tests on a continuous basis may seem like a no-brainer. A CI system is key, providing agile development teams with prompt visibility into the health of their applications. If automated tests are passing, then the application is healthy. If tests are failing, then the application is broken in some way. Or is it?

The unfortunate reality is that automated tests do not always give us such confidence, especially when running Selenium tests. Selenium tests can fail for many different reasons, regardless of whether the application is broken in any way or whether it changed. These problems are nicely detailed in a previous blog post. Here, I will explain how to modify the CI/CD pipeline to automatically fix those issues, improving confidence in the test results from the CI system.

Selenium CI Sorrow

CI/CD amplifies Selenium test problems. Let’s look at how this happens. First, development lands some code changes. The CI system detects the change and triggers the build pipeline. Modules are compiled, unit tests run and pass (yay!), and then the application is packaged for deployment. The deployment pipeline triggers next, staging a test environment, deploying the application under test, and then kicks off the integration tests, many of which are written using Selenium. Not all the Selenium tests pass at which point the CI system notifies various individuals of the test failures by email. Those individuals respond, sometimes burning a significant amount of time investigating and troubleshooting. Product owners, managers, and other interested parties also see the failures on their dashboards which divert their attention from other things. Knowing that the Selenium tests themselves are unstable, how can anyone have confidence that CI/CD reported a real regression or defect? How much time is wasted troubleshooting false positives?

Simply improving Selenium tests can be non-trivial, sometimes involving trial and error, especially when issues are intermittent or difficult to reproduce. From one build to another there are failures that can move around, where there is no single test to blame. They can be systemic or environmental issues that come and go. The CI system can be using VM or cloud-based executors that have inconsistent performance characteristics. Tests run and pass on developer workstations but fail in the CI automation. Sound familiar?

Selenium CI Solved

Parasoft Selenic was developed to help combat such challenges. Selenic automates the analysis and correction of Selenium test issues, providing fixes or insight into those issues, and makes those findings available by the time humans are notified by their CI system. The burden is shifted to the CI system, away from the humans who are reacting to notifications from the CI system. With just two easy steps, you can automate the process of fixing Selenium tests as part of the CI/CD pipeline using Parasoft Selenic.

Inject the Selenic Agent

To integrate Parasoft Selenic, you make a one-line change to your existing test execution script. Let’s say I have a Maven execution step in the pipeline that is driving my Java-based Selenium tests. Typically, I am specifying two Maven goals, “clean” and “test”, which are passed as command line arguments to Maven. Here, you are just adding one extra Maven command line argument:

-DargLine=-javaagent:${SELENIC_HOME}/selenic_agent.jar=selfHealing=true,sessionId=${BUILD_TAG}

In Jenkins CI, this might look something like this:

The Selenic Agent is now injected into the test execution process, provided that Selenic is installed and licensed at the location referenced by the “SELENIC_HOME” variable. The Selenic Agent can do various things but for test stabilizing purposes I am only enabling the “selfHealing” feature which attempts to identify and fix Selenium test issues on the fly.

One thing to carefully consider is whether your Selenium tests all live in one module or if there are tests in multiple modules. It is common for tests to be split into multiple modules for organizational purposes. For Selenic to aggregate information from multiple test modules a “sessionId” argument must be configured. In the above example, I am using “BUILD_TAG” which is a pre-defined variable in the Jenkins CI system that works nicely for this purpose. Regardless of what CI system you are using, my recommendation is to construct a unique session identifier using variables provided by your CI system.

After making this one-line change, Selenic automatically identifies and corrects test stability issues, giving everyone much more confidence in the test results published by the CI system. However, there’s one more Selenic component that I recommend integrating.

Execute the Selenic Analyzer

How do we know what problems Selenic found or what it corrected? Are there changes I should make to the Selenium tests to avoid the kinds of problems that Selenic is finding? To get this information, you would run the Selenic Analyzer as another execution step in the pipeline. This is a second one-line change but still trivial:

java -jar $SELENIC_HOME/selenic_analyzer.jar -report target/selenic-reports -sessionId ${BUILD_TAG}

In Jenkins CI, this might look something like this:

The Selenic Analyzer collects information that was logged earlier by the Selenic Agent, performs some additional analysis on the data, and then builds some reports which can then be archived by the CI system. By default, the analyzer processes information from the most recent session but for robustness I recommend explicitly configurating the “-sessionId” argument with the same value that was used earlier when configuring the Selenic Agent.

The Selenic Analyzer writes the reports in the current working directory by default. However, in order to make archiving of the reports easier you should add a “-report” argument to instruct Selenic to write reports to a “selenic-reports” folder in the build output directory of your project. After doing this, you would add one more step in the pipeline to archive ” target/selenic-reports/**”:

The report files are now available to everyone though the CI system. In the case of Jenkins CI, reports are conveniently accessible at stable URLs:

http://{ci_server}/job/{test_job}/lastSuccessfulBuild/artifact/target/selenic-reports/report.html

http://{ci_server}/job/{test_job}/lastSuccessfulBuild/artifact/target/selenic-reports/report.json

The first URL points to the Selenic HTML report that can be viewed directly in a web browser. The second URL is to the JSON report which developers can provide to the Selenic IDE plugin to import the recommendations from the last automated test run.

Other Considerations

Selenic has many other options that you may consider enabling depending on your test environment and other specific needs.

Selenic records data for every test session. The recorded data acts as a knowledge base, enabling Selenic to learn and make better self-healing decisions and recommendations over time. By default, this data is stored on the machine where the test executes. Therefore, if you have a pool of executors, then you would want this data to be in a shared folder that is accessible to all your executors. The data location can be explicitly configured by passing a “data={path}” parameter to the Selenic Agent and a “-data {path}” argument to the Selenic Analyzer, where “{path}” would be a file system path to a shared folder. On Windows, this could be a UNC path like \\HostName\SharedFolder. The recorded data will also build up indefinitely unless constrained. This can eventually lead to high disk usage. A limit on number of recorded test sessions can be configured by passing a “maxSessionDaysToKeep={num_days}” parameter to the Selenic Agent. The “{num_days}” is the number of days on which data was recorded and not calendar days. In other words, Selenic does not count any days where no test session data was recorded. This is helpful if run your tests infrequently, or at inconsistent intervals, or had server down time where you missed running tests for a few days.

Selenic can also be configured to capture additional diagnostic information at various levels of granularity. For example, Selenic can be configured to capture screenshots for every Selenium action or only for failing actions. To enable screenshots on test failures you can pass a “screenshot= failures” parameter to the Selenic Agent. Screenshots for failures are available in the HTML report produced by the Selenic Analyzer.

CI/CD + Selenic = ❤

Parasoft Selenic was built to help accelerate agile development using Selenium as part of CI/CD. Developers and testers can be more productive, spending less time chasing Selenium “ghosts” and more time working on real things. Increase confidence in your Selenium test results with Selenic. Download your free trial today.