Metrics That Matter for End-to-End Testing

How Do You Measure Software Quality?

From the results of multiple market surveys, it’s clear that industry executives agree that software quality is a top priority for delivering business value to customers. The challenge is how to define what that means and how to achieve it. This involves selecting and applying appropriate metrics to your software development and testing processes.

One way to improve software quality is by adding test automation to your development workflow. This enables you to shift testing earlier in the life cycle, finding and fixing bugs sooner with more robust tests than can be accomplished manually. End-to-end testing involves applying multiple testing techniques to address all facets of the application, from the code, through the APIs, to the user interface.

Recently, we had the opportunity to talk with Diego Lo Giudice, vice president and principal analyst at Forrester Research, about his thoughts on metrics for software quality. His expert insights are featured here and shared in the on-demand webinar replay, Deliver Superior Experiences With End-to-End Testing & Metrics That Matter.

End-to-End Testing Q&A

With Featured Guest Diego Lo Giudice, VP & Principal Analyst at Forrester Research

Q: In the webinar, Deliver Superior Experiences With End-to-End Testing & Metrics That Matter, you mentioned Agile and DevOps are two sides of the same coin. How do these practices help companies apply and integrate end-to-end testing into their CI/CD workflow?

A: Agile and DevOps are two sides of the same coin, meaning that one cannot do without the other.

Why?

Because if Agile helps tear down the wall between development teams and business and speeds up things while aligning on requirements, DevOps continues the job between development teams and I&O operations. Agile + DevOps brings complete end-to-end business agility.

From an organizational and practice perspective, Agile + DevOps must endure shifting testing left and making it part of what needs to be done in sprint. If not, continuous delivery/deployment cannot happen. Testing done late will break the continuous DevOps loop.

The use of CI/CD tools in Agile + DevOps can now orchestrate a workflow where code gets checked in, unit tests are run, functional tests are run, performance tests are run, and at each step, based on coverage and passed vs non passed tests, automatic decisions can be made to push the code forward or back to the team to fix and test more.

So now you can see, we have replaced what used to be a manual long governance process with an automated end-to-end test workflow, which normally would have been a lot of handovers, release meetings, and more to do the same. It reduces testing cycle time by automating governance. That is the help companies get with it.

Q: Early in the presentation, you noted that software quality is an executive concern. In what ways can IT and executives most effectively measure and quantify quality? What are some of the most important metrics for software testing and quality?

A: In the research for modern application metrics that really matter, the most important outcome metrics focus on business value. However, leading into business value are quality metrics.

The most important quality metrics are still metrics that measure software quality in production—escaped defects. Nobody wants defects to reach production because they cause revenue loss, hit the news with a bad reputation, and impact user experience.

The usual metrics to track include code density defects at unit testing level, escaped defects from development to testing, test automation execution time, number of commit test failures, and number of automation test failures. I would add to each of them “trends.”

Is the trend of speed of running automated tests going down? If yes, that might be an indication of problems in the tests.

Q: You also talked about avoiding a “sea of metrics.” So how do I know if I have too many? How do I know which ones to prune?

A: That is where frameworks like OKRs (objective key results) or GQM (goal, question, metrics) come in. Sticking to the principle that you can’t manage what you don’t measure, you need to decide what you need to manage.

What are the team goals (or the product goals)? From there, derive the useful metrics you need. There is no point in having performance metrics if performance is not the main pain your business has, and therefore is not a goal for the organization.

If instead, the issue is technical bugs, then you might want to put in the dashboard unit tests coverage and pass metrics.

Q: How can meaningful metrics help to justify the value of test automation?

A: Normally, as per the framework mentioned above, quality is a leading metric contributing to business value. That is a starting point to agree on with stakeholders. Then as a consequence, replacing manual tests with automated tests will speed up the delivery cycle, because also test execution cycle time decreases.

Overall development cycle time, even from cash to code, will be positively impacted by test automation. Quality also increases if automation is right because you can run the same tests (think regression) over and over. You build once but execute many times. However, be aware of how difficult automation might be since code and systems poorly coded can be hard to unit test. Or if a test does not need to be repeated, why automate it? Although this rarely happens, especially in Agile.