Metrics to Make Your Quality Practices Effective

Test Automation is considered the silver bullet that will enable teams to shift left and accomplish CI/CD. However, how confident are you with the value added by the automated tests in your organization and teams? Is your automation effective? Is it giving you the ROI you expected from it?

Likewise, how comfortable are you with the testing approach of your team, and how is it helping improve your product quality?

In the context of Quality, Testing, and Test Automation, for each role on the team, we will discuss the following:

• What are effectiveness and ROI?
• What are the commonly used metrics to measure effectiveness and ROI from your Testing & Test Automation activities? (Ex: code coverage, defect leakage, test execution status, test execution time, # of tests, etc.)
• Do these metrics help in identifying blockers and root causes of issues?
• How do these metrics help to make decisions to better product quality?
• Discuss other metrics that can help each role make meaningful decisions to improve product quality!
• Ex: Cycle Time, CLT, defect analysis with RCA, MTTR, Functional coverage, etc.

Outline/Structure of the Session

• Getting on the same page - understand commonly used terminology - 5 min
• Discuss with participants the metrics used in their teams / organizations to understand effectiveness of testing and automation - 10 min
• Discuss the metrics in the context of the team / organization - 5 min
• A fresh look at metrics that can help different roles on the team understand the state of automation and quality of the product - 10 min
• A few practices that will help in increasing the ROI for automation - 10 min
• Q&A

Learning Outcome

• Using metrics blindly, without context, does not help the team take decisions to improve the quality of the product
• Each role on the team may need a different set of metrics to understand the current situation w.r.t. automation and quality of the product.
• A reference set of metrics that can be used for each role