Eradicate Flaky Tests

Have you heard of “flaky tests”?

There are a lot of articles, blog posts, podcasts, conference talks about what are “Flaky tests” and how to avoid them.

Some of the ideas proposed are:

Automatically rerun failed tests a couple of times, and hope they pass

Automatically retry certain operations in the test (ex: retry click / checking for visibility of elements, etc.) and hope the test can proceed

Unfortunately, I do not agree with the above ideas, and I would term these as anti-patterns for fixing flaky / intermittent tests.

We need to understand the reasons for flaky / intermittent tests. Some of these reasons could be because of issues like:

•  timing issues (i.e. page loading taking time)

•  network issues

•  browser-timing issues (different for different browsers / devices)

•  data related (dynamic, changing, validity, etc.)

•  poor locator strategy (ex: weird & hard-wired xpaths / locators)

•  environment issue

•  actual issue in the product-under-test

In this session, with the help of demos, we will look at the following techniques you can use to reduce / eliminate the flakiness of your test execution:

•  Reduce number of UI tests

•  Use Visual Assertions instead of Functional Assertions

•  Remove external dependencies via Intelligent Virtualization

Outline/Structure of the Demonstration

• What are flaky tests? - 2 min
• Reasons for flaky tests - 10 min
• Demo of techniques to fix flaky tests - 25 min
• Intelligent & dynamic stubbing of external dependencies
• Visual Assertions instead of Functional Assertions

Learning Outcome

• Explore the real reason of flaky tests
• Implement Visual Assertions instead of Functional Assertions to increase test coverage, while reducing code and increase test stability
• Use Intelligent and Dynamic Virtualization to remove external dependencies