Have you heard of “flaky tests”?
There are a lot of articles, blog posts, podcasts, conference talks about what are “Flaky tests” and how to avoid them.
Some of the ideas proposed are:
Automatically rerun failed tests a couple of times, and hope they pass
Automatically retry certain operations in the test (ex: retry click / checking for visibility of elements, etc.) and hope the test can proceed
Unfortunately, I do not agree with the above ideas, and I would term these as anti-patterns for fixing flaky / intermittent tests.
We need to understand the reasons for flaky / intermittent tests. Some of these reasons could be because of issues like:
• timing issues (i.e. page loading taking time)
• network issues
• browser-timing issues (different for different browsers / devices)
• data related (dynamic, changing, validity, etc.)
• poor locator strategy (ex: weird & hard-wired xpaths / locators)
• environment issue
• actual issue in the product-under-test
In this session, with the help of demos, we will look at the following techniques you can use to reduce / eliminate the flakiness of your test execution:
• Reduce number of UI tests
• Use Visual Assertions instead of Functional Assertions
• Remove external dependencies via Intelligent Virtualization