A flaky test is the Darth Vader of testing. The scariest thing in the universe.
Flaky Tests are not one with the Force
Flaky tests are test scripts that fail for no apparent reason or change to the code. A lot of companies have major problems with them. Most teams only have two ways of dealing with them though, they either 1) fix the test or 2) delete the test.
Why deleting a flaky test is not a good idea.
Time:
Teams have spent a long time writing those tests, and they’ve written them for a reason. It is demoralizing to delete tests after you’ve spent time creating them, especially if you weren’t the reason they became flaky and or they work fine in your test environment.
Value:
Flaky tests still have value. They can still find new defects and even the flakiness of the test can be indicative of the overall health of the system. Let’s say you have a test which is flaky one time in a hundred. If that increases to ten in a hundred, that is very important for testers to know.
Additionally, it might not be the test’s fault they are flaky. The flakiness could be caused by the underlying infrastructure, such as your docker container or the mobile device test environment. In this case, deleting the test is unwarranted and the problem may either be patched or solved by changing mobile device. If you were to delete every flaky test only to discover it was caused by the mobile phone, that would be a huge loss to the project. And Costly!
How to handle a Flaky Test:
There are other solutions to flaky tests. You can deal with them in more elegant ways similar to how the larger corporations like Google and Facebook do. Your team can implement some of these workarounds themselves or at Appsurify this is part of our solution, TestBrain. We handle your Flaky Tests for you, so you don’t have to.
So please don’t delete flaky tests – let TestBrain handle them!