The way software teams execute their automation tests hasn’t evolved in years. Development and QA Teams are primed to jump into The 3rd Evolution of Automation Testing and discover how to to run automated tests faster and better.
- First Generation: Running Automated Tests Sequentially
- Second Generation: Running Automated Tests in Parallel or Concurrently
- Third Generation: Running Automated Tests by Priority
Running scripted tests has certainly sped up the delivery life-cycle. Why test functionality manually when it can be scripted and run automatically faster? Very logical jump there.
However, the problem that arises once automation testing matures is long common and experienced, at some point the scripted tests begin to take too long time to complete and slow productivity and increase infrastructure costs.
QA Teams with extensive test automation continuously fight to keep completion time within an acceptable range. Eventually, many find themselves running their tests overnight.
At that point, Developers don’t get to see test results to the next day leading to decreased output and delayed releases. Something most Quality Assurance Leaders have experienced in the past and which leaves a lasting impression.
Let’s go through the three Big Jumps in the Evolution of Automation Testing…
1st Generation: Running Automated Tests Sequentially
Sequentially, or running your tests one at a time.
When automated testing made its debut decades ago, it changed everything. It brought speed and relief to a manual and cumbersome process. Why perform hours of painstaking manual testing for something that can be automated to run faster.
However, as scripting of tests began to take hold, it didn’t matter how many tests a team had built, they were stuck running their tests one at a time. Or sequentially.
As teams built out automation test practices, the length of time to complete testing continuously grew with each newly built test. Meanwhile, Developers began to wait longer and longer to see results which eventually led to reduced output and delayed releases.
Hence, the evolution of running all tests in parallel was born.
2nd Generation: Running Automated Tests in Parallel
Parallel Testing aka Running your Tests Concurrently
This is where most Development and Testing teams find themselves today, running their tests in parallel or concurrently. It makes sense, if there are a lot of scripted tests – open some more pipes to push all your tests through to keep completion time down.
If you have 500 tests, open 5 parallel testing threads and run 100 tests through at a time.
However, Parallel Testing has several significant downsides teams soon encounter.
- High Hardware Utilization Rates (Heavy CPU and Memory Intensive).
- Collisions:
- Can’t always run tests in parallel as they pull from the same functional area, which can cause false failures or Flaky Tests.
- High Cost with any paid provider, i.e. Licensing or Cloud Testing Platform.
- May need to break tests apart, requiring high maintenance.
- Law of Diminishing Returns begins to apply.
It’s quite commonplace for teams to open more and more concurrent threads as their automated test-suites grow. Soon, a QA team might find themselves with 10 or more parallel threads running, with a very high bill and not the corresponding value associated with it. The value and time savings plateaus.
3rd Generation: Running Automated Tests by Priority
NextGen Tech: By Priority
So enters the next phase of how to run automated tests faster – only run the tests that matter.
Through predictive ML and AI, automatic risk based testing technology now exists so that when a developer makes a change – only the relevant tests are selected to be run.
If a team can plug in a tool that automatically selects which tests to be run given a certain area changed in the code, all the tests that don’t need to be executed fall to the side – thereby, cutting the fat out of test-suites and with it: completion times.
For example, if you have 600 E2E tests and a developer makes a change that affects 16% of the front-end – approximately 100 of those tests are not impacted by that change. If running by Priority, the tool determines which area of the code is being worked on and subsequently selects and executes only the actual 100 tests relevant to that developer’s work. Outcome for Developers, they get relevant and timely feedback 83% faster without the fluff.
Test only what Matters, aka, Developer Changes
Outcome for Testers, they just greatly alleviated their workload and looked like Rockstars while doing it. They also can now take lengthy overnight test runs that aren’t part of the daily build, and put them in the CI/CD for faster developer feedback.
Great news, this technology is complementary with parallel testing. Teams can continue to run their tests concurrently…but now they don’t need 20 threads…maybe now two….vastly reducing the hardware/cost associated with running their tests while still accelerating completion time.
Value, run automatic tests faster by only executing the ones that matter.
Tech titans, Google and Facebook, have already adopted this best practice.
By running tests by priority, teams can accelerate their automated test completion time, reduce CPU/Memory/Hardware costs associated with software testing (less CPU, Memory, Parallel Threads, etc.), and achieve a true Shift Left Strategy by tightening the feedback loop between Testing and Development.
Here’s a way for all teams to adopt the Third Generation of Running Automated Tests by Priority.
Run your Automated Tests Faster by Priority
The next hop in the evolution of running your automated tests is here, will you take advantage?
With the digital landscape in constant flux, Development and Testing teams need to be quick to adjust to changing factors. Running tests by Priority gives them the insights and feedback necessary to adjust fast and deploy confidently even faster.
Don’t let slow test results slow your team any further, accelerate their output to maximum speed by getting them their test results immediately after any change. For the teams running tests overnight, what difference would it make to give Developers results on the same day?
Learn how how to test by priority here.