FAQ

Search below to view our most pressing questions regarding automated testing tools. Have a question that’s not listed? Impressive. Contact us and we’re happy to provide an answer.

TestBrain Functionality

What is TestBrain?

TestBrain is the First QA Optimization Platform leveraging patent-pending Predictive Test-Selection Technology to accelerate Automated Test Runs and CI/CD Pipelines.

Who would benefit from TestBrain?

-DevOps Teams wanting Faster and Cleaner CI/CD Builds.
-Software Testing Teams wanting Faster Testing Results.
-Digital Leaders wanting Increased Team Output and Faster GoTo Market Timing.

How does TestBrain speed up automated testing results?

TestBrain uses patent-pending Predictive Test Selection Technology to find and run the subset of tests from your automated test suite that is relevant to the specific code changes instead of running the full suite of mostly unaffected tests.

Great for lengthy Regression, UI, E2E, and Integration Testing.

How much faster will TestBrain make my automated testing results?

A comprehensive analysis using the open source jsoup project showed that after 500 runs, TestBrain reduced the number of tests by 98.5% without missing a single defect.
For typical projects with a substantial code base and large test suite, TestBrain should reduce the number of tests to run by at least 90%, speeding up testing by 10x or more.

The TestBrain configuration allows you to choose between running a wider set of tests and having higher confidence in not missing any failures or running a smaller set and getting quicker results. That trade-off depends on how frequently you run the full test suite and how long the tests take.

How does TestBrain help manual testing?

Using its patent-pending technology, TestBrain analyzes each commit for the likelihood of defects then generates a RiskMap showing you which commits have the highest risks and mapping the commits to functional areas of the code. You can use this map to see which areas of the code need the most testing and which haven’t been changed at all.

What are risk alerts?

TestBrain also provides risk alerts to notify the test team of commits likely to contain defects and any changes to files you’ve marked as sensitive. This allows you to begin checking the riskiest changes immediately instead of waiting until the end of the sprint, or notifying the developers or code reviewers of the risk of these changes so they can be double-checked.

Does TestBrain reduce local or Cloud infrastructure utilization rates?

Yes, TestBrain is designed to only run the set of tests that have been impacted by a recent developer change. So instead of continuously running the full test-suite, whether locally or in the Cloud, TestBrain allows teams to run just the important tests – thereby, dropping infrastructure utilization rates by 70-80%.

What if the Smart Testing defers a defect?

TestBrain is configurable to your overall Test Strategy. Most fast-paced teams run Smart Test Subsets throughout the workday, then run a Full-Run overnight to catch any potential deferred failures.

TestBrain will learn from those deferred failures and recalibrate the Model on a rolling basis to continue to improve over time.

It is best to think of TestBrain as a way to get quick test results to developers by catching the vast majority of failures with a small set of tests during the workday, and reducing the frequency of running the full test suite to reduce costs.

What are flaky failures?

A flaky failure, or flaky test failure, or unreliable automation is an automated test that fails for reasons unrelated to the code changes. A defining characteristic of a flaky test failure is that the same test can pass or fail with exactly the same code. Flaky test failures are most common for UI tests where many external factors can interfere with the test execution.

What are the causes of flaky failures?

Flaky failures are caused by anything external to the code being tested that can interfere with the test execution. Typical causes include:

Device losing connectivity or power
Browser crashing
PC or phone downloading updates
Race conditions (thread.sleep)
Asynchronous calls
Stale element exception
Element not visible exceptions
Xpath and CSS locators
Unstable infrastructure
Unstable application

How does TestBrain eliminate flaky failures?

TestBrain’s machine learning analyzes the test results to determine if each failure was caused by a code defect, a flaky test, a broken test, or an open bug and tracks each separately. TestBrain allows you to configure which of types of failures will cause the build to fail. When testing each commit to get quick results to developers, TestBrain should be set to only fail the build on code defects. While flaky failures (and broken tests and open bugs) are tracked, they are filtered out so they don’t break the builds, giving a clean pass/fail signal to the developers, and saving the test team time analyzing all the failures manually.

Does TestBrain work if I am running my tests in Parallel?

Yes, TestBrain helps in this category. TestBrain would push less tests through your parallel threads.

So if you are attempting to reduce test execution times by running your tests in parallel, TestBrain will significantly enhance your current practice and see test execution times reduced by 95%+. TestBrain would also give teams the option to reduce the number of parallel threads required to achieve faster test times, thereby, freeing up significant CPU/Memory resources.

Does TestBrain work if I am pushing my tests to the Cloud?

Yes, TestBrain is designed to only push the important tests after any change. So if you are utilizing a Cloud testing platform, such as BrowserStack, SauceLabs, Google Device Farm, etc., TestBrain is designed to only push the prioritized tests up to the cloud – thereby, significantly decreasing your Cloud utilization.

TestBrain Installation and Integration

How do I install TestBrain?

There are two options for installing TestBrain:

SaaS or Cloud Version: TestBrain runs on Appsurify’s infrastructure on Microsoft Azure. Just add a GitScript or Webhook from your repo and a script into the CI/CD pipeline to connect TestBrain in the cloud.

On-Premise or Self-Hosted: Install TestBrain on a single docker image on a VM on your own servers with no connection to Appsurify.

What repositories does TestBrain work with?

TestBrain works with any git-based repository (GitHub, BitBucket, Azure DevOps, etc.) We also have integrations available for Perforce and Microsoft TFS/TFVC.

What testing frameworks does TestBrain work with?

TestBrain works with any automated tests that can generate JUnit, XUnit, or XML output and has a way to specify individual tests to run instead of an entire test suite. See full integration list here.

What type of tests does TestBrain work with?

TestBrain works with any automated tests that generate JUnit, XUnit, or XML output. This includes unit tests, integration tests, end-to-end tests, API tests, UI tests, etc. All that is required is a test that returns a pass/fail criteria.

Can I try TestBrain without changing how we work now?

You can try TestBrain by running it in “monitoring mode”. In monitoring mode, TestBrain sees the commits and test results, but doesn’t change your testing process. It can then show you which tests would have run if you’d turned on TestBrain’s test selection at different test priority levels, and how many tests you’d have avoided running. While in monitoring mode, the machine learning is learning from the test results, so it’s a good way to get started. Once you have confidence in the results, you can turn on test selection and decide how often you want to run the reduced test set vs. how often to run the full suite.

Will TestBrain create tests for me?

No. For automated testing, TestBrain finds the right tests from your existing test suite to check the specific code changes and connects to the CI/CD to have them run automatically. For manual testing, TestBrain shows you a risk profile of the changes so you can focus your testing on the areas of the code with the highest risks.

Is Jira required?

No. TestBrain offers optional integration to automatically open and close bugs in Jira based on the test results, but use of this feature is not required. TestBrain automatically tracks bugs internally, with the information available on the TestBrain dashboard.

What is required to be successful with TestBrain?

For prioritized automated testing, two items:

1. Access to the development code repository.
2. Install a small script to the CI/CD pipeline.

What CI/CD tools does TestBrain work with?

TestBrain works with any CI/CD that allows the addition of an executable script to do an API call. That is most if not all CI/CD tools. It has been tested with Jenkins, Codeship, TeamCity, CircleCI, and Travis.

Does TestBrain work with Selenium?

Yes. TestBrain works with any automated tests that can generate JUnit, XUnit, or XML output including Selenium.

How long does it take to install and configure TestBrain?

Installation and configuration takes approximately 30 minutes to connect to the repository, specify code areas, and add a script to your CI/CD. TestBrain then takes 1 to 8 hours to analyze the commit history before it is ready to use.

Can I try TestBrain on my PC?

In order to see any activity, TestBrain needs be connected to an active repository that’s getting commits and to a CI/CD that’s running automated tests. (You can use TestBrain for manual testing by connection only to the repository.) In order to provide benefit, TestBrain needs to be used on a test suite that takes a substantial amount of time to run or generates a large number of flaky failures.

Will TestBrain find bugs automatically?

TestBrain helps you run the right tests from your test suite instead of wasting time and money running tests that are certain to pass. But you have to create the tests. TestBrain itself is not a test tool. It is a tool to make your existing testing run faster and more efficiently.

TestBrain Technology

Does TestBrain use code coverage?

No. TestBrain does not read or analyze the code itself. Over time, it uses the history of test failures to map the tests to particular files. Initially, before there is enough data of test failures, TestBrain can either map the tests to the files and folders by matching names, or the user can set up the mapping. This allows TestBrain to be entirely independent of coding language while being more accurate than code coverage.

Is there a free version of TestBrain?

TestBrain is a paid tool that reduces the cost of cloud-based testing and testing infrastructure by far more than it costs. There is a free trial to give you time to determine how much time and money TestBrain will save you.

Does TestBrain use code tags?

No. TestBrain eliminates the need to tag the software. Instead, it uses machine learning to determine which tests are useful to check particular changes to the code.

FAQ