TestBrain is software that uses machine learning to make automated and manual QA faster.

TestBrain uses Predictive Test Selection to find and run the subset of tests from your automated test suite that are relevant to the specific code changes. (add links for PTS)

TestBrain then analyzes your test results to filter out flaky test failures (tests that fail for reasons unrelated to the code changes) and failures due to already open bugs so they don’t break the builds.

Together, TestBrain allows you to test each commit as it is applied and get useful results back to the developers immediately about whether their specific code changes pass or fail testing.

A comprehensive analysis using the open source jsoup project showed that after 500 runs, TestBrain reduced the number of tests by 98.5% without missing a single defect.

For typical projects with a substantial code base and large test suite, TestBrain should reduce the number of tests to run by at least 90%, speeding up testing by 10x or more.

The TestBrain configuration allows you to choose between running a wider set of tests and having higher confidence in not missing any failures or running a smaller set and getting quicker results. That trade-off depends on how frequently you run the full test suite and how long the tests take.

You should run the full set of tests periodically to catch any failures that are missed. TestBrain will learn from those failures and continue to improve over time. It is best to think of TestBrain as a way to get quick test results to developers by catching the vast majority of failures with a small set of tests, and reducing the frequency of running the full test suite to reduce costs.

A flaky failure, or flaky test failure, or unreliable automation is an automated test that fails for reasons unrelated to the code changes. A defining characteristic of a flaky test failure is that the same test can pass or fail with exactly the same code. Flaky test failures are most common for UI tests where many external factors can interfere with the test execution.

Flaky failures are caused by anything external to the code being tested that can interfere with the test execution. Typical causes include:
 

• Device losing connectivity or power
• Browser crashing
• PC or phone downloading updates
• Race conditions (thread.sleep)
• Asynchronous calls
• Stale element exception
• Element not visible exceptions
• Xpath and CSS locators
• Unstable infrastructure
• Unstable application

TestBrain’s machine learning analyzes the test results to determine if each failure was caused by a code defect, a flaky test, a broken test, or an open bug and tracks each separately. TestBrain allows you to configure which of types of failures will cause the build to fail. When testing each commit to get quick results to developers, TestBrain should be set to only fail the build on code defects. While flaky failures (and broken tests and open bugs) are tracked, they are filtered out so they don’t break the builds, giving a clean pass/fail signal to the developers, and saving the test team time analyzing all the failures manually.

TestBrain generates a heat map to show where to prioritize your testing. TestBrain analyzes each commit for the likelihood of defects then generates a RiskMap showing you which commits have the highest risks and mapping the commits to functional areas of the code. You can use this map to see which areas of the code need the most testing and which haven’t been changed at all.

TestBrain also provides risk alerts to notify the test team of commits likely to contain defects and any changes to files you’ve marked as sensitive. This allows you to begin checking the riskiest changes immediately instead of waiting until the end of the sprint, or notifying the developers or code reviewers of the risk of these changes so they can be double-checked.

There are 3 options for installing TestBrain:
 

• Cloud version: TestBrain runs on Appsurify’s infrastructure on AWS. Just add a webhook from your repo and a script into the Jenkins pipeline to connect to TestBrain in the cloud.
• Self-hosted: Install TestBrain on 3 docker images on VMs on your own servers with no connection to Appsurify.
• Agent: Install a small script on your own VM that connects to the repo and sends only the commit metadata to TestBrain running on AWS. This allows you to take advantage of TestBrain running in the cloud (so we handle all the management and updates) while guaranteeing while limiting our access to the repository to read-only access to metadata.

TestBrain needs to connect to the code repository to receive notifications of new commits via a webhook. For manual testing, that is all that is required. For automated testing, TestBrain installs a script as an executable step on the CI/CD to run a list of tests.

TestBrain works with any git-based repository (GitHub, BitBucket, Azure DevOps, etc.) We also have integrations available for Perforce and Microsoft TFS/TFVC.

TestBrain works with any CI/CD that allows the addition of an executable script to do an API call. That is most if not all CI/CD tools. It has been tested with Jenkins, Codeship, TeamCity, CircleCI, and Travis.

TestBrain works with any automated tests that can generate JUnit, XUnit, or XML output and has a way to specify individual tests to run instead of an entire test suite.

Yes. TestBrain works with any automated tests that can generate JUnit, XUnit, or XML output including Selenium.

TestBrain works with any automated tests that can generate JUnit, XUnit, or XML output. This includes unit tests, integration tests, end-to-end tests, API tests, and UI tests. All that it requires is a test that returns a pass/fail criteria.

Installation and configuration takes approximately 30 minutes to connect to the repository, specify code areas, and add a script to your CI/CD. TestBrain then takes 1 to 8 hours to analyze the commit history before it is ready to use.

You can try TestBrain by running it in “monitoring mode”. In monitoring mode, TestBrain sees the commits and test results, but doesn’t change your testing process. It can then show you which tests would have run if you’d turned on TestBrain’s test selection at different test priority levels, and how many tests you’d have avoided running. While in monitoring mode, the machine learning is learning from the test results, so it’s a good way to get started. Once you have confidence in the results, you can turn on test selection and decide how often you want to run the reduced test set vs. how often to run the full suite.

In order to see any activity, TestBrain needs be connected to an active repository that’s getting commits and to a CI/CD that’s running automated tests. (You can use TestBrain for manual testing by connection only to the repository.) In order to provide benefit, TestBrain needs to be used on a test suite that takes a substantial amount of time to run or generates a large number of flaky failures.

No. For automated testing, TestBrain finds the right tests from your existing test suite to check the specific code changes and connects to the CI/CD to have them run automatically. For manual testing, TestBrain shows you a risk profile of the changes so you can focus your testing on the areas of the code with the highest risks.

TestBrain helps you run the right tests from your test suite instead of wasting time and money running tests that are certain to pass. But you have to create the tests. TestBrain itself is not a test tool. It is a tool to make your existing testing run faster and more efficiently.

No. TestBrain offers optional integration to automatically open and close bugs in Jira based on the test results, but use of this feature is not required. TestBrain automatically tracks bugs internally, with the information available on the TestBrain dashboard.

No. TestBrain does not read or analyze the code itself. Over time, it uses the history of test failures to map the tests to particular files. Initially, before there is enough data of test failures, TestBrain can either map the tests to the files and folders by matching names, or the user can set up the mapping. This allows TestBrain to be entirely independent of coding language while being more accurate than code coverage.

No. TestBrain eliminates the need to tag the software. Instead, it uses machine learning to determine which tests are useful to check particular changes to the code.

TestBrain is a paid tool that reduces the cost of cloud-based testing and testing infrastructure by far more than it costs. There is a 60 day free trial to give you time to determine how much time and money TestBrain will save you.