Step 4: Add Script to CI/CD to Connect Tests

Installation Instruction #

To use TestBrain to optimize test automation, you will need to add an executable script to the CI to call the TestBrain API with the test conditions. 

To manually create using the provided syntax and for up to date versions of the CI script: please go here

Requirements: #

Python 3.7+

Installation Command #

pip install appsurifyci --upgrade

Available Arguments #

argumentoptions
--apikeyAPI Key from appsurify. Found in top right hand corner drop-down menu.
Can be stored as a Secret in CI server secret storage capabilities.
Reference the Secret via variable within the Appsurify Script execution in the CI configuration.
--urlUrl of the appsurify instance i.e., https://dummy.appsurify.com/
--projectName of the project
--testsuiteName of the testsuite
--branchName of the branch the tests are running against
--commitSHA of the commit

Recommended Arguments #

argumentoptions
--runtemplateOptions – all tests, no tests (used just to push results from prior command), prioritized tests (requires –percentage argument to be added and the percentage of tests to be run. See below for details.
--testtemplateOptions – mvn, cucumber mvn, sahi testrunner, sahi ant, testim, mocha, rspec, robotframework, cyprus, pytest, mstest, vstest, opentest. For additional integrations, talk to the Appsurify Team.
--runcommandCommand to execute tests in the target environment, if command is custom to your test suite. Note, runcommand will override generic testtemplate at time of final configuration and prioritized test execution.
--reportLocation of XML reports created by the test run if this is not the default location for the test type

Parameter Details – Tests Execution #

OptionDetails
all testsWill run all tests. Will fail on any failure. Recommended for nightly or weekly test executions
no testsWill run no tests. Will fail on any failure. Recommended when starting with Appsurify to just upload data
prioritized testsRequires the percentage argument. Will only fail the build on new / re-opened defects. Recommended for your most frequent run type.

EXAMPLE #

--runtestswithappsurify --url "https://ABC.appsurify.com" --apikey "api key" --project "project" --testsuite "testsuite" --runtemplate "all tests" --runcommand "mvn" --commit "commit id" --branch "branch" --report "report" --pipeoutput "true"

Example in CI Pipeline: #

Setup Commands

--runtestswithappsurify --url "https://testbraindemo.appsurify.com" --apikey "NTK6cD1Ua3doV3uNDMdWzz" --project "TestBrainDemoJunit" --testsuite "Selenium" --runtemplate "all tests" --runcommand "mvn" --commit "commit id" --branch "branch" --report "report" --pipeoutput "true"

Additional Arguments – For Customization #

argumentdefaultoptions / details
--failnewdefects, reopeneddefectsoptions newdefects, reopeneddefects, flakybrokentests, newflaky, reopenedflaky, failedtests, brokentests
--rerun“false”whether failed tests will be rerun
--maxrerun3the number of times failed tests should be rerun
--failfast“false”whether after each set of tests runs to determine if the build has failed (tests will still be rerun if this has been selected)
--executioncommand” “Command to be executed following completion of the script [[teststorun]] will be replaced with the formatted list of tests to run
--percentage” “Percentage of tests to be run if prioritized tests is selected in the runtemplate, example: percentage “20”

For CI specific integrations, please contact Appsurify

On Summary Page: Test Bind turns GREEN #

When we start seeing test runs come through successfully, in the Summary Page – the Test Bind button will turn Green!

Test Bind is now Green after successful Test Run Push

That’s great – you’re nearly there!

Now that the CI script is successfully installed, test runs should automatically start being pushed and your model will begin to grow in Maturity. You’ll see Test Runs populate below and your graphs being to populate with YOUR data.

Once your Model hit’s 100% or 50 Test runs, you’re Model will be trained and ready for you to begin Prioritize Testing!

Such as here!

Once Repo and Test Bind are Green, push 50 Test Runs to Mature your Model and begin enjoying Faster Test Feedback and Optimization!

Once your Model hit’s “100%” Maturity, your model will be trained and ready to begin optimizing your QA practice!