Defect Types and Lifecyle

How TestBrain categorizes defects and your options for configuration. #

Defect Types #

  • Project – The Standard Defect Type
  • Flaky – Tests which fail due to defects of type ‘flaky’ and will not break the build unless otherwise specified.
  • Invalid Test – Indicates that the test is or was invalid when the test failed.
  • Outside Scope – Select this defect type when the failure is a consequence of another test failure.
    • For example, if Login breaks and your Login tests fail and create a defect and all your other tests fail and create a defect which is secondary, then this defect would be marked as ‘Outside Scope,’ essentially saying that we don’t want to prioritize this defect as it is a corresponding failure to the real defect.

Defect Lifecycle #

Automatic Opening

Defects will automatically open when tests fail. The type will depend on a number of factors. By default, they are opened as a project defect, but if the model indicates that these failure were flaky the default type will be set to flaky.

The model for determining flakiness looks at a large number of features, such as:

  • Previous Defects
  • Failure Messages
  • Failure Types – fail vs. error vs. assert
  • Average run time vs when the test failed
  • History of that test
  • Area of the change

Automatic Closing #

Defects other than of type, ‘flaky,’ will be closed automatically when all the associated tests (tests that have failed because of this defect) pass on a commit which is a child of the commit which caused the defect.

Caused by Commits #

Caused by commits are the set of commits which caused the defect.

Closed by Commits #

Closed by commits are the set of commits which closed the defect.