Change-Aware Predictive Test Selection and Prioritization

Sammanfattning

Test Case Prioritization and Selection (TCPS) methods aim to order or select tests to minimize test execution time without significantly reducing test coverage. In collaboration with a System Analysis Team at Arm, this project demonstrated machine learning-driven prioritization in an industrial Continuous Integration (CI) pipeline, using predictive models to optimize test execution. The team runs a comprehensive nightly test set (N) to detect faulty code introduced during the day, but also executes a smaller subset of these tests (C ⊂ N) before any code change is added to the code base, aiming to catch faults as early as possible. We added a new test set (E ⊂ N \ C) which ran concurrently with C for a set time period and improved the total fault-finding rate by 600%. During this period, data about test outcomes, flaky tests and characteristics of code changes were collected. This dataset was then used to train machine learning models to predict the optimal test order of the combined tests in C ∪ E, given a certain code change, in order to catch all faults in minimal time. The top predicted tests were then selected to run on each change. In our Arm case study, the best performing model was a listwise ranking model, which could find >97% of all faulty code changes, 6.8 times as many as C, while saving on average >90% of the execution time needed by C. This approach makes no assumptions regarding software type, programming language, or technology stack, relying solely on features readily extracted from CI and version control systems, thus being widely applicable both within and outside Arm.