Reproducibility and Consistency of Machine Learning-Based Test Case Prioritization Techniques

Sammanfattning

Machine learning-based test case prioritization (TCP) has become an active area of software engineering research, promising more efficient fault detection in Continuous Integration (CI) environments by reordering test suites so that failure-revealing tests execute first. Despite a growing body of published work, two fundamental challenges remain unaddressed: many studies cannot be independently reproduced due to missing artifacts or undocumented preprocessing, and the consistency of reported results across different projects and configurations is poorly understood. This thesis empirically investigates the reproducibility and consistency of ML-based TCP by partially reproducing and extending the learning-to-rank framework of Bertolino et al. The framework is extended with a static call graph-based test selection mechanism, a proper Deep Q-Network agent for adaptive filtering, and the introduction of DART as a novel ranking algorithm. Six learning to-rank algorithms are evaluated over 30 independent repetitions on four open-source Java projects from the Apache Commons benchmark suite — Codec, Math, Compress, and Lang — using the Normalized Relative Prioritization Assessment (NRPA) as the primary metric. The results show that technical reproducibility was achieved but required non-trivial intervention, and that algorithm rankings are substantially inconsistent across datasets, with Spearman rank correlations between most dataset pairs being statistically non-significant (p > 0.05). Pairwise Wilcoxon signed-rank tests confirm that on all three non-trivial datasets, RankBoost, LambdaMART, and DART form a statistically indistinguishable top cluster, while no consistent ordering emerges across datasets. Four of six algorithms produced degenerate models on the simplest dataset (Commons Codec), exposing a disconnect between reported NRPA scores and genuine learned prioritization capability. DART ranked second or third across all four datasets, achieving inference times between 0.7ms and 25ms compared to a minimum of 385ms for any RankLib-based algorithm, establishing it as a practically promising TCP algorithm for time-constrained CI environments.