Exploring Heuristics for Predicting Microbenchmark Stability and Code Coverage using Static Code Analysis

Sammanfattning

Performance testing is a method to optimize performance and identify regressionsin software applications. This method can be employed through microbenchmarks,which measure the performance of a small unit of code. However, writing accuratemicrobenchmark tests is difficult as they have a high need for precision. Toolshave been developed to automate the creation of microbenchmarks, but the methodthey implement results in large benchmark suites and thus also long running timeswhen executing the suites. A solution to this problem is to only select a subsetof the benchmarks to reduce the execution time of the whole suite. The goal ofthis thesis was thus to explore heuristics for selecting benchmarks with high stabilityand/or high code coverage; stability and code coverage are two importantproperties of benchmarks that are useful for detecting performance regressions. Alaboratory experiment was conducted to explore two heuristics: firstly, a suitableheuristic for predicting the stability of microbenchmarks using only code featuresfrom static code analysis; secondly, a heuristic for a suitable approach for combiningthe stability of benchmarks with their code coverage. The experiment used 2250JUnit tests from three open-source projects by converting them to benchmarks withthe tool ju2jmh. Data from these benchmarks was used to design the heuristics.The first heuristic was created through regression models, where four separate candidateswere explored. The model with the best performance was a Random Forest,gaining an R2 value of 0.214 and a mean absolute error (MAE) of 3.491. This indicatesthat it performs better than always predicting the median value, but is stillof low explanatory power. The second heuristic was designed using average rankaggregation and showed promising results. A balance was achieved where strengthsin either stability or code coverage compensated for lesser performance in the other.