Evaluating LLM-Generated JUnit Tests with a Controlled Mutation-Testing Pipeline

Sammanfattning

This study evaluates the effectiveness of Large Language Models in automated unit test generation for Java by asking multiple API-accessible models to generate test suites under controlled conditions. We analyze their performance across compilation success, structural coverage, mutation testing, and sensitivity to repository complexity. The results show that while stronger models achieve high final compilation rates, many generated test suites initially contain errors and require automated repair before they become usable. Furthermore, although higher structural coverage is generally associated with stronger mutation scores, the relationship is not perfectly aligned, reinforcing the need to evaluate multiple quality metrics together. Compared to prior work focusing on single tools, our study highlights variability in model performance across different complexity levels and demonstrates that more advanced models are both more effective and more stable.