On the evaluation of AI-driven code and test generation agents : AI Agents

Sammanfattning

Background: AI coding tools are widely used in software development, but the code they produce often has problems that functional tests do not catch, such as security flaws, poor maintainability, type errors, and missing documentation. Objectives: This thesis investigates how AI quality agents can check code quality by running existing software development tools and organising their output into useful feedback. Methods: Two systematic literature reviews were conducted, covering 104 papers on code quality and 95 papers on testing. These reviews identified 191 quality metrics, 161 testing metrics, and 119 workflow patterns. A 16-metric evaluation framework was built, linking four quality dimensions (complexity, code quality, security, reliability) to 12 open-source Python tools with documented thresholds. A benchmark of 95 Python files, each containing 3 to 4 deliberate quality issues, was created for evaluation. Eight agent variants were compared using a 2 by 4 factorial design: single-pass versus iterative execution, crossed with four knowledge sources (brain-only, papers RAG, guidelines RAG, dual RAG). All agents ran on a 9-billion parameter model on a single GPU. Results: The single-pass Papers RAG agent performed best, with an F1 of 0.507, recall of 64.5 percent, and precision of 41.7 percent. RAG improved F1 by up to 12.7 percent over brain-only. Single-pass agents outperformed iterative agents (F1 0.488 versus 0.428) because the model treated the review loop as a completeness check rather than a gap-finding exercise. Security patterns were detected at 98 percent recall, while type errors were detected at only 10 to 20 percent. Conclusions: On a small model, RAG shifts attention rather than purely adding knowledge. Single-pass execution is more effective because the model cannot judge when to stop adding metrics. The model detects visible code patterns well but struggles with problems defined by absence.