Assessing TDD as a safeguard for LLM

Sammanfattning

Background. The integration of Large Language Models (LLMs) into software development has fundamentally altered how code is produced. LLMs allow code to be generated in seconds, however empirical studies consistently report that LLM-generated code frequently contain security vulnerabilities and code smells. At the same time, Test-Driven Development is a well established methodology shown to improve code quality and maintainability through its iterative Red, Green and refactor phase. Whether applying a TDD guided workflow to LLM-assisted code generationcan mitigate these security and quality issues remains unexplored. Objectives. This thesis investigates whether a modified TDD methodology can act as a safeguard against security vulnerabilities and code smells contained in code generated by an LLM. Specifically this study compares two development approaches, Standard prompting and TDD-guided prompting in regards to the presence, type and severity of security vulnerabilities and the presence and category of code smells. Ablation analyses further investigates whether developer prior experience and prompt quality can change the outcomes. Methods. A controlled experiment was conducted with 10 participants, the participants were quasi randomly assigned to either the Standard prompting group (SPG) or the Modified TDD group (MTG). All participants were instructed to build a Java webapplication. Security vulnerabilities were identified using OWASP ZAP v:2.17.0’s dynamic scanning, manual penetration testing and evaluation against OWASP ASVS. Code smells were detected with SonarQube v:26.4.0 and categorised. Mann-Whitney U test is used to statistically compare the two groups, and ablation studies are conducted based on the information collected in a post-task survey. Results. Neither vulnerability count nor code smell density differed significantly between the two treatments. The severity distributions of vulnerabilities were closely aligned, with the MTG accumulating two critical and six high severity findings compared to one critical and five high in the SPG. Unexpectedly, plain-text password storage occurred exclusively within the MTG group. This suggests that the constraints of the TDD ’Red’ phase may have overridden the LLM’s default security behaviour. The SPG completed more user stories on average than the MTG,reflecting the additional overhead imposed by the modified TDD process. The ablation analyses found no significant effect of prompt quality or developer experience on either dependent variable, but revealed a consistent crossover pattern across all three outcomes. As prompt quality increased, SPG outcomes improved while MTG outcomes worsened. Conclusions. Under the conditions of this experiment, the modified TDD approach did not reduce security vulnerabilities or code smells in LLM-generated Java code.The central contribution of this thesis is the demonstration that previously studied properties of TDD do not automatically transfer their quality and security benefits to LLM-assisted workflows. The findings raise a new concern that tests written in the TDD Red phase may constrain the LLM in ways that make the LLM forget its security defaults, this effect warrants a dedicated investigation in future works.