Evaluating Generative AI's Applicability in Software Testing : User study and quality evaluation of unit test generation

Sammanfattning

This thesis explores the current capabilities and perceptions of generative AI tools for software testing in an industrial environment, focusing on unit test generation. Despite the potential for generative AI to automate test generation and improve efficiency, the study reveals challenges related to output quality, compilability, and practical integration barriers. Using a commercial coding assistant, we generate unit tests for an industrial code base. We then evaluate the generated tests, revealing a compilation rate of 36.09% and emphasizing compilability as the primary obstacle in generating high-quality tests. To reveal the perceived benefits and challenges of using generative AI in software testing, we conduct interviews with engineers to gather qualitative data and perform a thematic analysis. The analysis reveals perceived benefits such as time savings, efficiency improvements, and organizational support and encouragement. It also highlights challenges related to security and data confidentiality concerns, validation requirements, and output quality and reliability. While engineers recognize the theoretical potential of generative AI tools in unit test generation, the adoption of such tools in industry remains limited, with some viewing it as a misapplication of the technology. This research contributes to more informed decision-making for companies considering integrating generative AI tools into their software testing workflows.