An Analysis of Generative AI Capabilities in Security Testing : Evaluating Static Code Analysis Performance

Sammanfattning

Background – In today's technology-driven world, ensuring the security of software systems is paramount due to increasing dependency on these systems across all sectors. Security testing, specifically static code analysis, plays a vital role in detecting vulnerabilities before they are exploited. Traditional static analysis tools, such as SonarQube, often struggle to detect complex vulnerabilities, prompting the exploration of Artificial Intelligence (AI) for enhanced security testing. Objectives – This thesis aims to evaluate the performance of two Generative AI models, ChatGPT and Gemini, in static code analysis for security testing and compare these AI models with each other and with a traditional static code analysis tool, SonarQube, to determine their effectiveness in detecting software vulnerabilities. Methods – Method used in this thesis is experimentation which enabled me to gather empirical evidence through a controlled environment with controlled variables. It enabled me to compare the performance of ChatGPT, Gemini & SonarQube, this comparison also helped me in identifying a superior performing model. Results – Both AI models outperformed SonarQube in vulnerability detection. ChatGPT demonstrated slightly better performance in identifying the specific code responsible for vulnerabilities compared to Gemini. Conclusions – Through the course of this thesis it has become evident that GenAI models offer solid performance when it comes to static code analysis in vulnerability assessment. They show promise and have presented their case by showcasing their superior performance, that they are very much able to assist or even replace the traditional SAST tools in some scenarios.