Exploring LLM Performance For Vulnerability Mitigation

Sammanfattning

With software systems becoming larger and more complex, traditional manual approaches to detecting and correcting security flaws are no longer efficient. On theother hand, LLMs have emerged as a promising solution for automated vulnerability repair; their reliability in security-critical workflows remains a significant concerndue to their non-deterministic nature and potential to introduce new vulnerabilities.Current research is often limited by a lack of standardized benchmarking and systematic comparisons across multiple models using real-world datasets.This thesis addresses these gaps by methodically evaluating the effectiveness of ninemodern LLMs, including code-specialized models like StarCoder2-15B and generalpurpose models like Phi-2 in generating secure and functionally correct code fixes.Utilizing the CrossVul dataset, which provides paired samples of vulnerable andfixed code across different languages: C, C++, Java, and Python, we investigatehow model performance varies across different programming languages and vulnerability types by four different zero-shot prompt categories: Basic, Role-based,Constraint-Based, and Comprehensive.To evaluate the performance of the selected models, our methodology combinesseveral quantitative metrics, like CodeBLEU, AST Edit Distance, and ROUGE tocompare the model-generated code with the ground truth, and further supports theseresults through manual analysis of the generated code. By conducting repeatedexecutions to measure consistency, this study aims to provide an evidence-basedframework for choosing reliable LLM-based agents. Ultimately, the findings willoffer practical guidance for software security professionals and contribute to theacademic discourse on leveraging AI for safer software development.