Benchmarking Large Language Models for Vulnerability Detection: Comparing Local and Cloud LLMs

Sammanfattning

The number and complexity of modern software systems have increased substantially since their inception. This unrelenting growth is making manual code review more difficult year by year, and the gap between the amount of code and security verification creates a critical need for automated tools that can assist developers in detecting vulnerabilities. A possible solution to this is large language model driven vulnerability detection. This thesis investigates the possibility of utilizing locally fine-tuned LLMs in order to discover and flag memory related security flaws in C/C++ code. Five locally fine-tuned models have been examined and compared to each other, their non-fine-tuned versions, as well as proprietary cloud models. The models were fed functions taken from C/C++ projects, and were asked to determine whether the function in question was vulnerable. Two different prompting methods were used during the evaluation, zero-shot prompting and few-shot prompting. After each evaluation, performance metrics such as accuracy and F1-score were calculated. While fine-tuning enhanced the performances of the local models with respect to F1-score, their ability to detect vulnerabilities remained unsatisfactory. The highest performing model, CodeLlama 7B, achieved a F1-score of only 0.12. Cloud models are orders of magnitude larger in parameter size and have had more extensive pre-training. However, as the cloud models did not outperform this, it indicates that the methods utilized in the thesis were sub-optimal. The two prompting methods did not significantly impact the results of any model. Further research to improve model performance may include Chain-of-Thought prompting, Retrieval-Augmented Generation, or fine-tuning of the cloud models.