Evaluating the Usability of Large Language Models as Tools in Cybersecurity : A Comparison of Censored and Uncensored Models in Penetration Testing and Digital Forensics

Sammanfattning

In today’s age, Large Language Models (LLMs) are becoming more prevalent in almost all fields of work. Both good and bad actors in the cybersecurity domain are leveraging artificial intelligence to gain strategic advantages. Existing research does not investigate the usability of LLMs in penetration testing and digital forensics, particularly in the context of censored and uncensored models. This thesis aims to evaluate the usability of a censored and an uncensored LLM and explore the impact censorship has on usability. Two LLMs, ChatGPT-4 and Dolphin Mistral 7B 2.2.1, were selected for evaluation. They were tested across varying difficulty levels in capture-the-flag and digital forensic challenges. Usability assessments were conducted using a Likert scale, alongside an analysis of the proportion of accurate responses to prompts. In the capture-the-flag challenges, the models aided in discovering two out of three flags, demonstrating favorable usability for ChatGPT-4, while Dolphin Mistral fell short of meeting the specified usability threshold. In the digital forensics challenges, the LLMs helped answer a majority of the questions, with ChatGPT-4 outperforming Dolphin Mistral in terms of usability. Our results show that both models could be useful when applied in various contexts. The censorship did not impact the usability, but the dataset, hardware, and knowledge of the user significantly influenced performance. However, more research is needed for a definitive conclusion.