Evaluating the Efficiency, Effectiveness, and Completeness of GPT-Generated Information Security Policies

Sammanfattning

This paper evaluates the efficiency, effectiveness, and completeness of information security policies (ISPs) generated by OpenAI’s GPT-4 model compared to those crafted by human experts, focusing on Small and midsize enterprises (SMEs) and smaller public organizations. The study reveals that GPT-4 can generate ISPs that closely match the quality of expert-generated ones, demonstrating no significant difference in efficiency, effectiveness, and completeness. The comparative analysis and double-blind evaluation by an expert panel suggest that employing GPT-generated drafts as a preliminary step, followed by expert auditing and customization, could be a viable strategy for organizations, mainly due to the time-consuming and costly nature of developing ISPs. Furthermore, our results highlight the potential applicability of using GPTs to generate ISPs in Swedish, broadening the usability of AI in crafting security policies across different languages. However, while GPT-4 can produce initial drafts efficiently, the study indicates a need for these AI-generated documents to undergo a thorough review by information security experts to ensure they meet specific organizational requirements and keep pace with evolving cyber threats. This approach promises a novel and cost-effective method for SMEs and smaller public organizations to develop robust information security frameworks.