Designing a short-form evaluation instrument for analog cybersecurity awareness games

Sammanfattning

Cybersecurity awareness training increasingly relies on analog games such as card games and tabletop exercises to engage non-expert audiences in interactive learning. Despite the growing use of these formats, no validated short-form evaluation instrument exists that is specifically designed to measure their effectiveness in practical training settings. Existing instruments are either too long for deployment in short sessions, developed exclusively for digital game contexts, or used without formal validation. This project addressed that gap by developing and pilot-testing a short-form evaluation instrument for analog cybersecurity awareness games. The project followed a four-phase sequential design. A structured scoping review of 21 peer-reviewed papers confirmed the research gap and identified six constructs for inclusion: perceived knowledge gain, self-efficacy, behavioural intention, enjoyability, engagement, and perceived usability. An evidence-based instrument design phase produced a seven-item Version 1.0 questionnaire, with each item adapted from validated measures identified in the literature. Face validity was tested through cognitive interviews with four participants using the think-aloud method, resulting in one substantive revision to produce Version 2.0. A feasibility pilot study was then conducted across two sessions in Sweden and India, involving 60 participants who played the Elevation of Privilege card game before completing either the short-form instrument or a 16-item comparison version in a between-subjects design. The short-form instrument met all four pre-specified feasibility thresholds: a median completion time of 41 seconds (SD = 9 seconds), a completion rate of 100%, an item nonresponse rate of 0.9%, and a perceived burden rating of Just right from 90% of participants. An exploratory internal consistency analysis yielded a Cronbach’s alpha of 0.74, meeting the commonly used threshold for acceptable reliability in earlystage instrument development. These findings provide initial evidence that a shortform questionnaire of seven items can achieve practical usability while demonstrating acceptable face validity and internal consistency in the context of analog cybersecurity awareness game evaluation.