Development of an AI Evaluator for Usability Testing with Think-Aloud Protocol

Sammanfattning

The think-aloud protocol is a well-established method in usability testing in which participants verbalize their thoughts, reasoning, and decisions while interacting with a system. Although the method provides rich insights into users’ cognitive processes and encountered usability issues, it traditionally relies on a human facilitator to guide sessions, encourage verbalization, and manage task progression. This thesis presents the design, implementation, and evaluation of an AI-based evaluator intended to support think-aloud usability testing. A real-time pipeline was developed to capture microphone input, transcribe spoken utterances using the Whisper framework, and facilitate structured think-aloud sessions through standardized prompts delivered via Kyutai text-to-speech synthesis. The system was designed to partially automate session facilitation while preserving human responsibility for qualitative analysis. The proposed framework was evaluated in a user study involving eight participants performing predefined tasks in Microsoft PowerPoint using the think-aloud protocol. All sessions were completed successfully without direct human moderation, demonstrating the technical feasibility of AI-based facilitation. Post-session questionnaires indicated consistently positive participant perceptions regarding clarity, responsiveness, and behavioral consistency. The findings suggest that AI-supported facilitation can maintain procedural integrity and structured data collection in think-aloud usability testing. While the system does not perform automated usability analysis, it establishes a baseline framework for AI-assisted evaluation and provides a foundation for future research exploring more advanced or agentic extensions of usability testing workflows.