Enhancing Thematic Analysis with Large Language Models: A Comparative Study of Structured Prompting Techniques : Leveraging Large Language Models to Automate and Enhance Inductive Thematic Analysis in Qualitative Research

Sammanfattning

Thematic analysis, while crucial for qualitative research, becomes challenging with large datasets due to the intensive manual effort involved. This thesis explores the capacity of Large Language Mod- els (LLMs) to automate and enhance this process through different prompting techniques, benchmarking their performance against human-generated analyses. In this thesis we compare three distinct approaches: (1) Zero-Shot prompting, a single-prompt solution relying solely on pre-trained knowledge; (2) Top- Down Structured Prompting (TDSP), leveraging prompt engineering principles by starting with broad themes and progressively refining analysis; and (3) Bottom-Up Structured Prompting (BUSP), mirroring the established six steps of human thematic analysis designed for efficient extraction of nuanced insights. Applying these techniques to three diverse qualitative datasets, we used a rubric-based scoring system to evaluate the outputs against findings from the original research. Our results demonstrate that structured prompting techniques, particularly TDSP, significantly outperform the Zero-Shot approach in accurately capturing key findings. On average, 86% of findings identified by human researchers were at least par- tially identified in the outputs generated using structured prompting techniques. We also explored using an AI-based evaluator for rubric scoring but found the results unreliable, indicating that human evalua- tion remains crucial. This study makes a compelling argument that AI can efficiently perform thematic analysis and, especially in the future, provide substantial support to qualitative researchers in their work.