Impact of Generative AI on Systematic Literature Review Task

Sammanfattning

Generative AI tools have rapidly transformed academic systems, however their impact on the learning outcomes of graduate students conducting systematic literature reviews remains challenged. Building on Kosmyna et al.'s (2025) neurophysiological findings that large language models induce lower cognitive effort while reducing learning skills, this thesis investigates how AI assistance affects the quality, workload, and knowledge retention of systematic literature reviews in academic work through a crossover quasi-experimental study conducted in a natural academic setting. Two graduate students from different educational backgrounds (HR Management and STEM) each completed two systematic literature reviews, one using AI-assisted methods (ChatGPT, Gemini) and one using fully manual approaches-across distinct research topics. This quasi -experimental design in a field setting provides a realistic academic setting that more closely reflects authentic student work than laboratory-based experiments while maintaining methodological rigor. Quality was assessed via peer-reviewed rubrics, Workload was assessed using NASA-TLX survey responses and detailed activity logs collected throughout the study and learning via pre/post knowledge quizzes and oral assessments conducted two weeks post-completion to measure durable retention. The results showed clear differences between AI-assisted and manual review approaches. For quality (Research Question 1), AI-assisted Systematic Literature Review(SLR)s achieved higher scores (23.5 out of 30 vs 20.5 out of 30,14.6% increase), excelling in synthesis and structure but showing citation accuracy deficits. For workload (Research Question 2), AI significantly reduced perceived cognitive load (NASA-TLX: 5.6 out of 10 vs 8.6 out of 10, 35% reduction), though total time remained similar (71h vs 70h). For learning outcomes (Research Question 3), manual reviews produced superior knowledge retention, with oral assessment scores 27% higher (5.5 out of 10 vs 4.0 out of 10), particularly on causal understanding and confidence calibration. These findings reveal that quality showed AI greater than or equal to Non AI, workload showed AI less than or equal to Non AI in time despite lower perceived effort, and retention definitively showed Non-AI greater than AI.