Generating Formative Feedback for Beginner Programming Assignments Using AI : Comparing Single-Prompt and Static Analysis-Supported Multi-Step LLM Feedback

Sammanfattning

Providing meaningful feedback on programming assignments is important for student learning, but it can be time-consuming for instructors, especially in larger courses. Existing automated approaches often focus on correctness rather than ex- planatory and constructive formative feedback. This thesis investigates how large language models (LLMs) can be used to generate formative feedback for beginner programming assignments. Two approaches are compared: a single-prompt LLM approach and a multi-step pipeline that combines static analysis with LLM-based feedback generation. The evaluation was based on two beginner-level JavaScript assignments. Gen- erated feedback was evaluated using predefined criteria: correctness, clarity, and actionable. Tool/model errors were reported separately from valid feedback out- puts. The results show that the pipeline approach was more reliable in the evalu- ated local-model setup, producing valid feedback for all evaluated outputs, while the single-prompt approach produced several tool/model errors. However, when only valid outputs were considered, the few valid single-prompt outputs received slightly higher quality scores than the pipeline outputs. Human reference feedback received the highest scores according to the rubric. The findings suggest that a structured pipeline can improve the reliability of LLM-generated formative feedback under local-model conditions, but it does not automatically guarantee higher feedback quality. The study highlights the impor- tance of prompt design, model capability, static-analysis configuration, and human review when using LLMs for programming feedback.