Developing a RAG system for automatic question generation : A case study in the Tanzanian education sector

Sammanfattning

Retrieval-augmented generation is typically employed for question-answer systems, wherein relevant information is retrieved from a knowledge source to help a language model answer a user query. This thesis presents a use case for this technology in education through a field study in Dar es Salaam, Tanzania. In cooperation with the Tanzania AI Community, we conducted interviews and observation studies with teachers in three local secondary schools to investigate the possibility of a technology intervention. The key findings of this study are the prevalence of WhatsApp as a medium of communication for teachers, the emphasis on continuous student assessment, and the presence of a competency-based curriculum in classroom settings. As a result, we propose Twiga, a chatbot system integrated with WhatsApp where teachers can request questions or exercises grounded in their course curriculum. Twiga’s exercise generation pipeline introduces a novel retrieval-augmented generation design built to produce contextually relevant and pedagogically accurate exercises. Due to the unique task of this system, we provide a comprehensive comparison of several pipelines. The review includes a curated set of automatic metrics, a multi-dimension grading survey, a pairwise comparison survey, and an ablation study. After integrating a selected pipeline with a WhatsApp chatbot, Twiga was trial-tested by two Tanzanian teachers. We discuss the results of this trial and reflect on the implementation factors relevant to the Tanzanian educational context.