Technical and Organizational Effects of Retrieval-Augmented Generation for Math Tutoring

Sammanfattning

While large language models (LLMs) have proven useful in many areas, they still have the disadvantage of sometimes hallucinating and providing incorrect or illogical answers. In areas like mathematics tutoring, faults like these are detrimental to the usability of LLM systems. Retrieval-augmented generation (RAG) is a method to combat these drawbacks. With RAG, an external database containing information relevant to the system’s area of application (e.g. a math textbook) is paired with the LLM. User prompts are combined with relevant documents from the external database, and these combinations are used as the final prompts to the LLM, thereby giving it access to the added context of the retrieved documents. In this article, we implement RAG for online elementary and high school mathematics tutoring to answer the question of whether RAG will bring improvements to LLM responses in this area. We also assess the attitudes toward using artificial intelligence at work among a group of tutors at an online mathematics tutoring organization, and what kind of effects RAG could have on these attitudes. Data to answer these research questions was collected via a survey sent to the tutors at the tutoring organization. Our results show that RAG had an insignificant effect on the assessed quality of LLM responses. However, RAG might be valuable from an organizational perspective as it brought (admittedly statistically insignificant) improvements in many important metrics. We theorize that the lackluster effects of RAG are explained by the LLM not needing the additional context supplied by the retrieved documents.