Design and evaluation of a RAG-based student assistant for navigating university information resources

Sammanfattning

Universities manage large volumes of heterogeneous administrative information, in-cluding regulations, schedules, and procedural guidelines, which are often difficultfor students to access efficiently through traditional keyword-based search or staticweb navigation. Recent advances in large language models offer flexible conversa-tional interfaces, but purely generative systems suffer from hallucinations and lackfactual grounding, making them unsuitable for administrative domains that requirehigh accuracy. This thesis investigates the design and evaluation of an assistant thatcombines information retrieval from university information resources with language-model–based generation. The system follows a Retrieval-Augmented Generation(RAG) approach to support accurate university administrative information access,with a focus on improving retrieval precision while maintaining scalability. A hy-brid retrieval architecture is proposed that combines semantic vector search withkeyword-based filtering, allowing the system to both understand the meaning ofuser queries and precisely match key administrative terms, which addresses commonentity-level lookup problems in academic administration. The system employs mul-tilingual sentence embeddings, a vector database with approximate nearest-neighborsearch, and dynamic query analysis to ground generated responses in authoritativedocuments. The proposed approach is evaluated on a curated benchmark of 30 repre-sentative student queries across 6 administrative categories using standard informa-tion retrieval metrics and the RAGAS framework. Evaluation focuses on two aspects:the performance of document retrieval, quantified using Precision@k and Recall@k,and the quality of generated responses, assessed through faithfulness and answerrelevance. Experimental results demonstrate that the hybrid RAG strategy signifi-cantly outperforms the university’s existing search solution, increasing Context Re-call from 0.37 to 0.98, and improves answer faithfulness to 0.93 while maintaininghigh retrieval coverage. These findings indicate that a schema-free hybrid architec-ture provides an effective middle ground between pure semantic retrieval and highlystructured knowledge-based systems.