Evaluating Modular Retrieval-Augmented Generation : A Case Study on the Språkrådet Language Advice Corpus

Sammanfattning

This thesis examines the integration of advanced Retrieval-Augmented Generation (RAG) modules within institutional settings demanding strict source-bounded accuracy. The primary research question addresses whether standalone advanced modules and a deterministic orchestrated pipeline improve retrieval metrics, context purity, and query refusal behavior compared to a basic RAG baseline. The methodology evaluates Corrective RAG (CRAG), Query Decomposition, and GraphRAG across standardized benchmarks (BIRCO) and a specialized Swedish language-advice corpus (Språkrådet), using an LLM-as-a-judge framework for end-to-end generative assessment. Results demonstrate that while high-dimensional dense embeddings establish a strong performance baseline, the CRAG module yields the most pronounced utility by systematically filtering out noise and raising context relevance. Conversely, topological GraphRAG underperforms in exact-passage extraction tasks. While orchestration does not produce a transformative leap in generative accuracy over individual standalone sub-components, it provides an auditable, step-by-step verification framework that achieves a 95% refusal rate on unanswerable queries. These findings indicate that structured context-filtering layers offer the necessary transparency and verification guardrails required for high-consequence institutional deployments.