Domain-Adaptive Language Models for Government Auditing : Fine-Tuning and Agentic Retrieval-Augmented Generation

Sammanfattning

Retrieval-Augmented Generation (RAG) is a question-answering method that combines information retrieval with the generative capabilities of Large Language Models (LLMs). By referencing specific text passages, RAG answers can be efficiently validated, reducing the concern for incorrect or fabricated responses. Previous research has proposed fine-tuning LLMs on domain-specific data and introducing agentic behavior to enable models to plan and act autonomously as methods to improve RAG performance. This thesis investigated whether fine-tuning LLMs and introducing agentic behavior can improve the performance of a RAG system in a Swedish publicsector auditing setting. To address this question, a pragmatic case study was employed, combining quantitative and qualitative data to evaluate an implemented system. An agentic RAG pipeline was developed using a Thought-Action-Observation loop, enabling multi-hop capabilities, document retrieval, and user clarification. Synthetically generated data derived from domain-specific documents was used to fine-tune three LLMs: Viking-7B, Llama-3.1-8B, and GPT-4o mini. The implemented agentic RAG system, powered by GPT-4o mini, was evaluated against a basic RAG baseline and a human expert through a blinded human evaluation conducted at the Swedish National Audit Office. The results reveal a statistically significant preference for the basic RAG baseline over the agentic fine-tuned system in terms of answer accuracy and user experience. Human expert answers were also preferred over the agentic system, although a statistically significant difference between expert and basic RAG preferences could not be established. These results indicate that finetuning, combined with agentic actions, in the studied setting resulted in lower answer quality. The results highlight the importance of empirically evaluating RAG implementations and provide insights into fine-tuning and agentic approaches for domain-specific question answering.