Comparing Retrieval Strategies in a Domain-SpecificRetrieval-Augmented Generation System

Sammanfattning

Retrieval-Augmented Generation (RAG) systems areincreasingly used to improve the accuracy of large languagemodels and reduce hallucinations by linking generation toexternal knowledge sources. However, most current studiesfocus on large knowledge bases and general domains, whilethe impact of different retrieval strategies within small-scale,domain-specific RAG systems remains insufficientlyunderstood.This study aims to evaluate the impact of three differentretrieval strategies TF-IDF, BM25, and Dense Retrievalwithin a small-scale RAG system dedicated to softwareengineering. It also aims to compare these systems with abaseline that relies on passing the entire corpus directly tothe language model without retrieval, to determine whetherretrieval is still necessary in small environments.A custom knowledge base was created, consisting ofapproximately 300 documents related to softwareengineering topics such as GitHub, Stack Overflow API,CI/CD, and OAuth. In addition, a set of questions andreference data was created, including direct, indirect, andrephrased questions.The RAG system was implemented using a native languagemodel via Ollama, and the systems were evaluated usingPrecision@K, Recall@K, and NDCG metrics to assess retrievalquality, as well as ROUGE-L and BERTScore to assess thequality of generated responses, with response time measuredfor each system.The results showed that Dense Retrieval performed bestacross most metrics, including retrieval quality and responsequality. The study also showed that the baseline based onpassing the entire corpus performed worst in terms of bothquality and response time, indicating that retrieval remains acritical component even within small knowledge bases.This study contributes to a better empirical understanding ofthe impact of different retrieval strategies within small-scaleRAG systems in the field of software engineering.