A Controlled Comparison of Sparse, Dense, and Hybrid Retrieval Strategies in Retrieval-Augmented Generation Systems

Sammanfattning

Retrieval-Augmented Generation (RAG) improves the factual grounding of Large Language Models by retrieving external documents and using themas context during answer generation. Since the retrieved context directly shapesthe final output, the choice of retrieval strategy is a central design decision in RAG systems, yet existing studies often evaluate retrieval in isolation orvary several pipeline components at the same time, which makes it difficult to attribute observed differences to the retrieval method alone. This thesis addresses that gap through a controlled experiment comparing sparse, dense, and hybrid retrieval in a fixed RAG pipeline, in which the dataset, chunking strategy, reader model, prompt structure, and evaluation procedure are held constant while only the retrieval method varies. The retrieval methods are evaluated on the NVIDIA TechQA-RAG-Eval dataset across retrieval effectiveness, final answer quality, and computational efficiency, with both mean scores and pairwise statistical tests reported. The results show that hybrid retrieval achieves the strongest retrieval effectiveness, with statistically significant improvements over BM25 on several metrics and smaller, partially significant differences from dense retrieval. At the final answer level, hybrid retrieval produces the highest mean similarity-based scores, but most differences are not statistically significant under the evaluated sample size, and faithfulness, which is evaluated through an LLM-based scoring step using the same model that generates the answers, does not differ significantly across the three methods. BM25 is the fastest method at the retrieval stage, but this advantage does not translate into a significant improvement in total response time, since answer generation dominates the end-to-end runtime. Taken together, the findings indicate that no single retrieval strategy is uniformly best and that retrieval choice should be guided by whether the application prioritizes retrieval effectiveness, answer similarity, grounding, or efficiency.