Retrieval-Augmented Generation for Enterprise Search : A Comparative User Study of RAG-Based and BM25-Based Information Retrieval Systems

Sammanfattning

Many employees at Sectra Communication, a Swedish company specializing in high-assurance communication, report difficulties finding information using the current enterprise search system. The company relies on an in-house application, Sectra Tools, to access Sectra.Doc and other organizational databases. Tools ranks documents using a BM25 function based on query-term occurrences in specific document fields. Retrieval-Augmented Generation (RAG) has recently emerged as a potential alternative to traditional search systems. In RAG, a large language model (LLM) is provided with information chunks retrieved based on semantic rather than purely lexical similarity. However, for organizations like Sectra that are interested in adopting a RAG-based enterprise search system, there is limited research comparing its utility to traditional systems. This work compares user performance and satisfaction when answering questions about information contained in Sectra.Doc using either Tools or a RAG-based information retrieval system. The comparison is based on a question–answer (QA) dataset designed to reflect typical employee information needs related to process descriptions, departmental specifications, and product documentation. A user test was conducted in which employees answered ten questions using either Tools or the open-source RAG engine RAGFlow, with the RAG system tuned on the QA dataset. User satisfaction was assessed using a Likert-style questionnaire. The results show that users answered questions 63 percent faster on average when using RAGFlow than Tools. No clear differences were observed in the truthfulness of user responses. Both search methods resulted in approximately 30 percent perfect answers, but RAGFlow produced a higher proportion of both acceptable and incorrect answers (26 and 32 percent, compared to 5 and 24 percent for Tools). These findings suggest that the suitability of a RAG-based system depends on whether an organization prioritizes faster access to partially correct information or slower retrieval with fewer incorrect responses. RAGFlow achieved a substantially higher average user satisfaction score than Tools (5.23 versus 2.28 on a six-point scale), indicating higher perceived system effectiveness, reduced user effort, shorter perceived search time, and increased trust in the retrieved information. The higher reported trust in information retrieved with RAGFlow is somewhat unexpected and potentially concerning, as the LLM essentially provides a secondary interpretation of documents accessible through Tools.