Local Retrieval-Augmented Generation System for Liquidity Risk Documents

Sammanfattning

Since their emergence, large language models (LLMs) have become a substantial part of personal and professional life. Their use in organisations that handle sensitive and domain-specific data is however constrained, both by confidentiality requirements and by the models' limited knowledge of the internal data. This thesis investigates the feasibility of addressing both constraints by developing a locally deployed Retrieval-Augmented Generation (RAG) chatbot for liquidity-risk documents at a financial institution. An end-to-end RAG-based chatbot was implemented against a corpus of 10 PDF documents, with tables and equations correctly recognised and indexed during parsing. The system was evaluated on two complementary question sets, a set of 75 short factual question–answer pairs and a set of 5 broader questions requiring multi-document reasoning. In total, 20 different pipeline configurations were tested. The best performing configuration achieved an accuracy of 93\% on the factual set evaluated using an LLM-as-a-judge procedure. On the reasoning set, the same configuration received an average grade of 2.5/5 from a panel of liquidity-risk experts. As an external reference the reasoning answers were also compared with answers produced by the Microsoft 365 Copilot system already deployed at the institution. M365 Copilot achieved the higher average score of 3.3/5, but the locally deployed chatbot outperformed it on two of the individual reasoning questions, which is a notable result given the limited project timeframe and the use of substantially smaller models. Regarding data security, the local chatbot has a clear advantage as all code and data are kept on internal servers. Overall, the findings suggest that a locally deployed chatbot operating against domain-specific documents can provide meaningful support and a potential efficiency gain. The system developed in this work is a first prototype rather than a production-ready deployment, but the results indicate that such a deployment is feasible. The proposed system could also be deployed in parallel with M365 Copilot, with the local chatbot reserved for the most sensitive data that cannot leave the organisation's own infrastructure. The thesis further discusses the limitations of the current approach and outlines directions for future work.