Enhancing performance of LLM-Based Problem-Solving RCA Chatbot in Technical Investigations

Sammanfattning

Root Cause Analysis (RCA) is a critical but time-intensive activity in industrial engineering. Generally, investigations rely on large volumes of noisy and unstructured documentation knowledge. Recent advances in Large Language Models (LLMs) offer promising opportunities to support RCA workflows. Their effectiveness is strongly influenced by the quality of retrieved information and the way the instructions are formulated. This thesis investigates how the performance and reliability of LLM-based root cause analysis systems can be improved through knowledge graph guided retrieval and prompt engineering when applied to noisy, unstructured industrial data. In this study, two chatbot architectures were evaluated: a baseline system using standard vector based retrieval and an enhanced architecture incorporating knowledge graph guided retrieval to narrow the retrieval corpus through structured domain relationships. In addition, different prompting strategies: simple prompting, Chain of Thought (CoT) and Persona prompting were examined. A qualitative, human centered evaluation was conducted with Subject Matter Experts (SMEs) using Likert-scale ratings, interviews and pairwise comparisons analyzed through Bayesian Bradley-Terry model. The results indicate that knowledge graph guided retrieval improves contextual grounding and consistency of responses, reducing inter user disagreement, while persona prompting produces the clearest and most diagnostically useful RCA outputs when combined with stable retrieval architecture. Overall, the findings demonstrate how prompt engineering and retrieval strategies enhance the reliability and practical usefulness of LLM assisted RCA in industrial investigation workflows.