From Base Model to Maintenance Agent: Designing and Evaluating LLM Assistants for Industrial Settings

Sammanfattning

This thesis investigates the design and evaluation of Large Language Model (LLM) agent assistants for industrial maintenance, developed in cooperation with SSAB. The study focuses on identifying what combinations of tools and architectures best serve the needs of maintenance technicians, with an emphasis on practical usability and factual correctness. Multiple agents were developed and compared, incorporating combinations of Retrieval-Augmented Generation (RAG), internet search, direct document upload, and numerical database retrieval from a simulated Computerized Maintenance Management System (CMMS). All agents were built on the ChatGPT-4.1 base model. Evaluation was carried out through LLM-as-a-judge tests and a user survey involving subject matter experts and non-experts, analysed using nonparametric statistical methods and the Total Survey Error framework. Results indicate that the combination of RAG with preprocessed data, limited internet search, and direct UI document upload yields the most effective performance for maintenance-related queries. Agents equipped with CMMS database retrieval via a Knowledge Base architecture performed poorly, suggesting that LLM-based retrieval logic is insufficient for reliable numerical reasoning in this context. The survey provided no significant evidence that increased functionality degrades the structural quality of agent responses. In conclusion, this thesis demonstrates that a multi-tool LLM agent can provide meaningful support in an industrial maintenance context, and establishes a foundation for future work, particularly the adoption of a Model Context Protocol-based architecture for more robust numerical data access.