Benchmarking Machine Learning Techniques For A Generative AI Troubleshooting Assistant In Automotive Diagnostics

Sammanfattning

Background: Modern automotive workshops, production, and aftermarket operations increasingly depend on advanced off-board Diagnostic Communication Platform (DCP) to perform Electronic Control Unit (ECU) flashing, identify root causes, and guide repairs. Meanwhile, Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) enable the creation of troubleshooting assistants that analyse logs and use existing diagnostic knowledge. That said, it is not yet clear which machine learning techniques and configurations provide accurate and reliable support in this specialised setting. Objectives: The thesis benchmarks machine learning techniques for a generative AI-based troubleshooting assistant within an automotive diagnostic communication platform. Two research questions are addressed in this study: (i) which state-of-the-art techniques are applicable when using diagnostic knowledge bases and tool usage logs, and (ii) how these techniques compare for retrieval effectiveness and response quality. Methods: We focus on two primary tasks: identifying root causes from Unified Diagnostic Services (UDS) Negative Response Codes (NRCs) and performing question-answering using resolved incidents. It integrates domain expertise through layout-aware parsing of ISO 14229-1 NRC tables and structured incident descriptions that include problem, symptoms, root cause, and solution. A Structured and Domain-Adaptive RAG retriever, we combine different query representations and a cross-encoder reranking. The evaluation includes two base models - TinyLlama and an OpenAI model - and we evaluate in three configurations: plain or baseline LLM, LLM with Structured Domain-Adaptive RAG, and LLM with Structured Domain-Adaptive RAG and reranking. Retrieval effectiveness we measure using Top-k accuracy, and response quality is measured using cosine similarity, ROUGE-L, BLEU, Jaccard, BERTScore, and statistical tests such as one-way ANOVA and Tukey's HSD. Results: In the root-cause analysis, the highest Top-10 retrieval accuracy of 66.15% we obtain when NRC entries are encoded using concise semantic fields such as service name, service ID, NRC, and error type. More formal byte-level or key-value encodings underperform. For question-answering, applying structured query expansion to resolved incidents results in over 96% accuracy for Top-5 retrieval, with reranking primarily improving the top-ranked results. Regarding TinyLlama, Structured DomainAdaptive RAG and reranking significantly enhance both lexical and semantic similarity to synthetic target values. For the OpenAI model, Structured RAG consistently increases lexical overlap and semantic coverage, while additional reranking has a limited impact. ANOVA analysis rejects the null hypothesis of equal performance across all the metrics for OpenAI model. Conclusions: The thesis suggests that the most statistically significant decisions for a generative AI troubleshooting assistant are Structured Domain-Adaptive RAG and query expansion, rather than relying solely on the baseline LLMs. Smaller local models can perform well when combined with strong retrieval and reranking techniques, while larger, hosted models mainly benefit from structured grounding. These results offer practical guidance for design and an evaluation framework for future research on incorporating generative AI troubleshooting assistants into diagnostic communication platforms. This thesis also emphasises the importance of using real workshop, production and aftermarket data, along with human evaluation methods.