Augmenting Large Language Models with Domain-Specific Insight: Establishing SC-KAE Framework for Improved Real-World Application of LLMs in Supply Chain

Sammanfattning

Recent advancements in artificial intelligence (AI) and natural language processing (NLP) have transformed digital interactions through the emergence of large language models (LLMs). These models demonstrate impressive capabilities in generating human-like text and handling general-purpose queries. However, their application in specialized domains, such as supply chain management (SCM), remains challenging due to limitations in comprehending domain-specific terminology, handling complex data structures, and navigating unique operational contexts. Addressing these issues requires tailored solutions that augment LLMs with domain-specific knowledge. This research explores the integration of Knowledge Graphs (KGs) into Retrieval-Augmented Generation (RAG) pipelines to enhance the performance of LLMs in domain-specific tasks. Using SCM as a test domain, the study investigates how KGs can provide structured, factual context to improve the accuracy, relevance, and robustness of LLM-generated responses. The proposed framework combines the generative strengths of LLMs with the factual grounding of KGs. It incorporates semantic entity extraction, subgraph construction, and knowledge augmentation to create a unified context for reasoning. Two datasets are used to evaluate the approach: a novel SCM benchmark dataset covering eight core supply chain functions (such as procurement, inventory management, logistics etc.) and the LTU chatbot QA dataset. Performance is measured using standard metrics like ROUGE and METEOR, as well as truthfulness scores assessed by LLM-based evaluation. The study evaluated the performance of various models, including smaller open-weight models like Gemma3, Llama3.1, GPT-OSS, and Qwen3, alongside a larger, state-of-the-art model such as GPT-5 (Nano). Results demonstrated that KG integration enhanced performance compared to traditional RAG approaches, with smaller models achieving notable gains that reduced the performance gap with larger models. This underscores the potential of KGs to enable cost-effective and scalable LLM-based solutions by leveraging structured, domain-specific knowledge. While the results are promising, challenges remain. The accuracy of the system heavily depends on the completeness and quality of the KG. Future work will focus on optimizing KG construction, improving retrieval efficiency, and exploring the framework’s applicability to other domains like healthcare and finance.