Towards Efficient LLM Deployment in Customer Service : An Exploratory Evaluation of a Quantized LLM

Sammanfattning

The improved performance of Large Language Models (LLMs) presents opportunities for automating customer support. However, the substantial capital and operational costs associated with state-of-the-art (SOTA) LLMs, whether accessed via API or self-hosted, create a significant barrier to entry, particularly for Small and Medium-sized Enterprises (SMEs). This thesis explores the viability of a more cost-effective approach: deploying a smaller, quantized, open-source LLM locally for first-line customer support tasks. A customer support pipeline was developed using LangChain and LangGraph, integrating a 4-bit quantized Qwen2.5 14B model. This system, featuring Retrieval-Augmented Generation (RAG) capabilities drawing from company-specific data (FAQs, documentation, support history), was evaluated against a SOTA benchmark (GPT-4.1 via API) in a simulated first-line customer support scenario within the same pipeline. Response quality was assessed through human evaluation (factual correctness, helpfulness, clarity, tone) and automated metrics (BERTScore), alongside a blind A/B preference test conducted with customer support professionals. The study also aimed to identify key technical and system-level factors influencing the observed response quality of the self-hosted solution (RQ2). Results indicate that the quantized, self-hosted LLM, when supported by the well-designed pipeline, can match and even outperform the SOTA benchmark in aspects such as factual correctness and helpfulness for the low-complexity, single-turn queries typical of first-line support. Key factors identified (RQ2) as critical for this performance, beyond model size and quantization, include the efficacy of the RAG pipeline, the "LLM-readiness" of the knowledge base, and the degree of control over system-level prompting. In conclusion, this study indicates that a smaller, quantized, self-hosted LLM, when integrated into a RAG pipeline, can achieve comparable, and in specific aspects such as factual correctness and helpfulness, even superior response quality to a SOTA benchmark for low-complexity, first-line support tasks. The research further highlights that the efficacy of the surrounding system architecture, notably the RAG implementation and knowledge base quality are critical factors, beyond intrinsic model properties influencing this observed performance. These findings suggest a viable path for SMEs, though results are context-specific.