Evaluating Persistent Vector Memory for Enterprise Conversational LLMs

Sammanfattning

Large Language Models (LLMs) demonstrate strong reasoning and language capabilities but remain limited in their ability to maintain operational state across multi-turn interactions. This limitation is particularly problematic in enterprise environments, where conversational systems must track evolving information over time. This thesis investigates whether a persistent vectorbacked memory architecture can improve state tracking in agentic AI systems and evaluates its effectiveness across both proprietary cloud-hosted models and locally deployed Small Language Models (SLMs). A controlled quantitative experiment was conducted using a synthetic enterprise logistics dataset comprising 500 multi-turn conversational sessions. The evaluation framework compared memory-augmented agents against stateless baselines under conditions with and without access to conversational history. Performance was assessed across storage efficacy, retrieval accuracy, temporal robustness, orchestration behaviour, and operational latency. Ten language models, including the GPT-5 family, Llama, Qwen, Gemma, Hermes, and Mistral, were evaluated within a common agentic architecture supported by a Qdrant vector database. The results demonstrate that persistent memory substantially improves the ability of conversational agents to maintain and recover operational state when compared with stateless configurations. However, the study also reveals that successful memory storage does not guarantee successful memory utilization. While storage performance remained consistently high across most models, retrieval and reasoning over retrieved information emerged as the primary bottlenecks. Furthermore, the findings show that semantic interference caused by conflicting information has a greater impact on performance than conversational distance alone. The results also indicate that model architecture and tool orchestration capabilities influence overall memory performance more strongly than model scale in isolation. The study contributes a reproducible evaluation framework for memory-augmented agents and provides empirical evidence regarding the strengths and limitations of persistent vector memory in enterprise-oriented conversational systems. The findings offer practical guidance for organisations considering the deployment of memory-enabled AI agents in environments where reliable state tracking, data sovereignty, and operational efficiency are critical.