Difficulty-Aware Query Routing : Balancing Cost and Accuracy Across Language Models via Semantic Routing

Sammanfattning

The growing deployment of large language models (LLMs) poses challenges regarding application programming interface (API) costs and energy consumption. Not all queries require the full capability of an LLM; many can be adequately answered by smaller, locally hosted small language models (SLMs). This paper addresses the problem of routing queries to the appropriate model by proposing a difficulty-aware semantic routing framework. We introduce an ensemble-based annotation strategy that leverages multiple SLMs to estimate query complexity using two proposed methods: a classification method and a regression method, thereby reducing the bias of individual SLMs. Query embeddings are used to train machine learning (ML) models on the annotated data from open-source datasets. The output of this ML model feeds into a Bayesian decision router guided by a utility function that jointly optimizes for answer quality, latency, and query cost. We evaluate our routing framework against three baseline alternatives: using only an SLM, using only an LLM, and an existing alternative, the vLLM Semantic Router, across benchmark datasets spanning multiple domains. Results show that our proposed router navigates a configurable cost-latency-quality trade-off space across a range of reasonable configurations. Our router shows the potential efficiency gains from specific config compared to LLM-only, SLM-only, and the vLLM Semantic Router. However, our study further shows the difficulty of routing by question difficulty. We further analyze the impact of misrouting, characterizing missed and failed escalations in terms of accuracy, latency, and cost relative to an oracle policy, and discuss sensitivity to utility parameterization.