MCP-Based Model Routing for Efficient Computer Vision : A Multi-Domain Evaluation

Sammanfattning

The growing computational cost of large vision models has intensified interest in mod-ular inference architectures that can reduce GPU memory consumption without sacrificingtask performance. This thesis investigates the viability of Model Context Protocol (MCP)as a dynamic dispatch layer for computer vision, proposing and evaluating a two-phaserouting pipeline as an alternative to monolithic vision-language model inference.The proposed system uses a lightweight host LLM (Qwen 3.5-2B) to first invoke animage classifier (ViT-Base-Patch16) via MCP, then route to one of three domain-specialiseddetection models based on the returned label: YOLO-Face for face/human centric images,Table-Transformer for document tables, and OWLViT as a general-purpose default. This iscompared against a monolithic baseline, Qwen3-VL-4B, which performs classification andlocalisation in a single forward pass. Both systems are evaluated on 8,674 images drawnfrom three benchmark datasets spanning distinct visual domains: COCO val2017, WIDERFACE, and PubTables-1M. Performance is measured across four dimensions: VRAM con-sumption, end-to-end latency, classification accuracy, and object localisation via IoU (Inter-section over Union).The routing system achieves a mean VRAM consumption of 2.80 GB against the base-line’s 7.05 GB, a 60% reduction, with a peak of 5.71 GB remaining well below the baseline’speak of 13.15 GB. Contrary to expectations, the routing system also demonstrates substan-tially lower latency, averaging 8.14 seconds per image against the baseline’s 33.48 seconds,with a significantly tighter distributional spread (P95: 9.33s vs 64.17s). Overall accuracyis comparable between the two systems at approximately 54.4% label match and 49% IoUsuccess, but the domain-level breakdown reveals a complete inversion. The routing systemsubstantially outperforms the baseline on face and table detection while underperformingon general object detection, where vocabulary misalignment between the ImageNet clas-sifier and COCO annotations limits zero-shot detection quality. A secondary retry config-uration, in which the pipeline invokes another of its specialist tools when no boundingboxes are returned, improved IoU success to 69.1% at a mean latency of 14.52 seconds,remaining approximately twice as fast as the monolithic baseline and suggesting a viableaccuracy-latency trade-off.These results support a cautiously positive assessment of MCP-based routing as a prac-tical architectural pattern for domain-specific computer vision deployments, demonstrat-ing that modular specialisation can deliver meaningful efficiency gains while matchingaggregate accuracy, at the cost of reduced generality.