Optimising Multimodal AI La-tency on Android : A Modular Pipeline Approach

Sammanfattning

Real-time multimodal AI interaction on mobile devices is constrained not by model capability, but by the data-processing bottleneck at inference time. This thesis investigates how a modular, edge-assisted pipeline can reduce Time To First Token (TTFT) in conversational, multimodal large language model applications on Android, addressing the core question: how do data reduction strategies and the choice between local and remote model execution affect latency? To explore this, the Android application TripleEM was developed using Flutter, and benchmarked across 12 configurations combining three image resolutions (1080p, 480p, 224p), two Voice-to-Text (VTT) execution modes (on-device vs. remote), and Region of Interest (ROI) detection toggled on or off. Tests were conducted on 100 images from the Microsoft COCO dataset, with TTFT as the primary metric and classification accuracy evaluated using an LLM-as-a-judge approach. Results show that the dominant driver of latency is VTT engine placement. On-device transcription via Whisper Tiny averaged 2019 ms, slower than the total TTFT of any remote VTT configuration, making it the single largest bottleneck. Image downscaling meaningfully reduced upload time and backend inference cost with no measurable accuracy degradation, even at 224p. ROI detection added minimal processing overhead (under 50 ms) but did not consistently reduce upload time, and in some cases harmed accuracy by discarding relevant content. The fastest configuration achieved a mean TTFT of 408 ms, meeting the 500 ms real-time threshold, while the slowest exceeded 3800 ms. An unexpected finding was a modality bias in the remote multimodal model: configurations offloading VTT to the backend produced lower visual classification accuracy, suggesting the model may over-attend to audio input during joint inference. The study concludes that downscaling combined with remote VTT offers the best latency, while on-device VTT is preferable where accuracy is critical.