Batch Hierarchical Inference for Edge Computing : Implementation of Batch Hierarchical Inference on the ExPECA testbed

Sammanfattning

Hierarchical Inference (HI) is an emerging strategy in edge computing that aims to balance inference accuracy with system efficiency by distributing Machine Learning (ML) tasks between resource constrained Edge Devices (EDs) and more powerful Edge Servers (ESs). Traditional HI frameworks typically process samples individually and rely on immediate offloading of uncertain predictions to the ES, which can result in excessive communication overhead and energy consumption, especially under constrained network conditions. This thesis investigates whether batch processing, aggregating multiple inference samples before offloading, can improve the performance of HI systems without sacrificing accuracy or incurring unacceptable latency. The central research question addressed is: How does batch processing influence HI in terms of system accuracy, latency, throughput, and communication efficiency in real world edge computing environments? To answer this, a two-tier Batch HI framework was implemented and deployed on the ExPECA testbed at KTH, enabling rigorous evaluation under realistic edge conditions. The results demonstrate that batch processing significantly reduces communication overhead and improves system throughput, while maintaining inference accuracy comparable to individual offloading. Specifically, a configuration combining adaptive threshold based offloading with a batch size of fifteen achieved near optimal accuracy, doubled throughput, and reduced offloading transmissions by over 80% compared to per-sample offloading. These findings indicate that batch processing, when carefully configured, provides a practical and effective enhancement to HI strategies in edge computing.