Evaluating Mixture-of-Experts Models for Federated Learning in Computer Vision

Sammanfattning

As the number of edge devices continues to grow, centralized machine learning (ML) faces substantial challenges. The sheer volume of data produced at the edge can make transfer infeasible, and the data itself is often sensitive, raising privacy concerns. Federated learning (FL) addresses both issues by training models directly on device without transferring raw data. However, deploying capable models at the edge requires balancing high capacity with computational efficiency. Mixture-of-Experts (MoE) models address this by dividing the network into specialized sub-networks called experts, with a gating network that activates only the subset of experts relevant to each input. This enables specialization across the data distribution, achieving high model capacity while keeping inference costs low. Whether these properties persist under the additional noise of FL is not well understood. This thesis investigates federated training of MoE vision models in which routing decisions are made per image patch. Model performance and expert specialization are assessed across independent and identically distributed (IID) and non-IID settings. Further, robustness to data heterogeneity is assessed under both label shift, where clients hold different label distributions, and covariate shift, where clients see the same classes under different visual conditions. Both evaluations use ImageNet-100, with a weather-augmented version serving as a separate dataset for the covariate shift setting. The results show that federated training at ImageNet-100 scale is feasible, with models reaching within 2–4\% of centralized counterparts. Expert specialization across classes, spatial locations, and to a lesser extent weather emerges alongside coherent routing strategies without specialized federated methods. However, in the federated setting a performance gap between MoE models and non-MoE baselines persists, and load balancing becomes unstable under label heterogeneity. These findings suggest that the core properties of MoE models transfer to federated settings, but closing the performance gap and handling label heterogeneity effectively will require novel methods.