Real-time Object Detection for the Visually Impaired An On-Device Federated Learning Approach

Sammanfattning

Visually impaired people often face difficulties in recognizing objects around them, which can make everyday tasks harder and reduce independence. Advances in artificial intelligence and computer vision now make it possible to design systems that detect and describe objects in real time. The goal of this project was to create such a system with a focus on speed, efficiency, and privacy, so it could run directly on affordable devices without relying on cloud services. The model was trained on the VizWiz dataset, which contains photos taken by blind and low-vision users. A lightweight YOLOv8-small model was trained and fine-tuned, then optimized through model compression techniques, including pruning, knowledge distillation, and quantization. The final version was converted into TensorRT engines in FP16 and INT8 formats, which are suitable for high-speed, low-power inference on devices such as the NVIDIA Jetson Nano. Privacy was a key consideration. Instead of sending images to external servers, the system was designed with future support for Federated Learning, allowing devices to train locally and share only model updates. Although full implementation of Federated Learning was not part of this work, the design allows easy integration. Testing on a laptop with a webcam showed that the system runs smoothly on limited hardware while providing useful object detection. Future improvements could include running tests on real devices such as smart glasses or mobile boards, involving visually impaired users for feedback, expanding the dataset, and allowing each model to adapt to personal needs. Together, these steps can turn the system into a practical and trustworthy tool that gives visually impaired people more confidence and independence in their daily lives.