Directional Visual Perception for Object Sonification in Indoor Environments : A tool for the blind

Sammanfattning

It is a well known fact that physical activity and sports provide significant health benefits. While many different assistive tools have been developed for visually impaired individuals throughout the years, they still often lack the assistive technology to participate in sports independently. In this thesis, the gap is addressed through the development of a prototypical platform that utilizes computer vision and machine learning to create directional perception for the sonification of objects in real-time. The system, built up of a Raspberry Pi 5 and an AI Camera, employs the pinhole camera model to translate 2D pixel coordinates of an object into a 3D target vector consisting of an azimuth, elevation and distance. These vectors are then rendered into spatial audio with Head-Related Impulse Responses (HRIR), and a sound is played inside a pair of headphones that corresponds to the direction of said object, along with a frequency attenuation dependent on the distance to it. A comparative analysis was conducted between two object detection algorithms in this context: MobileNet SSD V2 320x320 and Color Segmentation (ColSeg). The results demonstrated that MobileNet SSD maintained exceptional hardware stability, averaging a 20% CPU load with zero dropped targets over a one-hour stress test. This was possible due to the AI Cameras internal IMX500 sensor handling the image processing. However, MobileNet's object detection provided inconsistent bounding-box dimensions around human targets under movement, which led to unreliable distance estimates. Conversely, while ColSeg provided stable distance metrics for rigid objects like balloons, its processing demands over a one-hour stress test caused the Raspberry Pi to reach its 85°C thermal throttling limit. This resulted in a cascading hardware failure: HRIR calculation latency spiked over 30 ms, causing buffer backlogs, target drops, and severe audio stuttering. The study concludes that while the developed platform establishes a robust foundation for a non-invasive directional perception tool, relying on 2D bounding boxes for depth estimation is methodologically insufficient for dynamic sports environments. To ensure system survivability and user safety in future iterations of the system, the inclusion of hardware depth sensors, such as LiDAR or stereo cameras, and the use of pre-computed HRIR filters are recommended.