Real-time object tracking using spiking neural networks on neuromorphic hardware

Sammanfattning

This degree project has developed a set of algorithms and implemented them as a real-time system for tracking a moving object with the gaze of a robot head. As the algorithms are suitable for deployment on low power hardware and could be used in low latency settings, they enable future applications where a small mobile robot tracks fast-moving objects. The robot head is equipped with event cameras, a novel type of vision sensor with lower latency, higher dynamic range and that generates sparser data than conventional cameras. The system first computes the optical flow of the event camera input, where optical flow refers to the apparent motion in an image caused by relative motion between the observer and objects in the scene. This is achieved using a recently proposed spiking neural network (SNN) architecture. An SNN is a computational model inspired by biological neurons, where neurons communicate using ”spikes”. The project designed an SNN that maps the architecture to SpiNNaker, a hardware platform designed to facilitate efficient computation with SNNs, so called neuromorphic hardware. The optical flow is fed to a second SNN, developed from scratch and also running on SpiNNaker, that identifies the region of the image occupied by the moving object in a way that is robust to the robot head’s own movements. The key idea of this SNN is to determine where the local optical flow significantly deviates from the overall optical flow. Finally, the system controls the robot head’s actuators, adjusting its gaze to follow the tracked object. Compared to an earlier reference GPU implementation of the optical flow SNN, this project achieves a speedup of 8.4 to 80 times (depending on choice of GPU hardware), enabling real-time optical flow computation with comparable or fewer computing elements. This speedup is primarily due to the implementation leveraging the SpiNNaker platform’s highly flexible parallel execution capabilities, which allow the system to process synapses only when they transmit spikes. The full closed-loop system demonstrated stable behavior and some capability to track moving visual stimuli. Most importantly, this project highlights the potential efficiency of sparse computation using neuromorphic platforms.