High Frame Rate Video Synthesis using Event Cameras

Sammanfattning

High temporal resolution video is important in a variety of applications that involve objects in fast motion. These applications range from anything like the analysis of sports or animal behavior to the analysis of lightning and explosions. However, regular cameras are limited by frame rate and dynamic range due to bandwidth limitations, processing power, and data storage. Event cameras, on the other hand, address these limitations by providing microsecond temporal resolution at high dynamic range; however, event cameras are asynchronous and lack the texture rich appearance of standard images. What this study sets its focus on is event-based frame-interpolation. This study discusses the use of certain evaluation metrics for event-based frame interpolation and proposes two model architectures for the task, one relying on a continuous motion estimation through splines. The metrics, PSNR and SSIM, are used to evaluate the models on synthetic and real-world data as well as to investigate synthetic-to-real data transfer-learning. In addition, this thesis analyzes the effects of RGB-Event alignment errors from homography and model uncertainty based on the amount of movement. The findings suggest that synthetic-to-real data transfer learning is a good method for teaching a model the base-task of interpolation while alignment errors can make models unpredictable. Also reported is the relation between the amount of movement and the uncertainty of the models.