Eye in the Sky - Vision System for Automatic Detection and Tracking of Flying Objects

Sammanfattning

The widespread deployment of Unmanned Aerial Vehicles (UAVs) in both civilian airspace and military operations has made onboard perception an increasinglycritical capability. Detecting other airborne objects from a moving platform is particularly challenging: targets are often small, the camera itself is in motion, and IRimagery introduces additional complexity due to scarce training data. This thesispresents a modular system designed to handle these constraints in real-time, usingboth RGB and IR cameras mounted on a UAV.Object detection is handled by RF-DETR, a transformer-based detector suitablefor a real-time application. For tracking, we extend ByteTrack with optical-flowbased ego-motion compensation, which proves important when the camera platformitself is moving. Classification relies on a CLIP-based model that we adapt to IRthrough cross-modal knowledge distillation combined with Low-Rank Adaptation(LoRA) fine-tuning, allowing us to transfer visual representations from RGB to IRwithout requiring pretraining on IR imagery.Experiments on simulated UAV imagery show that the RGB detector achieves anF1 score of 0.924 and the IR detector 0.824, despite the latter having access to substantially less training data. The classifier adaptation has a large effect on dronerecall, which increases from 0.252 to 0.654 in RGB mode and from 0.049 to 0.617 inIR mode. Tracking performance, measured with the HOTA metric, reaches around20 in RGB mode and 14 in IR mode, with most errors caused by long occlusionsand very distant targets.The complete pipeline runs at approximately 21 fps on a standard laptop GPU, suggesting that it is capable of real-world deployment. The initial results are promisingand indicate that the proposed system can handle several real-world constraints.However, two key limitations remain: performance drops on far-range targets, andall evaluation is performed in simulation with pseudo-IR imagery, leaving a gap toreal-world deployment.