Real-Time Multi-Camera Computer Vision for Proximity-Aware Soft-Stop and Emergency Braking in Autonomous Farm Vehicles.

Sammanfattning

Autonomous agricultural vehicles are increasingly being deployed for tasks such as tilling, planting, and harvesting as well as for continuous crop monitoring. One such platform is the Drever 120, developed by TraktorArvid AB. Currently, the vehicle's safety architecture relies on a front-mounted laser sensor for initial deceleration and pressure-sensitive bumpers to detect physical collisions. While this contact-based system ensures fundamental machine safety, triggering these physical sensors forces the tractor into a hard emergency stop that completely shuts down autonomous operations. Because this binary safety response requires a human operator to physically travel to the vehicle, inspect the hazard, and manually restart the system, it creates significant operational downtime during time-critical farming windows. To address this limitation, this thesis proposes and evaluates a real-time, multi-camera computer vision system capable of proximity-based safety logic. The proposed architecture replaces the binary response with a multi-stage protocol that initiates a controlled "soft stop" deceleration when hazards are detected within an outer warning zone, reserving hard emergency stops for immediate threats. Because collecting physical data of agricultural hazards is dangerous and time-consuming, a synthetic data generation pipeline was developed in NVIDIA Isaac Sim. Two perception models, RT-DETRv4 for object detection and PIDNet for semantic segmentation, were trained using different domain randomization strategies, including physically simulated adverse conditions like fog, dust, and rain. During operation, a ROS 2 sensor fusion pipeline projects 2D detections from four overlapping ZED X stereo cameras into a unified 3D vehicle coordinate frame, utilizing Kalman filtering to track hazards. The evaluation indicates that training models on physically simulated volumetric adverse conditions significantly improves spatial generalization and robustness compared to standard baseline training or 2D image augmentation. RT-DETRv4 outperformed PID-Net in both detection accuracy and inference speed, showing suitability for real-time edge deployment at approximately 17.8 FPS. In closed-loop safety simulations, the system successfully converted 67.0\% of hard emergency stops into soft decelerations, with the existing contact-based safety system remaining as a fallback layer for the remaining scenarios. However, evaluation on a physically collected real-life dataset revealed a considerable performance drop across all models, suggesting that the sim-to-real domain gap remains a significant challenge requiring further improvements in simulation realism before physical deployment. Despite this, the findings indicate that a multi-stage, vision-based proximity system can successfully mitigate false positives and enhance the operational availability of autonomous farm vehicles.