Prompt-Guided Semantic Verification with Vision Language Model : A Post-Detection Framework for UAV-Based Vehicle Detection in Adverse Weather Conditions

Sammanfattning

Accurate vehicle detection from aerial imagery is critical for many modern applications, yet conventional object detectors often struggle under adverse conditions such as snow, low light, and occlusion. This thesis presents a hybrid detection–verification system designed to enhance robustness in such environments. The proposed system integrates a custom YOLO-based detector (YOLOv11s) with the Vision Language Model Gemma-3, which serves as a post-detection verifier to filter false positives through prompt-guided semantic analysis. Systematic prompt engineering was applied to guide Gemma-3 in interpreting visual content alongside contextual language cues, enabling it to assess the validity of detections. Multiple prompting strategies were systematically evaluated, including zero-shot prompting with manually authored and meta-generated prompts, as well as few-shot, region-based, chain-of-thought, and cascade prompting techniques. Among these, zero-shot prompting with meta-prompts and cascade prompting achieved the highest verification performance on the Nordic Vehicle Dataset (NVD), demonstrating superior precision in filtering false positives from YOLOv11s detections. Experimental results show that Vision Language Models, such as Gemma-3, demonstrate effective semantic verification under challenging Nordic weather conditions, significantly reducing false positives in UAV-based vehicle detection. The results show that verification performance strongly depends on prompt design, with well-scaffolded prompts that support reasoning across visibility, terrain, and object plausibility. Overall, they provide a reliable post-detection layer, particularly valuable in scenarios where precision must be prioritized.