Developing and Evaluating an Object Detection Application for Real-World Data

Sammanfattning

This thesis presents a pipeline for automated object identification and dot based annotation in User Generated Content (UGC) images. The need for this type of model is motivated by IKEA’s digital product ”Content Recommendations” which aims to enhance customer engagement through shoppable UGC images and addresses the challenge of detecting specific products in various, real-world visual contexts. The approach combines state of the art multimodal models: GroundingDINO for object localization based on textual prompts, CLIP for text-to-image classification, and EfficientSAM for instance segmentation. A custom dot placement algorithm, utilizing the segmentation masks from the previous step and then assigns coordinates (dot) to each detected object. The pipeline performs well across diverse and unstructured UGC images, successfully identifying and labeling both frequent and infrequent items. Due to inconsistent ground truth annotations and varying object prominence in UGC data, quantitative evaluation proved challenging. To address this, a proxy dataset containing manually verified dot annotations was used as ground truth. This enabled a more controlled comparison and supported assumptions about the pipeline’s expected performance on UGC images. Results indicate that multimodal AI methods are well suited for scalable, fine grained object detection and annotation in complex real-world visual data.