Analyzing the Downstream Implications of Pretrained Image Representations in Visual Topic Models

Sammanfattning

The explosive growth of online content, particularly images, presents a significant challenge for analysis, as traditional text-based methods often fail to capture the full context and nuance of multimodal communication. This thesis addresses the necessity of effectively understanding vast image datasets by exploring the application of visual topic models. It investigates how different image representation techniques influence the quality and interpretability of automatically derived topics. The problem was tackled by developing a robust pipeline for visual topic modeling, employing three distinct image embedding methods: DINOv2 (purely visual), SigLIP2 (multimodal), and LLM-based embeddings (textual descriptions). A comprehensive evaluation was conducted, combining quantitative metrics for hyperparameter optimization with two novel human-centric crowd-sourced tasks designed to assess topic coherence and interpretability from both semantic and visual perspectives. Results indicate that the choice of image representation significantly impacts topic quality. DINOv2 consistently generated the most coherent and interpretable topics, demonstrating surprising strength in semantic understanding despite its purely visual training. SigLIP2 achieved a balanced performance, effectively capturing both visual and semantic relationships. Conversely, while LLM-based embeddings excelled semantically, their visual coherence was found to be lower. This work demonstrates the critical need for selecting appropriate image representation models based on the specific analytical goals. The findings provide valuable guidance for researchers and practitioners engaged in large-scale visual content analysis, enabling more accurate and insightful interpretations of complex online visual data.