COMPARING WHEN DATA IS SCARCE: ZEROSHOT MULTIMODAL LLMs vs. CNN IN IMAGE CLASSIFICATION

Sammanfattning

Image classification is a fundamental machine learning task that has been heavily studied, and CNN architectures have long been considered the gold standard for image classification.However, the emergence of VLMs and multimodal LLMs provides a compelling alternative to CNNs: a fundamentally different paradigm pretrained on massive datasets of image--text pairs mapped to a shared semantic space.This gives VLMs strong zero-shot performance, meaning they can classify images using only a natural language prompt, with no labelled training data required.This thesis compares the two paradigms using multiple datasets demonstrating different levels of granularity: CIFAR-10, Stanford Cars, and StaffDetect, while simulating a data bottleneck for CNNs to determine whether VLMs truly provide a good alternative to CNNs in data-deprived image classification situations. Results show that zero-shot VLMs dominate on the general-purpose CIFAR-10 benchmark, with the best VLM outperforming the best CNN by approximately 21 percentage points, while fine-tuned CNNs hold an advantage on the fine-grained Stanford Cars task and the domain-specific StaffDetect task. Additionally, a precision-weighted ensemble of multiple generative VLMs is proposed and evaluated as a mechanism to improve performance over single models. The precision-weighted ensemble generally outperforms, with the degree of improvement dependent on constituent diversity of individual VLMs, with noticeable gains on CIFAR-10 of up to 4.54 percentage points.