Comparative Analysis of Image Classification Models for Fashion Theme Identification : Evaluating YOLOv5 and CLIP Variants using Social Media Image Data

Sammanfattning

The human brain is excellent at recognizing objects and understanding higher-level descriptors of these objects almost instantaneously. Descriptors that might not be obvious at first, such as styles of people's clothing. For instance, a person can quickly identify if an outfit is boho or formal. However, computer vision models are mostly made to detect specific types of objects in images, not the overall style or theme. While these models can be accurate and fast in determining whether an item is a dress or a suit, they might struggle to tell if it is a boho dress or a formal dress. Areas where classifying themes in images is of interest are in social media and e-commerce. This thesis explores the application of Image Classification models to detect and classify themes in fashion images taken from the social media application Stylemine. The Image Classification models used in this project are You Only Look Once version 5 for classification (YOLOv–5cls), and three different variations of Contrastive Language-Image Pre-Training (CLIP). The CLIP variations are the regular CLIP model, FashionCLIP, made specifically for fashion items, and a version of FashionCLIP fine-tuned on a dataset given from the application Stylemine. YOLOv5-cls was chosen as this is a classic image classification model, whereas CLIP and its variants utilize Natural Language Processing (NLP) to gain semantic understanding from textual data. The images used for training YOLOv5-cls and FashionCLIP, and for evaluation of all models are all posts from Stylemine containing outfits with labels corresponding to their fashion themes. The models were compared using metrics such as accuracy, precision, recall, F1-score, and a subjective qualitative analysis. Results show that the difference in performance between YOLOv5-cls and CLIP was not substantial enough to say that CLIP outperforms YOLOv5-cls. The fine-tuned model performed worst in all metrics, likely due to the training dataset and procedure skewing its understanding of words. However, FashionCLIP outperformed the other models in most aspects, suggesting that FashionCLIP is the model that fits this problem the best.