Uppsats

Birds, Bias, and Better AI: Improving Vision-Language Attention with Probabilistic Adapters : A post hoc probabilistic approach to attention guided learning from frozen vision language models

Kandidat-uppsats

Andersson, Svante, Flygar, Olle, Kemetli, Leo, Schmidt, Edvard

Uppsala universitet/Institutionen för informationsteknologi

Publicerad: 2025

Språk: Engelska

Nyckelord

klicka för att söka

⌕Artificial Intelligence ⌕Deep Learning ⌕vision–language models ⌕Robustness ⌕CLIP ⌕Aleatoric Uncertainty ⌕Epistemic Uncertainty ⌕Contrastive learning ⌕Grad-CAM ⌕Uncertainty Estimation ⌕Active Learning ⌕convolutional neural networks (CNNs)⌕multimodal learning ⌕ProbVLM ⌕BayesVLM ⌕probabilistic adapters ⌕Laplace approximation ⌕probabilistic embeddings ⌕heteroscedasticity ⌕generalized Gaussian distributions ⌕Kronecker-factored Hessian approximations ⌕ProbCosine ⌕Cosine similarity ⌕Attention mechanisms ⌕RISE ⌕GALS framework ⌕guided visual attention ⌕Waterbirds dataset ⌕spurious correlations ⌕fairness metrics ⌕worst-group accuracy ⌕balanced accuracy ⌕saliency maps ⌕Vision Transformers (ViT)⌕ResNet backbones ⌕frozen encoders ⌕cross-modal alignment ⌕intra-modal alignment ⌕probabilistic sampling ⌕transformer architectures ⌕supervised attention ⌕PyTorch ⌕Weights & Biases ⌕SLURM workload manager ⌕Alvis HPC cluster ⌕Apptainer containers ⌕image-text retrieval ⌕prompt-based evaluation ⌕Bayesian statistics ⌕maximum a posteriori estimation ⌕posterior covariance ⌕probabilistic cosine similarity scores ⌕uncertainty-aware attention ⌕interpretable AI ⌕dataset compositing ⌕training pipelines ⌕Heavy tailed distributions ⌕attention losses ⌕auxiliary loss functions ⌕fairness-critical machine learning tasks.

Sammanfattning

Vision-language models (VLMs) have demonstrated powerful capabilities across a wide range of tasks, but their deterministic nature limits their ability to express uncertainty, a key factor in robustness and fairness. This project explores the integration of two post-hoc probabilistic uncertainty estimation techniques, ProbVLM and BayesVLM, into the GALS (Guiding visual Attention with Language Specification) framework. ProbVLM leverages a probabilistic adapter trained on top of CLIP with a ResNet backbone, producing heteroscedastic embedding distributions and using Grad-CAM to generate attention maps. BayesVLM, on the other hand, applies a Laplace approximation to the projection layers of a ViT-based CLIP model, analytically propagating uncertainty to yield probabilistic cosine similarity scores, with RISE used for attention visualization. The experiments, conducted on the Waterbirds dataset under the GALS protocol, demonstrate that both probabilistic methods improve performance compared to the original GALS implementation. This improvement is evident in classification accuracy and in fairness-related metrics, indicating that uncertainty-aware attention mechanisms can better mitigate spurious correlations. These results highlight the promise of post-hoc probabilistic VLMs in enhancing both the interpretability and reliability of vision-language models in fairness-critical tasks.