Evaluation of Multimodal Fusion Strategies For Chest X-ray Pathology Classification

Sammanfattning

There is growing interest in combining medical imaging and clinical text to improve diagnostic support tools. However, there is limited evidence on which multimodal fusion strategy is most effective. This thesis presents a comparison of five fusion strategies—Late Fusion, Early Fusion, Hybrid Fusion, CLIP-style Joint Embedding, and Attention-Based Fusion—for multi-label chest X-ray disease classification on the MIMIC-CXR dataset, comprising 65,379 patients, 227,835 studies, and 14 CheXpert pathology labels. To ensure a fair comparison, all fusion models use the same image embeddings extracted from DenseNet-121 and the same text embeddings extracted from BioClinical- BERT, thereby isolating the effect of the fusion strategy from differences in feature extraction. Two complementary experimental settings are employed: a fixed patient-level split with five training seeds and a five-fold cross-validation framework. Model performance is evaluated using macro-average AUC, and statistical significance is assessed through paired t-tests. The results show that all five fusion strategies significantly outperform both image-only and text-only unimodal baselines. Among them, Hybrid Fusion achieves the best overall performance, with a macro AUC of 0.876, corresponding to an improvement of approximately five percentage points over the best unimodal baseline. In addition, all fusion models outperform parameter-matched unimodal baselines in the parameter-controlled comparison, indicating that the observed gains arise from multimodal integration rather than increased model capacity alone. Per-label analysis further reveals that different pathologies benefit from different fusion strategies. Overall, the findings demonstrate that multimodal fusion consistently improves chest X-ray disease classification, with Hybrid Fusion offering the most favourable balance between diagnostic accuracy and computational complexity.