Bimodal Inner Speech Decoding Using Generative Data Augmentation and Multimodal Fusion

Sammanfattning

Developing non-invasive Brain-Computer Interfaces (BCIs) for inner speech decoding remains fundamentally constrained by severe inter-subject neurophysiological variability and limited dataset availability. This thesis investigated whether multimodal neuroimaging integration, combining the high temporal resolution of Electroencephalography (EEG) with the high spatial resolution of functional Magnetic Resonance Imaging (fMRI), together with generative data augmentation, could improve cross-subject generalization. Domain-adversarial Conditional Variational Autoencoders (CVAEs) were employed to synthesize biologically plausible cognitive trials intended to reduce the inter-subject domain gap. Under a strict Leave-One-Subject-Out cross-validation framework, unimodal EEG and fMRI models were compared against multimodal fusion architectures incorporating both early feature fusion and latent cross-attention mechanisms. Results showed that label-conditioned augmentation failed to generalize during inference, causing unimodal models to collapse toward chance-level performance, while whole-brain multimodal fusion primarily introduced destructive interference rather than synergistic alignment. These findings suggest that overcoming inter-subject variability in inner speech BCIs may require moving away from label-conditioned generation and unconstrained whole-brain fusion strategies. Future approaches should instead prioritize unconditional biological denoising, targeted modeling of speech-related cortical regions, and self-supervised pretraining paradigms to improve robustness and clinical viability.