Brain-to-Language Model: Decoding Semantic Speech Representations from EEG via Cross-Modal Alignment

Sammanfattning

A key limitation of current hearing devices is the lack of user feedback to determine what speech content is actually being registered by the brain. To provide an objective measure of this cognitive processing for future brain-steered hearing devices, this study develops a brain-to-language model capable of decoding semantic-like information directly from non-invasive electroencephalography (EEG) signals. After modelling temporal response functions to confirm the correlation between Whisper embeddings and EEG signals, two distinct EEG encoders are trained via contrastive learning. Specifically, a dilated CNN architecture and a transformer model are utilised to map continuous neural data directly to the contextualised semantic embeddings extracted by Whisper. Since there is no observable ground truth for human semantic processing, this study is using Whisper’s artificial embeddings to represent them. Despite this limitation, results demonstrate that meaningful semantic patterns can be extracted from EEG recordings, with both models performing above chance. The transformer architecture achieved the best results by successfully mapping neural signals to the uncompressed semantic space, outperforming the dilated CNN which required dimensionality reduction.