Bridging the gap between Language and Radio Frequency Signals : Exploring what is needed to create a Multimodal Large Language Model for radio frequency signals to language, and how a CLIP model can be used for zero-shot modulation classification

Sammanfattning

The development of Multimodal Large Language Models (MLLMs) has significantly enhanced the ability to process and integrate multiple forms of data in the era of artificial intelligence. This thesis explores the application of MLLMs to bridge the gap between radio frequency (RF) signals and language. Specifically, it investigates the use of a Contrastive Language-Image Pre-training (CLIP) model for zero-shot modulation classification. The study involves adapting a CLIP model to use a convolutional neural network (CNN) as the signal encoder, paired with a language model, to classify RF signal modulations, here 22 different modulations are used like M-QAM. Experiments were conducted using simulated RF signal data to evaluate the performance of this adapted CLIP model compared to a traditional CNN classifier. The results demonstrate that the CLIP model, with appropriate settings, can achieve comparable accuracy to the CNN on known modulations and shows promise in zero-shot classification of unseen modulations. The findings emphasize the crucial need for data that pairs RF signals with natural language text to fully leverage the capabilities of advanced language models. Integrating advanced language models with signal processing can enhance the flexibility and adaptability of RF signal classification systems, paving the way for future research and development in this interdisciplinary field.