Query-by-Example Audio Search using Acoustic Word Embeddings : Transforming wav2vec 2.0 Embeddings using Contrastive Learning

Sammanfattning

The daily creation and consumption of audio and video content has grown dramatically, making large-scale speech analysis essential for trendspotting and market analysis. While transcription-based models are common, they are often costly and prone to errors, especially in low-resource languages. Query-by-Example (QbE) audio search offers an intuitive way to retrieve spoken content by using audio queries instead of text. In this thesis, we investigate how contrastive embedding learning can improve clustering of acoustic word embeddings, to filter out speaker characteristics and background noise, and to enhance QbE search performance. Starting from pre-trained wav2vec 2.0 embeddings, we train a projection network using triplet loss to better align similar word instances while increasing separation between dissimilar words. We evaluate this approach in two phases: (1) a clustering evaluation, which measures the embedding space structure before and after transformation, and (2) a QbE-focused information retrieval evaluation to quantify improvements in word retrieval accuracy. Our experiments compare a Swedish pre-trained model named VoxRex to a multilingual model XLS-R and conclude that the Swedish model shows high potential for QbE audio search. In contrast, the XLS-R needs further work before being viable. This work contributes partly to going completely transcription-free in audio search, which is an ambitious but attainable goal that could save both time and computational costs for businesses in the automatic speech recognition field, and also to understanding how contrastive learning can be used to construct a phonetically structured embedding space.