Timbre Transfer in the Latent Space of Descript Audio Codec : Transforming Beatbox Performances Into Acoustic Drum Kit Recordings

Sammanfattning

This thesis explores timbre transfer, which involves changing the sonic character of a recorded instrument performance while preserving the rest of its musical content. We present timbre transfer models which use latent space representations of paired instrument recording data. Existing approaches to timbre transfer often use output instrument-specific analysis–synthesis models such as variational autoencoders and concatenative synthesizers. However, we demonstrate that these architectures can experience difficulties generating rhythmically consistent recordings of non-pitched percussion instruments. We propose three models for timbre transfer within the pre-trained latent space of Descript Audio Codec, based on affine transformation, match search, and neural transformation. Additionally, we present an alternative approach to variational autoencoder-based timbre transfer using paired training data and Gaussian mixture model prior distributions. We compare our latent space models to standard and modified implementations of RAVE, an existing variational autoencoder, using paired beatbox and acoustic drum kit recordings. Results show that RAVE models trained using our proposed paired data and prior distribution frameworks generate more accurate recordings than the standard model. Our neural transformation models perform comparably to standard RAVE with significantly fewer parameters, while our affine and match search models are much faster to train.