Self-Supervised Learning of Multivariate Time Series Embedding via Diffusion Processes and using Imputation-Interpolation-Forecasting Masking : Representation Learning for Multivariate Time Series data using Diffusion Models

Sammanfattning

Multivariate Time Series (MTS) data, characterized by their sequential observations over time across multiple variables, are pivotal in diverse fields such as finance, healthcare, and environmental monitoring. Despite their widespread applicability, a significant challenge persists in effectively learning and leveraging embeddings from MTS data for various modeling tasks like imputation, forecasting, classification and anomaly detection. Traditional Self-Supervised Learning (SSL) approaches, including reconstructive, adversarial, contrastive, and predictive methods, for Time Series Representation Learning (TSRL) struggle with noise sensitivity and adequately capturing the intricate nuances of MTS data. This gap underscores a critical need for innovative methods that can robustly handle the complexity of MTS while providing versatile and informative representations. This thesis investigates the advanced generative capabilities of diffusion-based methods, which, until now, have primarily been applied to specific tasks such as imputation and forecasting, and explores their potential utility for generic TSRL. Our contribution, termed Time Series Diffusion Embedding (TSDE), marks a pioneering diffusion-based SSL approach to TSRL. TSDE segments Time Series (TS) data into observed and masked portions, employing an Imputation-Interpolation-Forecasting (IIF) mask. It utilizes dual-orthogonal Transformer encoders with a crossover mechanism to embed the observed data, and subsequently trains a reverse diffusion process that is conditioned on these embeddings to predict the noise added to the masked part. This method not only facilitates the self-supervised learning of embeddings but also ensures their applicability across various downstream tasks without the need for extensive labeled data. Extensive experimentation across tasks such as imputation, interpolation, forecasting, anomaly detection, classification, and clustering demonstrates TSDE’s superior performance relative to existing state-of-the-art methods, like CSDI model. An ablation study, embedding visualizations, and comparisons of inference speed further validate TSDE’s efficiency and effectiveness in learning robust representations of MTS data. The results of this thesis not only showcase a significant enhancement in performance, marked by an improvement up to 17.3% in RMSE over traditional methods, but also highlight the qualitative advantages of TSDE. This research marks a significant step forward in the processing and modeling of time series data, offering a flexible and competitive alternative to existing methodologies. By facilitating the learning of generic embeddings in a self-supervised fashion and leveraging Diffusion Models (DMs), TSDE opens new avenues for exploration and application in time series analysis across various domains.