Forecasting Using Deep State-Space Models

Sammanfattning

State-Space Models (SSMs) have shown promising results in time-series modeling, which often possess complex dependencies over long sequences. Deep SSMs offer an alternative to models such as Recurrent Neural Networks, Convolutional Neural Networks, or Transformers. However, SSMs have limitations in forecasting, i.e., making future predictions of a sequence given its history. This thesis investigates some of these limitations under three different forecasting approaches using Deep SSMs: zero-masking, closed-loop forecasting, and an encoder–decoder setup inspired by a recently proposed SpaceTime model. These approaches were evaluated using both real- and complex-valued state transition matrices that control the evolution of the hidden state. These approaches are evaluated on two synthetic datasets and on the small Electricity Transformer Temperature (ETT) datasets. The results show that complex-valued state transition matrices can be beneficial for forecasting oscillatory signals, substantially outperforming real-valued matrices on synthetic data. Zero-masking is effective on low-complexity data, but becomes less robust than closed-loop and the SpaceTime inspired model when the signals become more complex. On the univariate ETT-small datasets, the models are competitive and they all outperform the popular and widely used S4 model for the forecast horizon studied, but do not consistently outperform SpaceTime. These findings clarify the trade-off between expressivity and training duration for Deep SSMs, and show that a complex diagonal parameterization for state transition matrices of Deep SSMs provides a good framework for forecasting.