Beyond Addition: Enhancing Time Series Transformers with Hyperdimensional Binding

Sammanfattning

Traditional Transformers integrate positional information via additive Positional Encodings (PEs) [25], a simple superposition method with known limitations such as embedding space anisotropy [15], which can decrease performance on sequential data like time series. This thesis investigates alternative approaches to enhance Transformer efficacy for Time Series Classification (TSC) by leveraging Hyperdimensional Computing (HDC) binding operations, specifically component-wise multiplication and circular convolution, and by exploring PEs with explicitly designed similarity shapes. This research is further motivated by recent advancements in Large Language Models (LLMs), such as the development of novel PEs like Rotary Position Embeddings (RoPE) [23]; time series specific architectures like ConvTran, which uses encodings like Time Absolute Position Encoding (tAPE) [5]; adaptive encodings that learn optimal frequencies [24], and attention mechanisms like Multi-head Latent Attention (MLA) [2], which also move beyond simple additive integration to embed positional information more structurally. Using an encoder-only Transformer architecture, a systematic evaluation was conducted on a diverse suite of UCR/UEA TSC datasets [1]. The experiments first compared HDC binding methods against the standard additive approach, then assessed scalability with model depth and embedding dimension. Subsequent experiments investigate the impact of diverse PE schemes and benchmark the best-performing configuration against modern techniques, including RoPE [23] and the ConvTran architecture [5], to contextualize the findings. Key findings demonstrate that HDC binding methods significantly outperform the conventional additive approach. The top-performing HDC-based models achieved a mean accuracy of over 64%, substantially higher than the 54-58% range of the additive baselines. This superiority was maintained across various embedding dimensions, while the additive approach suffered a notable performance collapse at higher dimensions, where its accuracy dropped to near-random chance on several datasets. Contrary to the initial hypothesis, the performance advantage of HDC binding did not increase with model depth; instead, all configurations generally performed best with shallow architectures, highlighting a critical divergence from trends seen in other domains such as Natural Language Processing (NLP). These results strongly suggest that the algebraic operation used to integrate positional information is a critical and overlooked design choice. Moving beyond simple addition and toward more expressive HDC binding operations offers a robust and effective strategy for improving Transformer performance on TSC tasks.Code and experiment results are open-sourced at: https://github.com/JoseJuan98/transformer-attention-with-hdc-binding.