Transformes for Time Series: Is the New State of the Art (SOA) Approaching?

### Overview In this presentation, Ezekiel Lanza, an AI open source evangelist at Intel, explores the adaptation of Transformer architectures to time-series forecasting based on his thesis research. While Transformers have demonstrated massive success in natural language processing (e.g., GPT models) and computer vision (e.g., Stable Diffusion), their effectiveness and status as state of the art in time series remain open questions. Lanza investigates whether Transformers outperform classical and recurrent baselines, detailing the architectural adjustments required, specific models like Informer and SpaceTimeFormer, and experimental results from a real-world microservices latency prediction task.

### Key Topics - **Foundations of Transformers**: The basic architecture introduced in 2017 ('Attention Is All You Need'), including self-attention, queries, keys, values, and multi-head attention mechanisms designed to capture both short- and long-term dependencies. - **Time Series vs. Natural Language**: Differences in sequential dependency. In natural language, exact token ordering can tolerate variance while preserving semantics, whereas time series demands strict temporal ordering and continuous numerical representations. - **Traditional and Deep Learning Baselines**: Classical statistical approaches such as ARIMA (limited to linear dependencies and heavy seasonal/trend analysis), standard feedforward neural networks, recurrent neural networks (RNNs, which suffer from vanishing gradients), and Long Short-Term Memory networks (LSTMs). - **Transformer Bottlenecks in Time Series**: The quadratic computational complexity ($O(N^2)$) of full self-attention across long sequences, requiring specialized architectural pruning and embedding modifications. - **Specialized Architectures**: Informer (introducing ProbSparse attention and timestamp embeddings such as hour, day, week, month, and holidays) and SpaceTimeFormer (introducing spatial-temporal attention across both variables/features and timestamps).

### Technical Details Lanza emphasizes two primary areas of modification when adapting Transformers for time series: input position encoding/embeddings and attention computation efficiency. - **Input Representation**: Unlike NLP word embeddings (such as Word2Vec), time series architectures must encode temporal context. Informer incorporates learnable global timestamp embeddings (weeks, months, holidays) combined with positional embeddings. SpaceTimeFormer represents inputs by flattening feature and temporal dimensions, allowing the attention mechanism to capture relationships across both time and individual features rather than relying on handcrafted feature graphs. - **Attention Optimization**: Full pairwise attention becomes computationally prohibitive over long sequence lengths. Informer implements ProbSparse self-attention, using likelihood-based metrics to prune redundant attention calculations and drastically lower compute overhead. - **Microservices Latency Use Case**: Lanza evaluated models in a real-world setting predicting front-end user latency from microservice telemetry. The dataset included latency metrics across microservices (such as P95, P98, P99 values), throughput metrics, and timestamps, resulting in approximately 250 to 2,000 feature dimensions across thousands of timestamps. Training on an initial 8,000 data points and evaluating on holdout sets (predicting 36 and 120 steps into the future), Informer achieved lower Mean Squared Error (MSE) than LSTMs, particularly over longer forecasting horizons. However, LSTMs exhibited significantly faster batch processing and training times.

### Takeaways - **Long-Horizon Advantage**: Transformers like Informer demonstrate superior MSE over traditional LSTMs mainly when predicting over longer future horizons, whereas the performance gap narrows considerably over short horizons. - **Computational Trade-Offs**: Despite optimizations like ProbSparse attention, Transformers incur substantially higher computational and time costs compared to lightweight baselines like LSTMs. - **No Universal Pretrained Model**: Unlike NLP, where general foundational models can be fine-tuned across diverse tasks, time-series data differs widely between domains, requiring models to be trained from scratch on specific use cases. - **Pragmatic Implementation**: Practitioners should evaluate trade-offs carefully, monitoring evaluation metrics beyond aggregate MSE (such as peak detection) and leveraging open-source frameworks like tsai to benchmark models before deploying computationally heavy architectures.