Learn AI Create the Future — with Pranay
Main site

Time Series and Forecasting

LSTMs for forecasting

An LSTM reads a sliding window of recent values and predicts the next one, which works well on long series and loses to far simpler methods on short ones.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. The shape of the job
  4. Two things you must do first
  5. When it is worth the trouble
  6. What is honestly hard here
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

You cut the series into overlapping windows of recent days. The network is trained to guess the day that came right after each window.

The analogy you have already lived

Watch a batsman facing a bowler he has never seen before.

By the sixth or seventh ball, he has a feel for it. Slightly slow, drifts in, one quick one every over. He is not remembering every delivery of his career. He carries a rolling impression of what has been happening lately, and he updates it with each ball.

That rolling impression is what a network with memory does. It reads the recent stretch, keeps a summary of it, and predicts what comes next.

An LSTM is one design for that kind of memory, and the LSTM lesson explains how the memory itself works. This lesson is about the part specific to forecasting: how you feed it, and whether it is worth using at all.

The shape of the job

The trick is turning a long series into thousands of small practice questions.

   the series:   ... 31  28  30  52  49  29  34  36  32 ...

   window 1:  [31 28 30 52 49]  ->  answer 29
   window 2:  [28 30 52 49 29]  ->  answer 34
   window 3:  [30 52 49 29 34]  ->  answer 36
   window 4:  [52 49 29 34 36]  ->  answer 32

Each window slides forward by one. A series of nine hundred days gives you close to nine hundred practice questions, from one series.

The network sees a window, guesses, is told the true answer, and adjusts. That is the same loop as any other model, described in how neural networks learn.

Two things you must do first

Scale the numbers. Neural networks behave badly with inputs in the thousands. Convert your values to a small range around zero first, then convert the predictions back at the end.

Compute the scaling from training data only. Work out the average and spread using the early part of the series. Apply those same two numbers to the later part. Using the whole series would let the average of your test period sneak into training. That is the leakage problem from feature engineering for time series, wearing yet another hat.

When it is worth the trouble

Be honest with yourself before reaching for this.

An LSTM is a reasonable choice when:

  • You have years of frequent data, not two years of monthly numbers.
  • You have many related series and can train one model across all of them.
  • The pattern is genuinely complicated and shifts over time.

It is the wrong choice when:

  • You have under a few hundred observations. There is nothing to learn from.
  • A seasonal naive baseline is already close to the noise level in your data.
  • You need to explain the forecast to someone who will question it.

The developer section runs a real comparison. The LSTM does win. A plain linear model on the exact same windows comes within a hair of it. It takes a fraction of the time.

That result is not a curiosity. Serious research has repeatedly found simple models matching or beating complicated ones on standard forecasting benchmarks.

What is honestly hard here

Deep learning has a reputation on time series that is bigger than its record.

The world forecasting competition in 2018 was won by a method that combined a neural network with classical exponential smoothing. Not a pure network. The classical part carried the trend and the seasonality; the network learned what was left over.

There is also a practical cost nobody mentions in tutorials. An LSTM needs scaling, window sizing, layer sizing, a learning rate, an epoch count and an early-stopping rule. Each is a chance to go wrong. A seasonal naive baseline has none of them and takes one line.

Always build the baseline first. If the network cannot beat it, you have learned something valuable and cheap.

Remember this

  • Cut the series into sliding windows; each window's next value is the answer.
  • Scale using training data only, then convert predictions back.
  • Build the simple baseline first, and be willing to ship the baseline.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch numpy pandas scikit-learn

The CPU build of PyTorch is enough. The script below trains in about five seconds on an ordinary laptop with no GPU. If you install the CPU-only wheel from the PyTorch site the download is a few hundred megabytes rather than several gigabytes.

A complete forecaster, with its own baselines

Everything is here: windowing, training-only scaling, an LSTM, a linear model on identical inputs, and a naive baseline.

lstm_forecast.py
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from sklearn.linear_model import Ridge

torch.manual_seed(0)
rng = np.random.default_rng(9)

n, look, cut = 900, 28, 750
days = pd.date_range("2023-01-01", periods=n, freq="D")
y = (1000 + np.where(days.dayofweek >= 5, 250, 0)
     + 150 * np.sin(2 * np.pi * days.dayofyear.to_numpy() / 365.25)
     + rng.normal(0, 40, n))

mu, sd = y[:cut].mean(), y[:cut].std()          # scale using TRAINING data only
z = (y - mu) / sd

X = np.stack([z[i:i + look] for i in range(n - look)])   # each row is 28 consecutive days
T = z[look:]                                             # the day right after each window
Xtr, Ttr = X[:cut - look], T[:cut - look]
Xte, Tte = X[cut - look:], T[cut - look:]

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(input_size=1, hidden_size=32, batch_first=True)
        self.head = nn.Linear(32, 1)
    def forward(self, x):
        out, _ = self.lstm(x.unsqueeze(-1))      # add the "one number per step" channel
        return self.head(out[:, -1]).squeeze(-1) # use only the final step's memory

net = Net()
opt = torch.optim.Adam(net.parameters(), lr=0.01)
lossf = nn.MSELoss()
xt = torch.tensor(Xtr, dtype=torch.float32)
tt = torch.tensor(Ttr, dtype=torch.float32)

for epoch in range(1, 201):
    opt.zero_grad()
    loss = lossf(net(xt), tt)
    loss.backward()
    opt.step()
    if epoch % 50 == 0:
        print(f"epoch {epoch:3d}  training loss {loss.item():.4f}")

with torch.no_grad():
    lstm_pred = net(torch.tensor(Xte, dtype=torch.float32)).numpy() * sd + mu
actual = Tte * sd + mu
ridge_pred = Ridge(alpha=1.0).fit(Xtr, Ttr).predict(Xte) * sd + mu

print("\nLSTM, 32 hidden units    MAE:", round(float(np.abs(actual - lstm_pred).mean()), 1))
print("ridge on the same 28 lags MAE:", round(float(np.abs(actual - ridge_pred).mean()), 1))
print("seasonal naive            MAE:", round(float(np.abs(y[cut:] - y[cut - 7:-7]).mean()), 1))
print("noise floor, roughly      MAE:", round(0.8 * 40, 1))
Output
epoch  50  training loss 0.4655
epoch 100  training loss 0.1007
epoch 150  training loss 0.0712
epoch 200  training loss 0.0648

LSTM, 32 hidden units    MAE: 31.3
ridge on the same 28 lags MAE: 33.0
seasonal naive            MAE: 41.3
noise floor, roughly      MAE: 32.0

These exact numbers depend on your PyTorch version. torch.manual_seed(0) fixes the initialisation and the optimiser path on one machine, but floating-point kernels differ between versions and platforms. Expect the same story with slightly different digits.

Reading the result honestly

The LSTM reached 31.3, which is at the noise floor. The series has noise with a standard deviation of 40, so roughly 32 is the best any model could do. The network extracted essentially all the available signal.

It beat the seasonal naive by a wide margin, 31.3 against 41.3. Worth having.

Ridge regression on the identical windows scored 33.0. A linear model with 28 coefficients, fitted in milliseconds, landed within five percent of the network. It needed no epochs, no learning rate, no seed and no framework.

That is the whole argument in four numbers. On a series like this — one series, moderate length, a fixed weekly and yearly pattern — a linear model on lags is a formidable competitor. Run it every time, and only accept the network when it wins by an amount worth the operational cost.

Line by line, the parts that trip people up

x.unsqueeze(-1) turns shape (batch, 28) into (batch, 28, 1). PyTorch's LSTM expects three dimensions: how many examples, how many time steps, how many numbers per step. A univariate series has one number per step, and that final 1 is not optional. Forgetting it produces a shape error that reads as nonsense the first time.

out[:, -1] takes the output at the last time step only. out holds one output per step; you want the memory after the whole window has been read. Taking out[:, 0] by mistake trains the model on almost nothing.

mu and sd come from y[:cut]. Computing them from all of y would push information about the test period into every training input. The score would improve and the improvement would be fake.

hidden_size=32 with look=28 and 722 training windows. The model has roughly 4,500 parameters against 722 examples. It is over-parameterised, which is normal for neural networks, and the reason the training loss keeps falling. Watch a validation split rather than the training loss.

No mini-batching. With 722 examples the whole set fits in memory and full-batch gradient descent is fine. For a million windows you would use a DataLoader.

Going further, with a warning attached

python
nn.LSTM(input_size=1, hidden_size=64, num_layers=2, dropout=0.2, batch_first=True)

More capacity helps only when you have more data. On 722 windows, a two-layer 64-unit LSTM will fit the training set harder and generalise no better.

For multi-step forecasting, change the head to nn.Linear(32, horizon) and make the target a vector of the next horizon days. That is a direct multi-step model, and it avoids the compounding error you get from feeding predictions back in.

For several related series, add features per step so input_size is greater than one, and train one model over all of them. That pooling is where deep learning earns its keep — it is the setting where a single model sees millions of windows instead of hundreds.

Common mistakes

Scaling before splitting. StandardScaler().fit(y) on the whole series. This is the most common bug in every LSTM forecasting tutorial on the internet. Fit on the training slice.

Shuffling the windows and calling it a random split. Windows overlap. Window 100 and window 101 share 27 of their 28 values. Shuffling puts near-duplicates on both sides of the split, and your validation score becomes meaningless. Split by time, always.

Reporting the training loss as the result. The loss printed above is training loss on scaled data. It is not comparable to anything and it always goes down. The MAE on held-out days in original units is the result.

Skipping the baseline. If the network scored 41.5 here, it would have lost to a one-line seasonal naive. You would not know that without computing it.

Using an LSTM on 60 monthly observations. There is not enough data. Use exponential smoothing or ARIMA.

Forgetting to convert predictions back. Predictions come out on the scaled axis. Multiplying by sd and adding mu is required before any error metric means anything in real units.

Try it yourself

Change look from 28 to 7 and rerun, then to 90.

Predict which one wins before you run it. A window of 7 can see the weekly pattern but not the yearly one. A window of 90 sees more but gives the model three times as many inputs from the same 722 examples. Then run the same three window sizes through the ridge model and see whether the two models prefer the same one.

What to learn next

Researcher — Mathematics and papers.

The recurrence

For an input sequence $\mathbf{x}_{1:L}$, the LSTM cell of Hochreiter and Schmidhuber (1997), with the forget gate of Gers, Schmidhuber and Cummins (2000), is

$$ \begin{aligned} \mathbf{f}_t &= \sigma(W_f[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f) &\quad& \text{forget gate}\ \mathbf{i}_t &= \sigma(W_i[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i) &\quad& \text{input gate}\ \tilde{\mathbf{c}}_t &= \tanh(W_c[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c) &\quad& \text{candidate}\ \mathbf{c}_t &= \mathbf{f}t \odot \mathbf{c}{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t &\quad& \text{cell state}\ \mathbf{o}_t &= \sigma(W_o[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o) &\quad& \text{output gate}\ \mathbf{h}_t &= \mathbf{o}_t \odot \tanh(\mathbf{c}_t) \end{aligned} $$

  • $\mathbf{c}_t \in \mathbb{R}^d$ — the cell state, the additive memory path.
  • $\mathbf{h}_t \in \mathbb{R}^d$ — the hidden state, also the output at step $t$.
  • $\odot$ — elementwise product.
  • $d$ — hidden_size.

Parameter count is $4d(d + p + 1)$ for input dimension $p$. The additive update of $\mathbf{c}_t$ gives a gradient path whose Jacobian is $\mathrm{diag}(\mathbf{f}_t)$ rather than a repeated weight matrix, which is the mechanism that mitigates the vanishing gradient identified by Bengio, Simard and Frasconi (1994). It mitigates rather than eliminates: with $f_t < 1$ persistently, gradients still decay geometrically.

What the architecture assumes, and what it does not model

Three consequences follow directly from the formulation, and each is a real limitation in forecasting:

No explicit trend term. A network with bounded activations cannot extrapolate a level beyond its training range. Training on values scaled to $[-2, 3]$ and asking for a prediction at $z = 5$ gives a saturated output. This is the same defect as tree models, arrived at differently. The standard remedies are per-window normalisation (subtract the last value of each window, forecast the deviation) or differencing before windowing.

Sequential computation. The recurrence has an $O(L)$ serial dependency that no amount of hardware removes, which is why temporal convolutions and attention displaced LSTMs where $L$ is large.

Single-scale memory. Capturing both a weekly and an annual cycle from raw lags requires $L \ge 365$, and the effective memory of an LSTM over hundreds of steps is empirically much shorter. Explicit seasonal features are cheaper and more reliable than asking the recurrence to discover them.

The empirical record

M4 (2018). Smyl's winning entry was a hybrid: a Holt-Winters-style multiplicative level and seasonality with learnable smoothing coefficients per series, combined with a dilated LSTM stack shared across series, trained with a pinball loss and ensembled. The decomposition was classical; the network learned the residual dynamics. Pure neural entries placed below statistical baselines. Smyl (2020), A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting, IJF 36(1).

M5 (2020). Gradient-boosted trees over engineered features dominated. Neural entries were present but not at the top.

Zeng, Chen, Zhang and Xu (2023), Are Transformers Effective for Time Series Forecasting?, AAAI, showed that DLinear — a single linear layer over a decomposed input — matched or beat Informer, Autoformer and FEDformer across the standard long-horizon benchmarks. The paper's contribution is partly methodological: it argued that the benchmark protocols in several transformer papers were flawed. The result reproduced widely, and the ridge comparison in the developer block is a miniature of it.

Where neural methods do win is the global, many-series regime with covariates. DeepAR (Salinas et al., 2020) trains one autoregressive RNN across thousands of series, outputs the parameters of a chosen likelihood (negative binomial for counts, Student-$t$ for heavy tails) and samples forecast paths, giving calibrated probabilistic forecasts. N-BEATS (Oreshkin et al., 2020) uses a deep stack of fully connected blocks with backcast-forecast residual connections and beat the M4 winner on M4 data. Temporal Fusion Transformers (Lim et al., 2021) add static covariates, known-future inputs and variable selection with interpretable attention.

The pattern across all of them: neural architectures win when they can pool across many series and consume covariates, and lose when applied to one short univariate series.

Evaluation protocol requirements

Overlapping windows make several standard practices invalid.

  • Never shuffle windows across the split boundary. Windows $t$ and $t+1$ share $L-1$ values.
  • Purge a gap of at least $L$ between train and validation, or the last training windows overlap the first validation targets.
  • Report multiple seeds. Single-seed neural results on small data have a standard deviation across seeds that frequently exceeds the gap being claimed between methods.
  • Report a naive and a linear baseline on identical windows, or the comparison is uninterpretable.

Hewamalage, Bergmeir and Bandara (2021), Recurrent neural networks for time series forecasting: Current status and future directions, IJF 37(1), gives a systematic study of these protocol issues and of when RNNs are and are not competitive.

Reading

  • Hochreiter and Schmidhuber (1997), Long Short-Term Memory, Neural Computation 9(8).
  • Smyl (2020), A hybrid method of exponential smoothing and recurrent neural networks, IJF 36(1).
  • Salinas, Flunkert, Gasthaus and Januschowski (2020), DeepAR, IJF 36(3) — arxiv.org/abs/1704.04110.
  • Oreshkin et al. (2020), N-BEATS, ICLR — arxiv.org/abs/1905.10437.
  • Zeng et al. (2023), Are Transformers Effective for Time Series Forecasting? — arxiv.org/abs/2205.13504.
  • Hewamalage, Bergmeir and Bandara (2021), Recurrent neural networks for time series forecasting, IJF 37(1).

What to learn next