Seeds and reproducible runs

Setup#

pip install torch

Verified with torch 2.5.1, CPU. The specific digits below depend on the torch version — a different version may draw a different (but equally repeatable) sequence.

Seeds in action#

import torch

torch.manual_seed(42)
print(torch.rand(3))

torch.manual_seed(42)         # reset the sequence to the start
print(torch.rand(3))          # the same three numbers again

print(torch.rand(3))          # the sequence has moved on

g = torch.Generator().manual_seed(7)   # a private stream, untouched by others
print(torch.rand(3, generator=g))

tensor([0.8823, 0.9150, 0.3829])
tensor([0.8823, 0.9150, 0.3829])
tensor([0.9593, 0.3904, 0.6009])
tensor([0.5349, 0.1988, 0.6592])

The walkthrough#

Lines 1 and 2 match; line 3 does not. Every random draw advances the generator. Re-seeding rewinds it to the start. This also means the order of random calls is part of your experiment — insert one extra torch.rand(1) anywhere and everything downstream shifts.

The private Generator is underused. Give your data shuffling its own generator, and your model initialisation another. Then adding a dropout layer (which consumes global randomness) no longer silently changes your data order. Isolation of streams is what makes ablations clean.

Seeding for a full training script takes more than one call, because randomness enters through several doors:

import random
import numpy as np
import torch

def seed_everything(seed: int):
    random.seed(seed)               # Python's own generator
    np.random.seed(seed)            # NumPy's generator
    torch.manual_seed(seed)         # PyTorch CPU, and all GPUs too

torch.manual_seed covers CUDA devices as well; separate cuda.manual_seed calls are unnecessary in torch 2.x. What it does not cover: Python's random, NumPy, and worker processes in a DataLoader (which need a worker_init_fn or generator argument — see PyTorch basics for the loader itself).

Seeds are not full determinism — the honest part#

This confuses almost everyone at first, and it is worth reading twice.

With a seed, both runs draw the same random numbers. But some GPU operations execute additions in a timing-dependent order, and float addition cares about order. Result: same inputs, same seed, answers differing in the seventh decimal — which after thousands of steps can become visibly different curves.

If you need bit-for-bit repeatability:

torch.use_deterministic_algorithms(True)

This switches every operation to a deterministic implementation — or raises an error naming the operation that has none, so you can avoid it. Determinism costs speed; measure before adopting it permanently. cuDNN's convolution auto-tuner (torch.backends.cudnn.benchmark) also picks algorithms by timing races, so leave it off for reproducible work.

Common mistakes#

Seeding once, comparing across code changes that alter call order. Same seed, but your new augmentation consumes two draws where the old one consumed one — everything after diverges. Compare seeds and be aware the stream is positional. Private generators per component reduce this coupling.

Reporting one seeded run as "the" result. A seed makes luck repeatable; it does not remove luck. Score three to five seeds and report the range — a habit that saves public embarrassment when someone else's seed lands differently.

Expecting identical numbers across machines or versions. CPU versus GPU, different GPU models, different torch versions — each may legitimately draw or accumulate differently. Reproducibility claims should name the environment: "torch 2.5.1, CPU" is a claim; "always" is not.

Forgetting the loader workers. Multi-process data loading has per-worker random state. Without seeding it, augmentations differ run to run even though everything else is pinned.

Try it yourself#

Write a ten-line script: seed, build torch.nn.Linear(4, 2), print its weights. Run it twice and confirm identical weights. Then move the seed line below the layer creation and run twice more — watch repeatability vanish, and explain to yourself exactly why.

What to learn next#

• How the autograd graph is built and freed — the next section: what PyTorch records while you compute.
• Creating tensors and choosing a dtype — where the random creation functions live.
• PyTorch basics — the training loop all this reproducibility serves.