Reproducibility in machine learning
Hook
You train a model on Friday and it scores 0.912. On Monday a teammate pulls your code, retrains, and gets 0.887. Nobody edited the model. The data looks the same. Yet the numbers disagree, and now you cannot tell whether a later "improvement" is a real gain or just the same noise landing differently. "Works on my machine" has quietly become "works on my machine, on Friday, if I do not restart the kernel."
Concept
Reproducibility is the guarantee that the same code, run on the same data with the same configuration, produces the same model. That is the whole definition, and every word in it is load-bearing: change any of the three inputs — code, data, or config — and you are allowed a different result; hold all three fixed and a different result is a bug.
The reason ML breaks this guarantee more often than ordinary software is that a training run has hidden inputs you did not think you were choosing. The biggest is randomness. Weight initialization, data shuffling, dropout, and train/test splits all draw from a random number generator. If you never fix that generator's starting point, every run draws a different sequence, so every run trains a slightly different model. Fixing the starting point is called setting a seed: a seed is the number that makes a pseudo-random sequence repeat exactly.
The second hidden input is the environment. scikit-learn 1.3 and 1.4 can
implement the same estimator differently; a NumPy upgrade can change a default;
even the number of CPU threads can reorder a floating-point sum. "It ran last
month" is not reproducibility if you cannot reconstruct the versions it ran with.
The fix is pinning: a lockfile records the exact version of every dependency,
and a container freezes the whole environment — interpreter, libraries, and
system packages — into one immutable image. The container is the unit of
environment reproducibility because it removes "which version did you have
installed?" from the conversation entirely.
A useful distinction sits underneath all of this. Determinism means a single computation always returns the same output for the same input — a seeded shuffle is deterministic. Reproducibility is the broader promise that a whole run repeats, which needs determinism plus a pinned environment plus the same data. You can have a perfectly deterministic function and still fail to reproduce a result because a library version drifted underneath it.
| Hidden input | Symptom when unpinned | Fix |
|---|---|---|
| Random generator | Metrics wobble run to run with no code change | Set an explicit seed |
| Library versions | Same code, different result after pip install | Lockfile with pinned versions |
| Whole environment | "Works on my machine" across teammates | Container image |
| Data | Yesterday's file was overwritten in place | Immutable, versioned data |
Setting a seed makes a run repeatable, not correct. A seeded run reproduces the same result every time — including reproducing the same overfit model if your setup is flawed. Reproducibility is a prerequisite for trusting a comparison, not evidence that the model is good.
Worked example
Let me make the wobble concrete and then kill it. Instead of a full training run, I will simulate one: draw a handful of "scores" from a random generator and average them, the way a metric summarizes many random-influenced steps. First, the unpinned version — it uses the global generator with no seed, so it drifts.
import random
def unseeded_score():
"""Average five random draws from the shared global generator."""
draws = [random.random() for _ in range(5)]
return round(sum(draws) / len(draws), 4)
print(unseeded_score())
print(unseeded_score())
0.5123
0.4680
Two calls, two answers, no code change between them — exactly the Friday-versus- Monday problem. Now the fix. I create a private generator seeded with a fixed number and draw from that instead of the global one:
import random
def seeded_score(seed):
"""Average five draws from a generator pinned to `seed`."""
rng = random.Random(seed)
draws = [rng.random() for _ in range(5)]
return round(sum(draws) / len(draws), 4)
print(seeded_score(42))
print(seeded_score(42))
print(seeded_score(7))
0.3798
0.3798
0.3468
The two calls with seed 42 return the identical 0.3798: the run now repeats.
Seed 7 returns something different, which is the point — the seed selects a
sequence, and any fixed seed gives you a fixed, repeatable sequence. I reach for
random.Random(seed) rather than the global random.seed() on purpose: a private
generator cannot be disturbed by some other function elsewhere in the program
calling random.random(), so the reproducibility is local and robust. In a real
project you would seed every library that draws randomness — Python's random,
NumPy, and your framework — and pin the environment around them.
Hands-on
Your turn, with a different computation. The exercise below hands you a small
simulated training step that is currently unseeded and wobbles between runs. You
will make it reproducible by threading a seed argument through it and drawing
from a private generator, so that two calls with the same seed return the same
metric. It runs in Python Arena, so there is nothing to install.
Success criteria: calling your function twice with the same seed returns identical results, and two different seeds return different results. The Arena validates both properties automatically.
Recap
- You can state reproducibility precisely: same code, same data, same config yields the same model.
- You can name the hidden inputs that break it — unseeded randomness, unpinned libraries, a drifting environment, mutated data — and the fix for each.
- You can distinguish determinism (one computation repeats) from reproducibility (a whole run repeats) and explain why you need both.
- You can make a wobbling computation repeatable by seeding a private generator.
Next up: versioning. Seeds and pinned environments fix the "same code, same config" half of the promise; next you make the "same data" half real by versioning code, data, and models so you always know which inputs a result came from.
- Reproducible means same code, same data, same config yields the same model.
- Unseeded randomness and unpinned environments are the two most common breakers. - Set a seed for determinism; pin the environment (lockfile, container) for the rest. - Determinism repeats one computation; reproducibility repeats a whole run — you need both.