Experiment tracking basics
Hook
You spend a day tuning a model, trying learning rates and tree depths in a notebook, jotting the good numbers in a scratch cell. The best run hit 0.94. Now, two days later, you cannot reconstruct which combination produced it — the cell has been re-executed, the variables overwritten, the "0.94" a number with no config attached. The work happened; the knowledge did not survive.
Concept
An experiment is not one model, it is many — a search over configurations — and the thing worth keeping is not the best model but the record of how you found it. Experiment tracking is the practice of writing down every run as a structured entry so that a result is never separated from the inputs that produced it.
The unit is a run: one execution of your training code with one configuration. A well-formed run record has four parts. Params are the inputs you chose — learning rate, tree depth, the random seed. Metrics are the outputs you measured — accuracy, loss, training time. Artifacts are the files the run produced or consumed — the model hash and the data hash from the previous lesson. And metadata ties it to the world — the code commit and a timestamp. Together these make a run immutable and self-describing: reading the entry alone, you know exactly what was tried and what happened, with no notebook state required.
Once runs are logged this way, comparing runs becomes a query instead of a memory test. To find the winner you sort the log by a metric. To understand why one run beat another, you hold the params side by side and read off what differed — same data hash, same seed, depth 6 instead of depth 3, accuracy up five points. Because each run carries its data hash and commit, a comparison is honest: you are never accidentally comparing a run on last week's data with one on this week's.
This is exactly what a self-hosted tracking server such as an MLflow-style tool gives you at scale — a database of runs with a UI to sort and compare them. The server adds convenience, storage, and a team-wide view, but the idea underneath is the plain structured log you are about to build: params in, metrics out, artifacts and metadata attached, nothing thrown away.
| Run field | Example | Answers the question |
|---|---|---|
| Params | {learning_rate: 0.1, depth: 6} | What did I try? |
| Metrics | {accuracy: 0.91} | What happened? |
| Artifacts | data_hash: 7603..., model_hash: 02e1... | On which exact inputs? |
| Metadata | commit: a1b2c3, timestamp: ... | When, and from what code? |
Worked example
Let me build the smallest tracking log that still works: a list of run records, logged as I "train," then queried for the best. I will use plain dicts so the structure is obvious — this is the same shape a tracking server stores in a database.
runs = []
def log_run(params, metrics):
"""Append one run record to the in-memory tracking log."""
runs.append({"params": params, "metrics": metrics})
# Three runs of a search over learning rate and tree depth.
log_run({"learning_rate": 0.1, "depth": 3}, {"accuracy": 0.86})
log_run({"learning_rate": 0.1, "depth": 6}, {"accuracy": 0.91})
log_run({"learning_rate": 0.5, "depth": 6}, {"accuracy": 0.88})
best = max(runs, key=lambda r: r["metrics"]["accuracy"])
print("runs logged: ", len(runs))
print("best accuracy:", best["metrics"]["accuracy"])
print("best params: ", best["params"])
runs logged: 3
best accuracy: 0.91
best params: {'learning_rate': 0.1, 'depth': 6}
The winning run is not a number I have to remember — it is a record I can query,
and its params come attached. Reading the log, the comparison explains itself:
runs one and two share learning_rate 0.1 and differ only in depth, so the
jump from 0.86 to 0.91 is attributable to depth, not luck. Run three tells the
complementary story — raising the learning rate to 0.5 at the same depth lost
three points. That is the payoff of tracking: differences between runs become
readable instead of anecdotal. A real logger would also store each run's data
hash, model hash, and commit so the winner is fully reproducible, which is
precisely what you will add in the lab.
Hands-on
Your turn, with a richer log. The exercise below hands you several run records that each already carry params, metrics, and a data hash, and asks you to write the query that selects the best run by a chosen metric and confirms every run shared the same data hash. It runs in Python Arena, so there is nothing to install.
Success criteria: your query returns the run with the highest metric and flags whether all runs used the same data hash. The Arena validates both against a hidden set of runs automatically.
Recap
- You can define a run as one execution recorded with its params, metrics, artifacts, and metadata.
- You can explain why a run must be immutable and self-describing — a result is worthless once separated from the config that produced it.
- You can compare runs by sorting on a metric and reading off which params differed to explain the gap.
- You can see how a self-hosted tracking server is this same structured log, scaled up with storage and a UI.
Next up: the lab. You will assemble everything — a seed from Lesson 1, a content hash from Lesson 2, and a run log from this lesson — into a small reproducible- experiment logger you run on your own machine.
- A run is one execution recorded with params, metrics, artifacts, and metadata. - Log every run so a result is never separated from the config that produced it. - Comparing runs is sorting the log by a metric and reading off what differed. - A tracking server is this structured log scaled with storage and a UI.