Lab 1 — Build a reproducible experiment logger

Objective

By the end of this lab you will have a runnable Python program, experiment.py, that fingerprints an input dataset with a content hash, runs a seeded mock experiment twice, and appends each run — params, metrics, data hash, run hash, and a passed-in timestamp — to a JSON log. Because the run is seeded and the data is hashed, the two runs produce identical fingerprints, and a validation script proves it. It is a hand-built version of the tracking servers the rest of the school uses, and it runs on only the Python standard library (hashlib, json, random, pathlib), so it costs nothing and runs anywhere.

Skills exercised:

Seeding a computation with a private generator so a run repeats exactly
Fingerprinting a dataset with a SHA-256 content hash
Appending self-describing run records to a JSON log
Proving determinism by comparing two runs' hashes

Prerequisites and setup

Prerequisites: the three lessons in this module — Reproducibility in machine learning, Versioning code, data, and models, and Experiment tracking basics.

Setup: you need Python 3.10 or newer. There are no packages to install; everything uses the standard library. Verify your version and create a working folder.

macOS / Linux
Windows (PowerShell)

python3 --version
mkdir ml101-logger-lab && cd ml101-logger-lab

python --version
mkdir ml101-logger-lab; cd ml101-logger-lab

Expected output from the version command is a line like Python 3.11.6. If the command is not found, install Python from python.org before continuing. In the steps below, use python3 on macOS/Linux and python on Windows.

note

Every step writes or edits files in this one folder. If you close your terminal and come back, re-run the cd ml101-logger-lab command to return to it; your files are still there. experiment.py is built up across Steps 2 to 4 — keep adding to the same file rather than replacing it.

Step 1 — Create the input dataset

Goal: create the dataset your experiment will fingerprint. Its exact bytes determine the data hash, so it must be saved exactly as shown.

Save this exactly as dataset.csv:

sample_id,feature,label
1,0.42,churn
2,0.71,stay
3,0.15,stay
4,0.93,churn
5,0.55,stay

Checkpoint: confirm the file exists with the expected number of data rows.

python3 -c "print(sum(1 for _ in open('dataset.csv')) - 1, 'data rows')"

You should see 5 data rows. If you get a FileNotFoundError, the file is missing or misnamed — re-save it as dataset.csv in the current folder.

Step 2 — Hash the dataset and run a seeded experiment

Goal: create experiment.py with two functions — one that fingerprints the dataset, and one that runs a seeded, deterministic mock experiment and returns its metric plus a reproducibility fingerprint.

Create experiment.py with this content:

import hashlib
import json
import random
from pathlib import Path

LOG_PATH = Path("runs.json")


def hash_dataset(path):
    """Return the SHA-256 fingerprint of a file, read in chunks."""
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            digest.update(chunk)
    return digest.hexdigest()


def run_experiment(data_hash, seed, params):
    """Run a seeded, deterministic mock experiment.

    Returns (metrics, run_hash). The run_hash fingerprints everything that
    must match for two runs to count as identical: the data, the seed, the
    params, and the resulting metric. It excludes the wall-clock timestamp,
    which is allowed to differ between runs.
    """
    rng = random.Random(seed)
    draws = [rng.random() for _ in range(params["samples"])]
    accuracy = round(sum(draws) / len(draws), 6)
    metrics = {"accuracy": accuracy}
    fingerprint = f"{data_hash}|{seed}|{params['samples']}|{accuracy}"
    run_hash = hashlib.sha256(fingerprint.encode("utf-8")).hexdigest()
    return metrics, run_hash

The experiment stands in for real training: instead of fitting a model, it draws samples numbers from a generator pinned to seed and averages them into an accuracy. Because the generator is private and seeded, the same inputs always yield the same metric — determinism from Lesson 1.

Checkpoint: fingerprint the dataset and run the experiment once.

python3 -c "import experiment as e; h = e.hash_dataset('dataset.csv'); \
m, rh = e.run_experiment(h, 42, {'samples': 100}); \
print('data_hash', h[:12]); print('accuracy', m['accuracy']); \
print('run_hash', rh[:12])"

You should see exactly:

data_hash 7603461386a6
accuracy 0.479569
run_hash 02e1eb71ceab

If your data_hash differs, dataset.csv does not match Step 1 byte for byte — re-save it. If accuracy differs, check that run_experiment seeds a private random.Random(seed) rather than the global generator.

Step 3 — Build a run record and append it to the log

Goal: add the two functions that turn a run into a self-describing record and append it to a JSON log on disk.

Add these two functions to experiment.py, below run_experiment:

def make_record(params, metrics, data_hash, run_hash, timestamp):
    """Build one self-describing run record."""
    return {
        "params": params,
        "metrics": metrics,
        "data_hash": data_hash,
        "run_hash": run_hash,
        "timestamp": timestamp,
    }


def append_run(log_path, record):
    """Append one run record to a JSON-array log; return the new count."""
    if log_path.exists():
        runs = json.loads(log_path.read_text(encoding="utf-8"))
    else:
        runs = []
    runs.append(record)
    log_path.write_text(json.dumps(runs, indent=2), encoding="utf-8")
    return len(runs)

The timestamp is passed in rather than read from the clock inside the function. That keeps the record honest about wall-clock time while leaving the run's reproducibility fingerprint — computed in Step 2 without the timestamp — untouched.

Checkpoint: append one throwaway record to a scratch log, then remove it.

python3 -c "import experiment as e; from pathlib import Path; \
p = Path('smoke.json'); p.unlink(missing_ok=True); \
r = e.make_record({'samples': 1}, {'accuracy': 0.5}, 'd', 'rh', '2026-01-01T00:00:00'); \
print('rows', e.append_run(p, r)); p.unlink()"

You should see rows 1. The scratch smoke.json is created and then deleted, so your real runs.json stays empty until Step 5. If you see a TypeError about JSON serialization, confirm make_record returns plain dicts and lists only.

Step 4 — Wire the two runs together

Goal: add the entry point that hashes the dataset once, runs the experiment twice with the same seed, and appends both records to runs.json.

Add this to the bottom of experiment.py:

if __name__ == "__main__":
    seed = 42
    params = {"samples": 100}
    data_hash = hash_dataset("dataset.csv")

    for timestamp in ("2026-07-17T09:00:00", "2026-07-17T09:05:00"):
        metrics, run_hash = run_experiment(data_hash, seed, params)
        record = make_record(params, metrics, data_hash, run_hash, timestamp)
        count = append_run(LOG_PATH, record)
        print(f"run {count}: accuracy={metrics['accuracy']} "
              f"run_hash={run_hash[:12]}")

Checkpoint: confirm the entry point parses by asking Python to compile the file without running it.

python3 -c "import py_compile; py_compile.compile('experiment.py', doraise=True); print('experiment.py compiles')"

You should see experiment.py compiles. If you get an IndentationError, the if __name__ block was pasted at the wrong indentation — it must start at the left margin, not inside another function.

Step 5 — Run the logger end to end

Goal: run the whole program and inspect the log it writes.

warning

append_run appends; it never overwrites. If you have run experiment.py before, delete runs.json first so the log holds exactly the two runs from this invocation. On macOS/Linux: rm -f runs.json. On Windows PowerShell: Remove-Item runs.json -ErrorAction SilentlyContinue.

python3 experiment.py

Checkpoint: the two printed lines are identical except for the run number.

run 1: accuracy=0.479569 run_hash=02e1eb71ceab
run 2: accuracy=0.479569 run_hash=02e1eb71ceab

Both runs report the same accuracy and the same run_hash — that is determinism you can see. A runs.json file now holds two records with matching data_hash and run_hash values. If the two lines differ, the experiment is not seeded the same way on both passes — re-check the loop in Step 4 uses the one seed variable for both iterations.

Validation

Verify the whole build against the objective: two logged runs whose data hash and run hash are byte-for-byte identical. Save this as validate.py:

import json
from pathlib import Path

runs = json.loads(Path("runs.json").read_text(encoding="utf-8"))

if len(runs) < 2:
    print("FAILED - need at least 2 runs, found %d" % len(runs))
    raise SystemExit(0)

first, second = runs[-2], runs[-1]
same_data = first["data_hash"] == second["data_hash"]
same_run = first["run_hash"] == second["run_hash"]

if same_data and same_run:
    fingerprint = first["run_hash"][:12]
    print("PASSED - two runs produced identical hashes:", fingerprint)
else:
    print("FAILED - runs diverged (same_data=%s same_run=%s)"
          % (same_data, same_run))

Run it:

python3 validate.py

Checkpoint: the command prints PASSED - two runs produced identical hashes: 02e1eb71ceab. The validator compares the two most recent runs in the log, so it still passes if you ran experiment.py more than once. If it prints FAILED, re-run python3 experiment.py after deleting runs.json, then validate again.

Teardown

Nothing is billable — this lab runs entirely on your machine. Keep the ml101-logger-lab folder if you want to reuse experiment.py as a reference for later course work; otherwise remove it with rm -rf ml101-logger-lab (macOS/Linux) or Remove-Item -Recurse ml101-logger-lab (Windows). You can delete the generated runs.json and any leftover smoke.json at any time.

Troubleshooting

Symptom	Likely cause	Fix
`data_hash` differs from `7603461386a6` in Step 2	`dataset.csv` does not match Step 1 bytes	Re-save `dataset.csv` exactly, with no trailing edits or extra rows
`accuracy` differs from `0.479569` in Step 2	Experiment uses the global generator	Seed a private `random.Random(seed)` inside `run_experiment`
`ModuleNotFoundError: No module named 'experiment'`	Running from the wrong folder	`cd` into `ml101-logger-lab`, where `experiment.py` lives
Step 5 shows `run 3` / `run 4`	Leftover `runs.json` from an earlier run	Delete `runs.json`, then re-run `python3 experiment.py`
Validation prints `FAILED` with `same_run=False`	The two runs used different seeds	Confirm the Step 4 loop reuses the one `seed` for both iterations
`JSONDecodeError` when running `validate.py`	`runs.json` was hand-edited or truncated	Delete `runs.json` and re-run `python3 experiment.py` to regenerate it

Objective​

Prerequisites and setup​

Step 1 — Create the input dataset​

Step 2 — Hash the dataset and run a seeded experiment​

Step 3 — Build a run record and append it to the log​

Step 4 — Wire the two runs together​

Step 5 — Run the logger end to end​

Validation​

Teardown​

Troubleshooting​