Lab 1 — Simulate an S3 medallion lake in pure Python

Objective

By the end of this lab you will have a runnable Python program, medallion_lake.py, that models an S3 medallion lake end to end: it reads raw "objects" from a bronze "bucket," runs a pure-Python "Glue job" that cleans and deduplicates them into silver, aggregates silver into a gold table partitioned by date, and answers a revenue query by reading only one partition. Everything runs on your own machine with the Python standard library — no AWS account, no boto3, no network — so it costs nothing and can never bill you.

This is a faithful local model of the AWS pattern, not a real deployment. Local directories stand in for S3 buckets, CSV and JSON files stand in for S3 objects, a plain Python function stands in for a Glue ETL job, and a directory scan stands in for an Athena query. The shape is exactly the medallion architecture from the lessons; the hands-on real-AWS versions of this flow live in the Cloud school's platform courses. Master the pattern here for free.

Skills exercised:

Laying out a lake in bronze, silver, and gold prefixes
Writing a transform that cleans, validates, and deduplicates raw records
Producing a gold table partitioned Hive-style by date
Answering a query by pruning to a single partition

Prerequisites and setup

Prerequisites: the three lessons in this module — The AWS data stack, The S3 medallion lakehouse, and ETL with Glue and querying with Athena.

Setup: you need Python 3.10 or newer and nothing else — no packages, no cloud account. Verify your version and create a working folder.

macOS / Linux
Windows (PowerShell)

python3 --version
mkdir de203-medallion-lab && cd de203-medallion-lab

python --version
mkdir de203-medallion-lab; cd de203-medallion-lab

Expected output from the version command is a line like Python 3.11.6. If the command is not found, install Python from python.org before continuing. In the steps below, use python3 on macOS/Linux and python on Windows.

note

Every step writes files under this one folder. If you close your terminal and come back, re-run the cd de203-medallion-lab command to return to it; all your files are still there, so you can resume at the step you left off.

Step 1 — Land raw objects in the bronze bucket

Goal: create the lake's directory structure and land two raw "objects" in bronze — one CSV export and one JSON export, each a day of orders, each carrying deliberate defects your Glue job will clean out.

The lake/ folder is your "S3 bucket," and each subdirectory is a prefix. Create the bronze layout:

macOS / Linux
Windows (PowerShell)

mkdir -p lake/bronze/orders_csv lake/bronze/orders_json

mkdir lake\bronze\orders_csv, lake\bronze\orders_json

Save this exactly as lake/bronze/orders_csv/orders_2026-07-01.csv:

order_id,order_ts,category,amount,status
1001,2026-07-01T08:00:00,books,20.00,paid
1002,2026-07-01T09:00:00,electronics,300.00,paid
1003,2026-07-01T10:00:00,books,,paid
1004,2026-07-01T11:00:00,electronics,150.00,refunded
1002,2026-07-01T09:05:00,electronics,300.00,paid

Save this exactly as lake/bronze/orders_json/orders_2026-07-02.json:

[
  {
    "order_id": "2001",
    "order_ts": "2026-07-02T08:00:00",
    "category": "books",
    "amount": "50.00",
    "status": "paid"
  },
  {
    "order_id": "2002",
    "order_ts": "2026-07-02T09:00:00",
    "category": "toys",
    "amount": "-5.00",
    "status": "paid"
  },
  {
    "order_id": "2003",
    "order_ts": "2026-07-02T10:00:00",
    "category": "electronics",
    "amount": "250.00",
    "status": "paid"
  },
  {
    "order_id": "",
    "order_ts": "2026-07-02T11:00:00",
    "category": "books",
    "amount": "10.00",
    "status": "paid"
  }
]

These objects carry defects on purpose: 1003 is missing its amount, 1004 is refunded rather than paid, the second 1002 line is a duplicate order, 2002 has a negative amount, and the last JSON record has an empty order_id. Your silver job will drop all five.

Checkpoint: confirm both objects landed and are readable.

python3 -c "import csv, json; print(len(list(csv.DictReader(open('lake/bronze/orders_csv/orders_2026-07-01.csv')))), len(json.load(open('lake/bronze/orders_json/orders_2026-07-02.json'))))"

You should see 5 4 — five CSV data rows and four JSON records, nine raw objects in all. If you get a FileNotFoundError, re-check that both files are saved at exactly the paths above.

Step 2 — Read the bronze prefix

Goal: start medallion_lake.py with the lake's path constants and a read_bronze reader that streams every raw object under the bronze prefix — the local stand-in for a Glue crawler discovering objects in a bucket.

Create medallion_lake.py with this content:

import csv
import json
from pathlib import Path

LAKE = Path(__file__).parent / "lake"
BRONZE = LAKE / "bronze"
SILVER = LAKE / "silver"
GOLD = LAKE / "gold" / "daily_category_revenue"


def read_bronze(bronze_dir):
    """Yield every raw object under the bronze prefix, CSV then JSON."""
    for path in sorted(bronze_dir.rglob("*.csv")):
        with open(path, newline="", encoding="utf-8") as f:
            for row in csv.DictReader(f):
                yield row
    for path in sorted(bronze_dir.rglob("*.json")):
        with open(path, encoding="utf-8") as f:
            for row in json.load(f):
                yield row

Checkpoint: count the raw objects the reader streams.

python3 -c "import medallion_lake as m; print(sum(1 for _ in m.read_bronze(m.BRONZE)))"

You should see 9 — the five CSV rows plus the four JSON records, streamed as one sequence. If you get ModuleNotFoundError: No module named 'medallion_lake', you are in the wrong folder; cd into de203-medallion-lab where the file lives.

Step 3 — Run the Glue job into silver

Goal: add to_silver, the pure-Python stand-in for a Glue ETL job. It applies the same rules a real PySpark job would — filter to paid orders, coerce amount to a number, drop non-positive amounts and empty ids, deduplicate by order_id — and derives an order_date for partitioning downstream.

Add this function to medallion_lake.py:

def to_silver(rows):
    """Clean, validate, and deduplicate raw rows into silver records."""
    seen = set()
    for row in rows:
        order_id = (row.get("order_id") or "").strip()
        if not order_id:
            continue
        if row.get("status") != "paid":
            continue
        try:
            amount = float((row.get("amount") or "").strip())
        except ValueError:
            continue
        if amount <= 0:
            continue
        if order_id in seen:
            continue
        seen.add(order_id)
        yield {
            "order_id": order_id,
            "order_date": row["order_ts"][:10],
            "category": row["category"],
            "amount": round(amount, 2),
        }

Checkpoint: count the records that survive into silver.

python3 -c "import medallion_lake as m; print(sum(1 for _ in m.to_silver(m.read_bronze(m.BRONZE))))"

You should see 4 — orders 1001, 1002, 2001, and 2003. The other five raw rows were dropped by the rules above. If you see 9, none of your guards are firing; if you see 2, a guard is too aggressive — check that the dedup seen set is only skipping exact repeats.

Step 4 — Aggregate into partitioned gold

Goal: add write_gold, which rolls silver up into revenue per (order_date, category) and writes each date to its own Hive-style partition directory (order_date=…) — the local model of partitioned Parquet in a gold prefix.

In a real lake these gold files would be Parquet; here they are JSON Lines so the lab stays stdlib-only, but the layout — one directory per partition value — is exactly what Athena would prune over.

Add this function to medallion_lake.py:

def write_gold(silver_rows, gold_dir):
    """Aggregate revenue by (date, category); write partitioned by date."""
    totals = {}
    for r in silver_rows:
        key = (r["order_date"], r["category"])
        totals[key] = round(totals.get(key, 0.0) + r["amount"], 2)
    for (order_date, category), revenue in sorted(totals.items()):
        part = gold_dir / f"order_date={order_date}"
        part.mkdir(parents=True, exist_ok=True)
        record = json.dumps({"category": category, "revenue": revenue})
        with open(part / "part.jsonl", "a", encoding="utf-8") as f:
            f.write(record + "\n")
    return totals

Checkpoint: smoke-test the aggregation into a throwaway directory.

python3 -c "import medallion_lake as m; from pathlib import Path; t = m.write_gold(m.to_silver(m.read_bronze(m.BRONZE)), Path('lake/_check')); print(len(t), round(sum(t.values()), 2))"

You should see 4 620.0 — four (date, category) cells summing to 620.00 in revenue. You can delete the smoke-test directory with rm -rf lake/_check (macOS/Linux) or Remove-Item -Recurse lake\_check (Windows); the real run in Step 5 writes the true gold prefix.

Step 5 — Query gold with partition pruning, end to end

Goal: add query_gold_day, which answers a one-day revenue question by reading only that day's partition — the local model of Athena pruning to a single order_date= directory — then add a __main__ block that runs the whole pipeline and prints a summary.

Add this function and the entry point to medallion_lake.py:

def query_gold_day(gold_dir, order_date):
    """Sum revenue for one day, reading only that day's partition."""
    part = gold_dir / f"order_date={order_date}"
    total = 0.0
    for path in part.glob("*.jsonl"):
        for line in path.read_text(encoding="utf-8").splitlines():
            total += json.loads(line)["revenue"]
    return round(total, 2)


if __name__ == "__main__":
    bronze_rows = list(read_bronze(BRONZE))
    silver_rows = list(to_silver(bronze_rows))
    SILVER.mkdir(parents=True, exist_ok=True)
    with open(SILVER / "orders.jsonl", "w", encoding="utf-8") as f:
        for r in silver_rows:
            f.write(json.dumps(r) + "\n")
    totals = write_gold(silver_rows, GOLD)
    print(f"bronze objects read: {len(bronze_rows)}")
    print(f"silver records kept: {len(silver_rows)}")
    print(f"gold cells written:  {len(totals)}")
    print(f"2026-07-01 revenue:  {query_gold_day(GOLD, '2026-07-01')}")

Checkpoint: run the whole pipeline.

python3 medallion_lake.py

You should see exactly:

bronze objects read: 9
silver records kept: 4
gold cells written:  4
2026-07-01 revenue:  320.0

The query read only the order_date=2026-07-01 partition to get 320.00, ignoring the 2026-07-02 files entirely — that is partition pruning. A lake/gold/ tree now exists with one directory per date. If 2026-07-01 revenue is not 320.0, re-run after confirming Step 3 kept exactly four records.

Validation

Verify the whole build against the objective: a correct, partitioned gold table. Save this as validate_lake.py:

import json

import medallion_lake as lake

GOLD = lake.GOLD
EXPECTED_CELLS = 4
EXPECTED_DAY1 = 320.00
EXPECTED_GRAND = 620.00

cells = 0
grand = 0.0
for part in sorted(GOLD.glob("order_date=*")):
    for path in part.glob("*.jsonl"):
        for line in path.read_text(encoding="utf-8").splitlines():
            grand += json.loads(line)["revenue"]
            cells += 1
grand = round(grand, 2)
day1 = lake.query_gold_day(GOLD, "2026-07-01")

if cells == EXPECTED_CELLS and day1 == EXPECTED_DAY1 and grand == EXPECTED_GRAND:
    print(
        f"PASSED — gold has {cells} cells, 2026-07-01 revenue {day1:.2f}, "
        f"grand total {grand:.2f}"
    )
else:
    print(f"FAILED — cells={cells} day1={day1} grand={grand}")

Run it:

python3 validate_lake.py

Checkpoint: the command prints PASSED — gold has 4 cells, 2026-07-01 revenue 320.00, grand total 620.00. If it prints FAILED, delete the lake/gold directory and re-run python3 medallion_lake.py first — a stale or double-written gold tree (from running the pipeline twice without clearing it) is the usual cause, because write_gold appends.

Teardown

Nothing is billable — this lab is a local model and runs entirely on your machine, so teardown is a one-line no-op: there are no cloud resources to delete. Keep the de203-medallion-lab folder if you want to reuse medallion_lake.py as a reference; otherwise remove it with rm -rf de203-medallion-lab (macOS/Linux) or Remove-Item -Recurse de203-medallion-lab (Windows).

tip

On real AWS this section would be the most important one in the lab: delete the S3 bucket's objects, drop the Glue Catalog tables, and confirm the console shows nothing running, so the account returns to a $0 resting state. The habit of scrolling to Teardown before you walk away is what this no-op is training.

Troubleshooting

Symptom	Likely cause	Fix
`ModuleNotFoundError: No module named 'medallion_lake'`	Running from the wrong folder	`cd` into `de203-medallion-lab`, where `medallion_lake.py` lives
`FileNotFoundError: 'lake/bronze/...'`	A bronze object is missing or misnamed	Re-save the Step 1 files at exactly the paths shown
Step 2 checkpoint prints `8` instead of `9`	CSV missing the duplicate `1002` row	Re-save `orders_2026-07-01.csv` with all five data rows from Step 1
Step 3 checkpoint prints `9`	Silver guards not returning early	Ensure each rule uses `continue` to skip the bad row, not `pass`
Grand total is `1240.0` or gold cells are `8`	Pipeline run twice; `write_gold` appended	Delete `lake/gold`, then re-run `python3 medallion_lake.py` once
Validation prints `FAILED` with `day1=0.0`	Wrong partition directory name	Confirm gold partitions are named `order_date=YYYY-MM-DD`

Objective​

Prerequisites and setup​

Step 1 — Land raw objects in the bronze bucket​

Step 2 — Read the bronze prefix​

Step 3 — Run the Glue job into silver​

Step 4 — Aggregate into partitioned gold​

Step 5 — Query gold with partition pruning, end to end​

Validation​

Teardown​

Troubleshooting​