The S3 medallion lakehouse

⏱ 35 min

Hook

Six months into a project, the lake has one bucket and everyone writes wherever they like: raw dumps, half-cleaned extracts, and a colleague's one-off export all share a prefix. Nobody can tell which files are trustworthy, a single "revenue by day" query in Athena scans the entire bucket, and the bill for that one query is larger than a month of storage. This is a data swamp, and it is what a lake becomes without structure. The fix is not a new tool — it is a layout.

Concept

The medallion architecture organizes a lake into three layers of increasing quality and value: bronze, silver, and gold. Each layer is just an S3 prefix, and data flows one direction through them, getting cleaner and more refined at every hop. The layers map directly onto the staging-core-mart pattern from DE-201; on S3 they become directories.

Bronze is raw data landed exactly as it arrived — original format, original values, nothing dropped. Bronze is append-only and immutable: it is your system of record, the thing you can always reprocess from if a downstream bug corrupts everything else. Silver is cleaned and conformed: bad rows removed, types coerced, duplicates dropped, one row meaning one thing. Gold is business-ready: aggregated, joined, and shaped to answer specific questions — the tables analysts and dashboards actually read. The discipline is that each layer reads only from the one before it, so lineage runs in a straight line.

Layering keeps the lake trustworthy, but two more choices keep it cheap, and both attack the same enemy: bytes scanned. The first is partitioning — splitting a table's files into subdirectories named column=value, called Hive-style partitions. A table partitioned by date lays out as gold/revenue/order_date=2026-07-01/ and so on. When a query filters on that column, the engine reads only the matching directories and skips the rest, a saving called partition pruning. Partition on the columns you filter by — date is the classic choice; a high-cardinality column like customer_id creates millions of tiny directories and makes things worse.

The second choice is the file format. Bronze often arrives as CSV or JSON — row-oriented text, which an engine must read in full to answer any query. Silver and gold should be Parquet: a columnar, compressed format that stores each column separately and carries min/max statistics per block. A query touching three of twelve columns reads only those three, compression shrinks the bytes several-fold, and the statistics let the engine skip blocks that cannot match. Partitioning cuts which files you read; Parquet cuts how much of each file you read. Together they turn a terabyte scan into a megabyte one.

Layer	Content	Format	Partitioned by	Read by
Bronze	Raw, as-landed, immutable	CSV / JSON	ingest date	ETL jobs only
Silver	Cleaned, typed, deduplicated	Parquet	event date	ETL and analysts
Gold	Aggregated, business-ready	Parquet	query key/date	dashboards, Athena

🧠 Knowledge check

1. Why is the bronze layer kept raw, append-only, and immutable?

a) Because raw CSV is faster to query than Parquetb) Because it is the system of record you can always reprocess from if a downstream layer is corruptedc) Because analysts query bronze directly for dashboardsd) Because S3 forbids editing objects once written

2. A query filters on order_date. How does partitioning the table by order_date reduce cost?

a) It compresses every file to zero bytesb) The engine reads only the matching date directories and skips the rest — partition pruningc) It converts the query from SQL to Python automaticallyd) It moves the data into Redshift for free

Worked example

Let me put a price on layout, because the case for Parquet-on-partitions is easiest to see in dollars. Scenario: a year of daily order data, roughly 2 GB per day as raw CSV, and a common query that asks for a single day's revenue. Athena bills at $5 per TB scanned. I will compare three layouts for that one-day query.

ATHENA_RATE = 5.0            # dollars per TB scanned
days = 365
csv_gb_per_day = 2.0
full_csv_gb = days * csv_gb_per_day

unpartitioned_scan_gb = full_csv_gb      # no pruning: scan the whole year
partitioned_scan_gb = csv_gb_per_day     # prune to one day's directory
# Parquet: query touches 3 of 12 columns, ~4x compression.
parquet_scan_gb = partitioned_scan_gb * (3 / 12) / 4


def cost(gb):
    return (gb / 1024) * ATHENA_RATE


print(f"unpartitioned CSV:   {unpartitioned_scan_gb:6.1f} GB  ${cost(unpartitioned_scan_gb):.4f}")
print(f"partitioned CSV:     {partitioned_scan_gb:6.1f} GB  ${cost(partitioned_scan_gb):.4f}")
print(f"partitioned Parquet: {parquet_scan_gb:6.3f} GB  ${cost(parquet_scan_gb):.6f}")

unpartitioned CSV:    730.0 GB  $3.5645
partitioned CSV:        2.0 GB  $0.0098
partitioned Parquet:    0.125 GB  $0.0006

The same question, three layouts, a 6,000-fold spread in what it scans. Partitioning alone takes the query from scanning the whole year to scanning one day — 365 times fewer bytes — because pruning skips every non-matching directory. Switching that day's file to Parquet cuts another factor of sixteen, because the query reads only 3 of the 12 columns and compression shrinks each of those. The unpartitioned CSV query costs more than three dollars; the partitioned-Parquet version costs six hundredths of a cent, and it returns in a fraction of the time. Layout is not housekeeping — it is the single biggest lever on both cost and speed in the lake.

Hands-on

Your turn, with a different lake. The exercise below hands you descriptions of datasets — "raw JSON exactly as the API returned it," "deduplicated orders with coerced types," "daily revenue rolled up by category" — and asks you to assign each to the bronze, silver, or gold layer and build its partition path. It runs in Python Arena, so there is nothing to install.

▶ Python Arenade203-assign-medallion-layer

Open in Python Arena ↗

Success criteria: each dataset maps to the correct layer, and your partition path for a dated gold table renders as gold/<table>/order_date=<value>/. The Arena validates your assignments against a hidden set of datasets automatically.

Recap

• You can lay out an S3 lake in bronze, silver, and gold layers and explain what each contains and who reads it.
• You can partition a table by the column queries filter on, and explain how pruning skips non-matching directories.
• You can justify columnar Parquet over row-based CSV for silver and gold, and estimate the scan savings the two choices combine to give.

Next up: the jobs and queries that move data through these layers — Glue ETL that writes the Parquet, and Athena that reads it — plus the cost drivers and teardown that keep the whole thing at $0 when you walk away.

🧠 Knowledge check

1. Why should silver and gold use Parquet instead of the CSV that bronze arrives as?

a) Parquet files cannot be deleted, so the data is saferb) Parquet is columnar and compressed, so a query reads only the columns it needs and far fewer bytesc) Athena refuses to read CSV under any circumstancesd) CSV cannot store more than one column

2. Which column is the poor choice to partition an orders table by, and why?

a) order_date, because dates change every dayb) customer_id, because its high cardinality creates millions of tiny directoriesc) A low-cardinality region code, because there are too few valuesd) Any column — partitioning always helps equally

📌 Key takeaways

• The medallion architecture layers a lake into bronze (raw, immutable), silver (cleaned, typed), and gold (aggregated), each an S3 prefix reading only from the one before it. - Hive-style partitioning (column=value directories) lets a filtered query prune to matching directories and skip the rest. - Columnar, compressed Parquet for silver and gold reads only the needed columns; combined with partitioning it turns terabyte scans into megabyte ones.