The AWS data stack

⏱ 35 min

Hook

Open the AWS console and search "data." You get back a wall of services — buckets, catalogs, crawlers, a query editor, three things with "Redshift" in the name, and a dozen more you have never heard of. A new engineer's instinct is to learn all of them. That is the wrong instinct, and it is how people end up running an always-on warehouse to answer a query that a serverless engine would have handled for pennies. The stack is smaller than it looks: four services carry almost every data pipeline, and each one does exactly one job.

Concept

The organizing idea of the whole AWS data stack is separation of storage and compute. Your data lives in one cheap, durable place and stays there; the engines that process and query it are separate, and you pay for them only while they run. This is the opposite of a traditional database, where storage and compute are welded together in an always-on server. On AWS, storage is the constant and compute is rented by the second.

The lake is Amazon S3 (Simple Storage Service) — object storage that holds any file, at any scale, for roughly two cents per GB per month. S3 is the lake: every other service reads from and writes to it. It stores objects (files) in buckets (top-level containers), addressed by a key that looks like a path. S3 does not know or care whether an object is a CSV, a Parquet file, or a JPEG; it just stores bytes durably and cheaply. Because storage is decoupled, the same bucket can feed a query engine, an ETL job, and a warehouse at once.

A lake of loose files is not queryable until something knows their schema — the column names and types. That something is the Glue Data Catalog, a central metadata store that records "the objects under this S3 prefix are a table named orders with these columns." Glue plays two roles: its catalog is the shared schema every query engine reads, and its crawlers can scan an S3 prefix and populate that schema automatically. Glue also runs ETL jobs — managed PySpark that transforms data — which the next lessons cover.

Once the catalog knows your tables, Athena queries them as SQL directly over S3, with no server to start or manage. Athena is the query engine: you write standard SQL, it reads the files S3 holds, and you pay per terabyte scanned. It is serverless, so an idle Athena costs nothing — there is nothing running to bill. That makes it the default for interactive and ad-hoc analysis over the lake.

When a workload needs a genuine warehouse — heavy concurrent dashboards, complex multi-table joins, sub-second repeat queries — Redshift Serverless fills that role. It is a columnar analytical database that scales compute on demand and bills per second of use rather than for an always-on cluster. The practitioner's rule: reach for Athena first because it is cheapest at rest, and graduate to Redshift only when query patterns justify a dedicated engine.

Role	Service	Bills on	Reach for it when
Lake	S3	GB stored per month	Always — it is where data lives
Catalog + ETL	Glue	Crawler + DPU-hours	You need schema and transformations
Query	Athena	TB scanned per query	Ad-hoc and interactive SQL over S3
Warehouse	Redshift Serverless	Compute-seconds used	Heavy concurrency or complex joins

🧠 Knowledge check

1. What does "separation of storage and compute" mean for the AWS data stack?

a) Data must be copied into every engine before it can be queriedb) Data lives durably in S3 while separate engines are rented only while they runc) Storage and compute always scale up and down together automaticallyd) Compute is free and only storage is billed

2. An analyst needs occasional ad-hoc SQL over files already sitting in S3, with no steady dashboard load. Which service is the cheapest default?

a) Redshift Serverless, because it is a real warehouseb) Athena, because it is serverless and bills only per queryc) Glue, because crawlers can answer SQLd) S3, because it can run SQL on its own

Worked example

Let me map a concrete pipeline to services and estimate its monthly bill, because the choice between Athena and Redshift is really a cost decision. Scenario: a retail team lands raw web-order events in S3, a nightly Glue job curates them into Parquet, and analysts run a handful of ad-hoc queries a day. The curated lake holds about 120 GB, Glue runs ten short jobs a month, and Athena scans roughly 0.8 TB across all queries.

I price each service on its own meter, then sum — as code, so the estimate is reproducible instead of a guess:

S3_RATE = 0.023        # dollars per GB-month
ATHENA_RATE = 5.0      # dollars per TB scanned
GLUE_DPU_HOUR = 0.44   # dollars per DPU-hour

lake_gb = 120
athena_tb_scanned = 0.8
glue_dpu_hours = 10 * 2 * 0.25   # 10 runs x 2 DPUs x 15 min each

s3_cost = lake_gb * S3_RATE
athena_cost = athena_tb_scanned * ATHENA_RATE
glue_cost = glue_dpu_hours * GLUE_DPU_HOUR
monthly = s3_cost + athena_cost + glue_cost

print(f"S3 storage: ${s3_cost:.2f}")
print(f"Athena:     ${athena_cost:.2f}")
print(f"Glue ETL:   ${glue_cost:.2f}")
print(f"monthly total: ${monthly:.2f}")

S3 storage: $2.76
Athena:     $4.00
Glue ETL:   $2.20
monthly total: $8.96

Under nine dollars a month, and the arithmetic tells me why this shape is right. The whole pipeline rests on cheap storage, and compute is bought only when a job runs or a query fires. Now imagine I had reached for an always-on Redshift cluster instead: even the smallest node runs around $0.25 an hour, which is roughly $180 a month whether or not anyone queries it — twenty times this entire bill, for a workload that is idle most of the day. The rates above are illustrative and shift by region, but the decision rule is stable: match the engine to the query pattern, and let idle mean free.

Hands-on

Your turn, with a different set of workloads. The exercise below hands you short descriptions of data tasks — "store 5 TB of raw logs," "run one ad-hoc SQL query a week," "power a 200-user live dashboard" — and asks you to route each to the service whose job it is. It runs in Python Arena, so there is nothing to install.

▶ Python Arenade203-route-data-to-services

Open in Python Arena ↗

Success criteria: your router returns s3 for storage tasks, athena for ad-hoc SQL, glue for catalog-and-transform tasks, and redshift for heavy-concurrency warehouse tasks. The Arena validates your routing against a hidden set of workloads automatically.

Recap

• You can name the four core services and the single role each plays: S3 the lake, Glue the catalog and ETL, Athena the query engine, Redshift Serverless the warehouse.
• You can explain separation of storage and compute and why it lets an idle engine cost nothing.
• You can choose Athena over Redshift for ad-hoc SQL, and justify the choice on query pattern and cost.

Next up: the lake itself — how to lay out S3 in bronze, silver, and gold layers, with partitioning and file formats that keep every one of those Athena scans small.

🧠 Knowledge check

1. Why does the Glue Data Catalog matter for a lake of loose files in S3?

a) It compresses the files so they take less storageb) It records the schema so query engines can read the files as tablesc) It is the only service allowed to write to S3d) It runs the SQL queries itself instead of Athena

2. A team runs an always-on Redshift cluster to serve two ad-hoc queries a day. What is the core mistake?

a) Redshift cannot run SQL, so it is the wrong tool entirelyb) They pay for a dedicated engine sitting idle when a serverless one would bill only per queryc) Ad-hoc queries are impossible on AWSd) S3 should be storing the queries, not the data

📌 Key takeaways

• The AWS data stack is four services with one job each: S3 (lake), Glue (catalog + ETL), Athena (query), Redshift Serverless (warehouse). - Storage and compute are separated: data rests cheaply in S3, and engines are rented only while they run, so idle can mean free. - Default to Athena for ad-hoc SQL and graduate to Redshift only when concurrency or query complexity justifies a dedicated engine.