Data Engineering on AWS (DE-203)
You know how to model a warehouse and how to write PySpark that survives a 50 million-row join. Now someone hands you an AWS account and a Slack message: "the raw clickstream lands in a bucket every hour, analysts want a SQL table by Friday, and the finance team is watching the bill." Nothing you learned in DE-201 or DE-202 tells you which of the forty-plus AWS services to reach for, where the data physically lives, or how a "just for testing" cluster quietly turns into a four-figure invoice.
DE-203 is the course that answers those questions. It teaches the AWS data stack the way a pipeline engineer actually uses it: a small set of services, each with one clear job, composed around cheap object storage. You will learn where your PySpark runs on AWS, how a lake becomes queryable SQL without a database server, and — running through every lesson like a spine — how to reason about cost before you create a resource and how to tear everything back down to nothing when you are done.
- Name the core AWS data services — S3, Glue, Athena, Redshift Serverless — and explain the single role each plays in a pipeline - Design an S3 medallion (bronze/silver/gold) lake with partitioning and columnar file formats that keep scans small - Describe how a Glue ETL job transforms raw data into curated Parquet and how Athena queries it as SQL, with no server to manage - Estimate what a workload costs from its storage, bytes scanned, and compute, and identify the driver that moves the bill - Apply a strict teardown discipline so every exercise returns your account to a $0 resting state
Who this course is for
DE-203 is a Practitioner-level course in the School of Data Engineering. It is written for:
- The cohort learner already inside the migrated hitavirtech AWS codelabs, converting that progress into the full credentialed track.
- The backend engineer pivoting who is comfortable with Python, Git, and now PySpark, and needs the cloud-platform layer that turns a local job into a running pipeline.
- The analyst crossing over who has finished the modelling and PySpark courses and wants to see where those skills land on a real cloud.
Prerequisites
This course assumes you have finished DE-202 Batch Processing with PySpark, because the Glue ETL jobs here are PySpark jobs — the DataFrame API you already know, running on a managed AWS runtime instead of your laptop. It also assumes DE-201 Data Modelling and Warehousing: the bronze/silver/gold layers you build on S3 are the staging, core, and mart layers from that course, now expressed as directories and file formats.
You do not need prior AWS experience. What you do need is an honest respect for the bill — this course is explicit about cost at every step, and the habits it drills are the ones that keep a cloud data platform affordable.
Real AWS accounts charge real money. The lessons in this course teach the AWS services conceptually and the lab runs entirely on your own machine at no cost, so nothing here can bill you. When you later run these patterns against a live account in the Cloud school's hands-on courses, the teardown discipline this course drills is what stands between you and a surprise invoice. Treat "what does this cost, and how do I delete it?" as part of the definition of done.
Modules
DE-203 is roughly twenty-two hours of effort across eight modules. This slice delivers three core lessons and a hands-on lab covering the heart of the stack — the lake, its layout, and the ETL-and-query loop that turns raw objects into answers.
| # | Module | What you leave with |
|---|---|---|
| 1 | The AWS data engineering landscape | Which service plays which role, and how they compose |
| 2 | S3 for engineers | Layout conventions, lifecycle, and versioning |
| 3 | AWS Glue: catalog and ETL | Crawlers, the Data Catalog, and Glue ETL jobs |
| 4 | Amazon EMR and Spark on AWS | When managed clusters beat serverless |
| 5 | Redshift for engineers | Loading, distribution styles, and vacuuming |
| 6 | Lambda and event-driven ingestion | Triggering pipelines from object events |
| 7 | IAM for data engineers | Least-privilege pipeline roles |
| 8 | Cost engineering | Tagging, monitoring, and the bill-shock catalog |
The three lessons and the lab below cover the core of Modules 1, 2, and 3 and give you the mental model the rest of the course extends.
Outcomes
By the end of DE-203 you can:
- Map a data workload to the right AWS services and justify the choice on cost and operational grounds.
- Lay out an S3 lake in bronze, silver, and gold layers with partitioning and Parquet so queries stay cheap.
- Explain the Glue-catalog-then-Athena-query loop and how a Glue ETL job fits into it.
- Estimate a workload's monthly bill and name the single biggest cost driver.
- Return an account to a $0 resting state, because you know exactly what you created and how to delete it.
Where the real AWS labs live
The lab in this course is a faithful local model of the AWS medallion pattern: you build the whole bronze-to-gold flow in pure Python on your own machine, so you learn the shape of the architecture without touching a cloud account or spending a cent. Hands-on labs against live AWS — real S3 buckets, real Glue crawlers, real Athena queries under the free tier — live in the Cloud school's platform courses, which go deeper on provisioning, IAM, and multi-service operations than a data-engineering course needs to. Master the pattern here for $0; apply it on real infrastructure there.
Do the lessons in order. Each builds the next: Lesson 1 places the services, Lesson 2 lays out the lake they operate on, and Lesson 3 runs the ETL-and-query loop across it. The lab then makes you build that loop yourself.