Orchestration with Airflow (DE-204)
You have three jobs. One extracts orders from a source database, one runs a PySpark transform, one loads the result into a warehouse. Each works when you run it by hand. Then you wire them into cron, walk away, and a week later the warehouse holds Tuesday's numbers twice and Wednesday's not at all — because the transform started before the extract finished, failed silently, and cron cheerfully fired the load anyway. Nobody was told. This course is about never being in that position again.
DE-204 is the course that turns jobs into systems. You will learn what an orchestrator actually buys you over a scheduler, how Apache Airflow models a pipeline as a directed graph of tasks, and what it takes to run that pipeline in production — where things fail, arrive late, and get rerun. The work is deliberately tool-honest: the concepts here outlast any one orchestrator, and you prove you understand them by building a miniature scheduler yourself, in plain Python, with no Airflow install required.
- Explain why orchestration exists and the specific failure modes of cron that a DAG-based orchestrator removes - Describe Airflow's core model — DAGs, tasks and operators, the scheduler, sensors, and XComs — and how they fit together - Design tasks that are idempotent, so a rerun or backfill produces the same result instead of duplicating data - Operate a pipeline in production using retries, SLAs, alerting, and safe idempotent reruns - Implement the heart of a scheduler yourself: dependency resolution via topological sort, ordered execution, and retry-on-failure
Who this course is for
DE-204 is a Practitioner course in the School of Data Engineering. It is written for the engineer who can already build the individual jobs and now needs to run them together, reliably, on a schedule someone depends on. If you have written PySpark transforms and loaded a warehouse but have only ever triggered them by hand or by cron, this is the course that closes the gap.
Prerequisites
This course assumes you have already built the batch jobs it orchestrates:
- DE-202 — Batch Processing with PySpark (course). You should be comfortable writing a structured batch job and reasoning about idempotent output, because idempotency is the backbone of safe scheduling.
- DE-203 — Data Engineering on AWS (course) is strongly recommended. Later course labs orchestrate AWS-hosted jobs, with a local-executor fallback for learners without a cloud account.
You need Python 3.10 or newer for the lab. You do not need to install Airflow, run Docker, or have a cloud account to complete the material on this page: the lab runs entirely on the Python standard library.
Modules
DE-204 is roughly twenty hours of effort across seven modules. This slice of the course delivers the three foundational lessons and your first lab — the conceptual spine the rest of the course builds on.
| # | Module | What you leave with |
|---|---|---|
| 1 | Why orchestration | Cron's failure modes and the DAG mental model |
| 2 | Airflow architecture | Scheduler, executor, metadata DB, and workers |
| 3 | Writing DAGs | Operators, dependencies, idempotent task design |
| 4 | Scheduling semantics | Data intervals, catchup, and backfills done right |
| 5 | Connections, variables, secrets | Handling credentials without hardcoding them |
| 6 | Sensors, branching, mapping | Waiting, choosing, and fanning out dynamically |
| 7 | Operating Airflow | Retries, SLAs, alerting, and the day-2 checklist |
The three lessons and the lab below cover the core of Modules 1, 3, and 7, and give you a working model of a scheduler that makes the rest of the course concrete.
Outcomes
By the end of DE-204 you can:
- Argue precisely why an orchestrator beats cron for anything with more than one step, in terms of dependencies, retries, visibility, and backfills.
- Read an Airflow DAG and name each moving part — task, operator, scheduler, sensor, XCom — and what it is responsible for.
- Write a task that is safe to rerun, and explain why idempotency is what makes a backfill trustworthy.
- Draw up a day-2 operations posture for a pipeline: what retries, what alerts, what SLA, and how a failed day gets repaired without double-loading.
Where this leads: the capstone
This school converges on DE-350: Production-Grade Pipeline, where you build and operate a complete pipeline against a realistic source scenario. The orchestration deliverable there — a scheduled DAG with correct scheduling semantics and a demonstrated three-day backfill — is exactly what this course prepares you for. The DAG runner you build in this course's lab is a stripped model of the scheduler that Airflow gives you for free, and understanding it is what lets you trust the real thing.
Do the lessons in order. The first builds the mental model, the second names the machinery, and the third puts it under production pressure. The lab then has you implement the one piece that ties them together — the scheduler's own core loop.