Airflow's core model
Hook
You have read that Airflow "runs your DAGs," and you have seen a screenshot of a green grid. But when a task is stuck yellow at 03:00 and the on-call rotation is you, "it runs your DAGs" is not enough. You need to know which component decided to start that task, where it recorded the attempt, what it is waiting on, and how the next task will ever learn this one produced a filename. Airflow is not magic; it is five or six named parts with clear jobs. Learn the parts and the green grid stops being a mystery.
Concept
Apache Airflow is an open-source orchestrator that lets you define pipelines as Python code and then schedules, runs, and monitors them. Everything in this lesson is a name for one of its parts. You are not learning to configure Airflow yet — you are learning its vocabulary so the machinery is legible.
Start with what you write. A DAG in Airflow is a Python file that declares
tasks and their dependencies — the same directed acyclic graph from the previous
lesson, now expressed in code. A task is one node in that graph: a single
unit of work. You rarely write a task's guts from scratch; instead you configure
an operator, a reusable template for a kind of work. A PythonOperator runs
a Python function; a BashOperator runs a shell command; a
SparkSubmitOperator submits a Spark job. An operator is the class; a task is
one configured instance of it sitting in your graph.
That is the authoring side. The running side is a set of long-lived processes.
The scheduler is the heart. It continuously reads your DAG files, decides which task instances are due — every upstream succeeded, the interval has arrived — and hands them off to run. It is the real-world version of the "run a task only when its dependencies are in the succeeded set" loop you saw last lesson. The executor is the scheduler's arm: it decides where a ready task actually runs — in a local subprocess, on a pool of workers, or in a container. The metadata database is Airflow's memory: every DAG run, every task attempt, every state transition (queued, running, success, failed) is recorded there. When the UI shows a green square, it is reading the metadata DB. When the scheduler asks "did extract succeed for July 16?", it asks the metadata DB. Nothing about a run lives only in a worker's head.
Two more parts handle the cases a plain graph cannot express.
A sensor is a special task that waits for a condition before it succeeds: a file to land in a bucket, a partition to appear, an external job to finish. Instead of guessing that the upstream file "should be there by 2 a.m.," a sensor task polls until it genuinely is, then lets its downstream tasks proceed. It turns "wait for something outside this DAG" into a first-class node in the graph.
An XCom (short for cross-communication) is how one task passes a small piece of data to another — a filename, a row count, a computed date. Tasks run in separate processes and cannot share variables, so when the extract task needs to tell the load task which file it wrote, it pushes that string as an XCom and the load task pulls it. XComs are for small control values, not for moving your dataset; the data itself belongs in storage, and the XCom carries the pointer.
| Part | Its one job |
|---|---|
| DAG | Declares the tasks and their dependencies, in Python |
| Operator | Reusable template for a kind of work; a task configures one |
| Scheduler | Decides which task instances are due and dispatches them |
| Executor | Decides where a ready task runs (local, workers, containers) |
| Metadata DB | Records every run and state; the single source of truth |
| Sensor | A task that waits for an external condition before succeeding |
| XCom | Passes a small control value from one task to another |
Worked example
Let me read a small Airflow DAG the way the scheduler does, naming each part as it appears. This is a daily sales pipeline: wait for the raw file, transform it, load it. Read it as illustration — you are not running Airflow yet.
from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.sensors.filesystem import FileSensor
def transform(**context):
# Pull the path the upstream sensor confirmed, via XCom.
raw_path = context["ti"].xcom_pull(task_ids="wait_for_raw", key="path")
print(f"transforming {raw_path}")
return "warehouse://sales/2026-07-16" # returned value becomes an XCom
def load(**context):
target = context["ti"].xcom_pull(task_ids="transform")
print(f"loading into {target}")
with DAG(
dag_id="daily_sales",
schedule="@daily",
start_date=datetime(2026, 7, 1),
catchup=False,
) as dag:
wait_for_raw = FileSensor(
task_id="wait_for_raw",
filepath="/data/raw/sales_{{ ds }}.csv",
)
do_transform = PythonOperator(task_id="transform", python_callable=transform)
do_load = PythonOperator(task_id="load", python_callable=load)
wait_for_raw >> do_transform >> do_load
Walk it as the scheduler would. The DAG(...) block declares the graph and its
schedule — @daily, so one run per day. The last line, wait_for_raw >> do_transform >> do_load, sets the dependencies: >> means "then," so it
declares the same arrows you drew by hand last lesson. wait_for_raw is a
sensor: the scheduler will not consider transform due until that sensor
succeeds, which it does only once the file for that day's date ({{ ds }})
actually exists. When transform runs, it pulls an XCom to learn the path
the sensor found, does its work, and returns a value — that return becomes
an XCom automatically, which load then pulls to learn where to write. The
scheduler dispatched each task the moment its upstream reached success in
the metadata DB; the executor chose the process each one ran in. Every
part from the table is on the page.
Notice what you did not have to write: no clock arithmetic, no "sleep until the file shows up," no manual passing of the filename through a global. The model supplies all of it.
Hands-on
Your turn. The exercise below gives you a short DAG definition with the parts scrambled and mislabeled — a sensor called an operator, an XCom push with no matching pull, a dependency arrow pointing the wrong way — and asks you to identify each part correctly and fix the one broken data handoff. It runs in Python Arena, so there is nothing to install.
Success criteria: every part is labeled with the correct role, and the corrected XCom push/pull pair uses matching task ids and keys. Python Arena checks your answers against the reference automatically.
Recap
- You can name Airflow's core parts and state the one job of each: DAG, operator, scheduler, executor, metadata DB, sensor, and XCom.
- You can distinguish an operator (the template) from a task (one configured instance in the graph).
- You can explain the scheduler-plus-metadata-DB loop as the real-world version of "run a task once its dependencies have succeeded."
- You can describe a sensor as a waiting task and an XCom as a small control value passed between tasks, with the dataset itself living in storage.
Next up: what changes when this pipeline runs unattended in production — SLAs, failure handling, alerting, and reruns that do not corrupt data.
- You author DAGs, tasks, and operators; Airflow runs them through the scheduler, executor, and workers. - The metadata database is the single source of truth for every run and state. - A sensor is a task that waits for an external condition before it succeeds. - An XCom passes a small control value between tasks; the dataset itself stays in storage.