Operating pipelines in production
Hook
The DAG is green in the demo. Then it goes live, and week two teaches you what the demo hid: the source API rate-limits you at midnight, a downstream table locks during the nightly vacuum, and one Sunday the whole run simply does not fire and nobody notices until Monday's standup. Getting a pipeline to run once is a coding problem. Keeping it trustworthy every day, and knowing the moment it stops being trustworthy, is an operations problem — and it is the one that decides whether people rely on your data or quietly stop.
Concept
Operating a pipeline in production rests on four practices. None is exotic; together they are the difference between a DAG that runs and a DAG you can sleep through.
Retries handle the transient. Most production failures are not bugs — they are a reset connection, a brief 503, a node that lost a race for memory. You configure each task with a retry count and a delay between attempts, so the orchestrator absorbs the blip without a human. The judgment is in the numbers: too few retries and transient noise pages you; too many, or too short a delay, and you hammer a struggling downstream system. A common starting posture is two or three retries with an exponential backoff — wait longer after each failure — and it is a starting point, not a law.
There is a hard rule attached: only retry idempotent work. Retrying a task that appends rows means each retry appends again. Before you raise a retry count, make sure a second run of the task overwrites its interval rather than adding to it — the idempotency discipline from the first lesson is what makes retries safe here.
SLAs make lateness visible. An SLA (service-level agreement) is the promise that a task or DAG finishes by a certain time — "the sales table is fresh by 06:00." You attach the expectation to the task, and the orchestrator records when a run misses it. The point is not punishment; it is that "the report is three hours late" becomes a signal the system emits on its own, rather than something finance discovers and reports to you.
Alerting decides who learns what, when. A pipeline that fails silently is worse than one that fails loudly, because silent failure erodes trust in every number downstream. Wire failure and SLA-miss events to a channel a human actually watches, and — this is the part teams skip — make the alert actionable: which DAG, which task, which data interval, and a link to the logs. "daily_sales failed" at 3 a.m. with no interval and no link is a page you will resent. Alert on the things that need a human (a task exhausted its retries, a run missed its SLA), and resist alerting on every transient retry, or people learn to ignore the channel.
Idempotent reruns are how you recover. When a day genuinely fails — bad source data, a bug you have now fixed — you rerun that interval. This is the backfill from lesson one, now under real pressure: you are reprocessing July 14 through 16 into tables that already hold partial, wrong data for those days. Only idempotent tasks make this safe. If loading an interval overwrites that interval's partition, a rerun repairs it cleanly; if loading appends, every rerun doubles the damage.
| Practice | The failure it addresses | The discipline it requires |
|---|---|---|
| Retries | Transient errors (resets, 503s) | Retry only idempotent tasks |
| SLAs | Silent lateness | State the freshness promise |
| Alerting | Failures nobody notices | Make alerts actionable, not noisy |
| Idempotent reruns | Recovering a bad day | Overwrite the interval, not append |
The thread through all four is idempotency. It is why retries are safe, why a missed SLA can be recovered by a rerun, and why an alert can say "just rerun the interval" instead of "stop everything and reconcile by hand."
Worked example
Let me show, in plain Python, why an idempotent load is the linchpin of every practice above — by rerunning the same interval and watching what happens. Here are two loaders writing a day's revenue into a tiny table keyed by date. One appends; one overwrites the interval.
def load_append(table, interval, value):
"""Not idempotent: every run adds another row for the interval."""
table.append({"interval": interval, "value": value})
return table
def load_idempotent(table, interval, value):
"""Idempotent: replace any existing row for this interval, then write."""
table = [row for row in table if row["interval"] != interval]
table.append({"interval": interval, "value": value})
return table
# Simulate a retry or backfill: run the SAME interval twice.
appended = []
appended = load_append(appended, "2026-07-16", 4200)
appended = load_append(appended, "2026-07-16", 4200) # rerun
replaced = []
replaced = load_idempotent(replaced, "2026-07-16", 4200)
replaced = load_idempotent(replaced, "2026-07-16", 4200) # rerun
print("append rerun ->", appended)
print("idempotent rerun ->", replaced)
append rerun -> [{'interval': '2026-07-16', 'value': 4200}, {'interval': '2026-07-16', 'value': 4200}]
idempotent rerun -> [{'interval': '2026-07-16', 'value': 4200}]
The append version now holds July 16 twice — and a downstream sum of revenue is silently doubled. This is not a hypothetical: it is exactly what a single retry or one backfill produces against a non-idempotent load. The idempotent version reruns to the same single row, because it deletes the interval before writing it. That "delete-then-write the interval" pattern is the whole game. Once your load has it, you can turn retries up, recover a missed SLA by rerunning, and let an alert say "rerun July 16" without anyone reconciling by hand. Without it, every safety mechanism in this lesson becomes a way to corrupt data faster.
Hands-on
Your turn, with a new operations scenario. The exercise below gives you a task that is failing in production and a short incident description, and asks you to choose the correct response — adjust retries, set an SLA, fix idempotency, or rerun the interval — and to convert a non-idempotent load into an idempotent one so the rerun is safe. It runs in Python Arena, so there is nothing to install.
Success criteria: your idempotent loader produces the same table whether it runs once or three times for the same interval, and your operations choice matches the failure type. Python Arena validates both automatically.
Recap
- You can name the four day-2 practices — retries, SLAs, alerting, idempotent reruns — and the specific failure each addresses.
- You can set a sane retry posture and explain why you retry only idempotent work.
- You can state what an SLA and an actionable alert are for: making lateness and failure into signals the system emits, not surprises your consumers report.
- You can convert a non-idempotent load into an idempotent one and explain why that single change is what makes every other safety mechanism trustworthy.
Next up, in the lab, you build the engine underneath all of this: a working DAG runner that resolves dependencies, executes in order, and retries failures up to a cap — the core scheduler loop, in plain Python.
- Retries absorb transient failures — but only on idempotent tasks. - SLAs turn silent lateness into a signal the orchestrator emits. - Alerts must be actionable (DAG, task, interval, logs) and not so noisy that people ignore them. - Idempotent reruns are how you recover a bad day; overwrite the interval, never append.