The data engineering lifecycle

⏱ 25 min

Hook

A teammate hands you a broken pipeline and says "the numbers are wrong." That sentence is useless until you can point at where in the pipeline they went wrong. Was the source data bad when it was generated? Did ingestion drop rows? Did a transformation double-count? Without a shared map of the stages data passes through, every debugging session starts from zero.

Concept

The data engineering lifecycle is that shared map. It breaks the journey of any dataset into five stages, each with one job. Data moves left to right, and problems are almost always traceable to a specific stage.

• Generation — data is created in a source system: a row is inserted, a click fires an event, a sensor emits a reading. You usually do not control this stage; you receive from it.
• Ingestion — data is pulled or pushed out of the source into your platform. This is where you decide how often (batch or streaming) and how much (full snapshot or only what changed).
• Storage — data lands somewhere durable: a data lake, a warehouse, an object store. Storage sits under every other stage, which is why it is drawn as a foundation, not just a step.
• Transformation — raw data is cleaned, joined, deduplicated, and reshaped into something consumers can use. Most of the "correctness" logic lives here.
• Serving — transformed data is delivered to its consumers: dashboards, ML models, exports, or other teams' pipelines.

Running underneath all five stages are the undercurrents — concerns that are not stages of their own but apply everywhere: data quality, security, orchestration (deciding what runs when), and cost. A dataset can be technically present at every stage and still be wrong because an undercurrent failed — for example, no one checked quality after transformation.

The lifecycle is a diagnostic tool, not a rigid recipe. When something breaks, you walk the stages: Did generation produce the row? Did ingestion carry it? Is it in storage? Did the transformation keep it? Did serving expose it? The first stage where the answer is "no" is where you start.

Stage	One-line job	A failure looks like
Generation	Produce the source record	Source never wrote the row
Ingestion	Move it into your platform	Ingest ran late or dropped rows
Storage	Keep it durably and queryable	Wrong partition, unreadable file
Transformation	Clean and reshape it	Bad join double-counts rows
Serving	Deliver it to consumers	Stale cache serves yesterday

🧠 Knowledge check

1. Which lifecycle stage is responsible for deduplicating and reshaping raw data?

a) Generationb) Ingestionc) Transformationd) Serving

2. Why is storage drawn as a foundation beneath the other stages rather than a single step?

a) Because it is the most expensive stageb) Because every other stage reads from or writes to durable storagec) Because it always comes lastd) Because only streaming pipelines use it

Worked example

Let me walk a real dataset through all five stages and mark the undercurrent checks along the way. The scenario: a ride-share app needs a daily count of completed trips per city.

1. Generation. The app writes a trips row when a ride ends, with a status and a city. You do not own this code; you consume its output.
2. Ingestion. Each morning you pull yesterday's trips. You choose an incremental pull keyed on ended_at, so you move only new rows.
3. Storage. You land the raw pull as-is before touching it, so you always have the untransformed record to re-run from.
4. Transformation. You keep only completed trips and count by city.
5. Serving. You expose the per-city counts to the dashboard.

The transformation and its quality-undercurrent check, in plain Python:

def transform_trips(trips):
    """Keep completed trips, count per city (the transformation stage)."""
    counts = {}
    for t in trips:
        if t["status"] != "completed":
            continue
        counts[t["city"]] = counts.get(t["city"], 0) + 1
    return counts

def quality_check(counts):
    """An undercurrent: fail loudly instead of serving nonsense."""
    if not counts:
        raise ValueError("No completed trips — refusing to serve an empty report")
    return counts

trips_raw = [
    {"city": "Pune", "status": "completed"},
    {"city": "Pune", "status": "completed"},
    {"city": "Pune", "status": "cancelled"},   # dropped by transform
    {"city": "Lagos", "status": "completed"},
]

report = quality_check(transform_trips(trips_raw))
print(report)

{'Pune': 2, 'Lagos': 1}

The cancelled trip is dropped in transformation, exactly where it should be — not in generation (you do not control the app) and not in serving (too late). And the quality_check is the undercurrent made visible: if a bad ingestion delivered zero rows, the pipeline refuses to serve an empty dashboard instead of silently showing zeros. That is the lifecycle working as a diagnostic: every concern has a home.

Hands-on

Your turn, with a different dataset and a different bug. The exercise below gives you a payments pipeline where refunded payments are leaking into a revenue total. Walk the lifecycle, locate the stage where the bug lives, and fix the transformation so refunds are excluded — then add a quality check that rejects a negative total.

▶ Python Arenade101-lifecycle-payments-fix

Open in Python Arena ↗

Success criteria: refunded rows are excluded in the transformation stage, the served total matches the expected value, and a negative total raises instead of serving. Python Arena grades your fix automatically.

Recap

• You can name the five lifecycle stages — generation, ingestion, storage, transformation, serving — and state the one job of each.
• You can explain the undercurrents (quality, security, orchestration, cost) as concerns that cut across every stage.
• You can use the lifecycle as a diagnostic, walking stage by stage to locate where a dataset went wrong.

Next up: batch versus streaming — a decision you make at the ingestion stage that ripples through the entire lifecycle.

🧠 Knowledge check

1. A dashboard shows a wrong total, but the source rows are correct and present in storage. Which stage should you suspect first?

a) Generationb) Storagec) Transformationd) None — the data is fine

📌 Key takeaways

• The lifecycle has five stages: generation, ingestion, storage, transformation, serving. - Undercurrents — quality, security, orchestration, cost — run across all stages. - Walk the stages left to right to pinpoint where a dataset broke.