Skip to main content

What is data engineering

⏱ 25 min

Hook​

An analyst on your team pings you at 9 a.m.: the sales dashboard shows zero revenue for yesterday. Nothing crashed. No error email went out. The report just quietly served a wrong number to everyone who opened it before coffee. Someone has to figure out where, in the journey from a checkout button to that chart, the data went missing — and make sure it never happens silently again. That someone is a data engineer.

Concept​

Data engineering is the practice of building and operating the systems that move data from where it is produced to where it is consumed, reliably and repeatably. The key word is systems. A one-off script that copies a file is not data engineering; a pipeline that copies that file every night, notices when the file is late, refuses to load it twice, and tells you when its shape changes — that is.

The mental model to carry through this whole school is the producer-to- consumer chain. Data is born in producer systems (an app's database, a stream of clicks, a vendor's API), and it is needed by consumers (a dashboard, a machine learning model, a finance export). The data engineer owns everything in between.

Here is the shift that trips up people coming from analysis or backend work: in data engineering, the pipeline is the product, not the query you ran once to answer a question. A product has users (the consumers above), a contract (the data will be correct and on time), and a maintainer on the hook when it breaks. That framing changes how you build. You stop asking "does this produce the right number today?" and start asking "will this produce the right number every day, and how will I know the morning it doesn't?"

It helps to place the role against its neighbors:

RoleOwnsTypical question
Data engineerPipelines and data platformsIs the data correct, fresh, and reliable?
Data analystInsight from prepared dataWhat does the data say about the business?
Software engineerApplication featuresDoes the feature work for users?

These overlap, and titles blur between companies. The distinction that holds is ownership: the data engineer owns the movement and shape of data as an ongoing system.

🧠 Knowledge check
1. What most distinguishes data engineering from writing a one-off analysis query?
2. In the producer-to-consumer chain, which is a data producer?

Worked example​

Let me trace one dashboard number back through the chain so the "pipeline as a product" idea becomes concrete. Suppose a marketing lead asks, "How many customers signed up yesterday, by country?"

The data is produced by a signups table in the app's PostgreSQL database. The consumer is a daily chart. Between them, a small pipeline runs each morning. Even without a framework, I can express its steps as plain Python so the stages are visible:

from datetime import date, datetime

def extract_signups(rows, target_day):
    """Keep only rows created on the target day (the 'ingest' stage)."""
    return [r for r in rows if r["created_at"].date() == target_day]

def aggregate_by_country(rows):
    """Transform: count signups per country."""
    counts = {}
    for r in rows:
        counts[r["country"]] = counts.get(r["country"], 0) + 1
    return counts

# Sample of what the producer emitted (created_at is a timestamp):
signups_raw = [
    {"created_at": datetime(2026, 7, 16, 9, 5), "country": "IN"},
    {"created_at": datetime(2026, 7, 16, 11, 30), "country": "IN"},
    {"created_at": datetime(2026, 7, 16, 18, 2), "country": "BR"},
    {"created_at": datetime(2026, 7, 15, 23, 59), "country": "US"},  # prior day
]

yesterday = date(2026, 7, 16)
daily = extract_signups(signups_raw, yesterday)
report = aggregate_by_country(daily)
print(report)
{'IN': 2, 'BR': 1}

Notice what the pipeline already does that a throwaway query would not. It filters to a specific day, so a late-arriving July 15 row does not leak into July 16's count. It centralizes the "count by country" logic in one function, so every consumer sees the same definition of a signup. If tomorrow the producer renames country to country_code, exactly one function breaks and tells you where — instead of a silent zero on a dashboard. That is the difference between a script and a product.

Hands-on​

Your turn, with a different dataset. The exercise below loads a stream of raw orders records (not signups) and asks you to write the two-stage pipeline: filter to a target date, then aggregate revenue by product category. It runs in Python Arena, so there is nothing to install.

▶ Python Arenade101-orders-daily-revenue
Open in Python Arena ↗

Success criteria: your function returns one total per category for the target day only, and rows from other days are excluded. The Arena validates your output against a hidden test set automatically.

Recap​

  • You can define data engineering as building and operating data-moving systems, not writing one-off queries.
  • You can describe the producer-to-consumer chain and place the data engineer as the owner of everything between.
  • You can explain why treating the pipeline as a product changes how you build — reliability, a data contract, and a maintainer on the hook.

Next up: the data engineering lifecycle, which zooms into that middle box and names the five stages every piece of data passes through.

🧠 Knowledge check
1. Why does 'the pipeline is the product' change how you build it?
📌 Key takeaways
  • Data engineering builds and operates repeatable data-moving systems. - Data flows producer to consumer, and the data engineer owns the middle. - A pipeline is a product: it has users, a reliability contract, and an owner.