Data structures for pipelines

⏱ 25 min

Hook

A colleague hands you a script that passes each record around as a plain dict. It works, until someone upstream renames customer_id to cust_id. Nothing crashes. row.get("customer_id") simply returns None, that None flows through three transformations, and a day later a report shows every customer as "unknown." The bug was invisible because the data structure never promised the field would be there.

Concept

Every pipeline moves records, and the structure you pick to represent a record decides how many of these silent failures you catch. Python gives you four workhorses, and the skill is matching the structure to the job.

A dict is the natural landing spot for raw input: a CSV reader or a JSON parser hands you dicts, and their keys map cleanly onto columns. The weakness is exactly the one in the hook — a dict makes no promise about which keys exist, so a typo or a renamed field degrades into a None instead of an error.

A set is the right structure whenever the question is "have I seen this before?" Membership tests and deduplication are what sets do in one line, and they do it in roughly constant time instead of scanning a list.

A tuple is a fixed, ordered, immutable group of values — useful for a composite key like (order_id, line_number) that you never mutate and often use as a dict key or set member.

A dataclass is the structure you convert raw dicts into once you have validated them. It names every field, so a missing or misspelled field fails where the record is built rather than three stages downstream, and each field can carry a type hint that documents what the value should be.

row = {"customer_id": "c-1", "country": "IN"}
# A typo reads as a missing key, not a crash, if you use .get():
print(row.get("county"))       # None — a silent bug waiting to happen
# A set answers membership and dedup in one structure:
seen = set()
for cid in ["c-1", "c-2", "c-1"]:
    if cid in seen:
        print("duplicate:", cid)
    seen.add(cid)

None
duplicate: c-1

The rule of thumb that holds across this school: use a dict at the boundary where raw data enters, then promote validated records to a dataclass for everything downstream. The boundary is where you accept mess; past it, your code should work with structures that fail loudly.

Structure	Use it for	Fails loudly on a bad field?
dict	Raw input at the boundary; flexible keys	No — missing key returns None
set	Membership tests and deduplication	N/A
tuple	Fixed composite keys, immutable value groups	N/A
dataclass	Validated records passed through the pipeline	Yes — missing field is an error

🧠 Knowledge check

1. Raw rows arrive from a CSV reader as dicts. Why promote each validated row to a dataclass before passing it downstream?

a) Dataclasses run faster than dicts on every operationb) A dataclass names its fields, so a missing or misspelled field fails where the record is built instead of silently downstreamc) Dataclasses automatically write themselves to diskd) Dicts cannot be used as function arguments

2. You need to detect whether an order_id has already been processed in this run. Which structure fits best?

a) A listb) A setc) A tupled) A dataclass

Worked example

Let me turn one raw reading into a validated record. Suppose an IoT feed emits temperature readings as dicts, with everything as strings and a status field that is "OK" only when the sensor self-reports healthy. I want a typed SensorReading I can trust downstream.

from dataclasses import dataclass


@dataclass
class SensorReading:
    sensor_id: str
    reading_c: float
    ok: bool


raw = {"sensor_id": "s-42", "reading_c": "21.7", "status": "OK"}
reading = SensorReading(
    sensor_id=raw["sensor_id"],
    reading_c=float(raw["reading_c"]),
    ok=raw["status"] == "OK",
)
print(reading)
print(reading.reading_c + 1)   # a real float, so arithmetic just works

SensorReading(sensor_id='s-42', reading_c=21.7, ok=True)
22.7

Notice what the conversion bought me. The raw reading_c was the string "21.7"; adding 1 to it would have raised a TypeError or, worse, quietly concatenated if I had been careless. By coercing to float at the boundary, I guarantee every downstream stage gets a number. And by reading raw["sensor_id"] with square brackets instead of .get(), a missing sensor_id raises a KeyError right here — the loud failure I want — instead of an invisible None. The dataclass is now a contract: anything holding a SensorReading knows it has a string id, a float temperature, and a boolean health flag.

Hands-on

Your turn, with a different feed. The exercise below hands you raw order dicts (not sensor readings) and asks you to define an Order dataclass and a function that converts a raw dict into a validated Order — coercing amount to a float and rejecting rows missing an order_id. It runs in Python Arena, so there is nothing to install.

▶ Python Arenade103-raw-dict-to-order

Open in Python Arena ↗

Success criteria: your converter returns an Order with a float amount for valid input and raises for a row with no order_id. The Arena validates your converter against a hidden set of good and bad rows automatically.

Recap

• You can match a structure to the job: dict at the input boundary, set for membership and dedup, tuple for fixed composite keys, dataclass for validated records.
• You can explain why a dict hides missing-field bugs and a dataclass surfaces them at construction.
• You can convert a raw dict into a typed, validated record that fails loudly on bad input.

Next up: reading and writing the files those raw dicts come from — CSV, JSON, and JSONL — without loading a giant file into memory all at once.

🧠 Knowledge check

1. What is the "boundary" rule of thumb for structures in a pipeline?

a) Use dataclasses everywhere, including for raw inputb) Accept raw data as dicts at the boundary, then promote validated records to dataclasses downstreamc) Never use dicts; they are unsafed) Convert every structure to a tuple before processing

📌 Key takeaways

• Pick the structure that matches the job: dict, set, tuple, or dataclass. - Dicts are right at the input boundary but hide missing-field bugs as None. - Promote validated rows to a dataclass so bad fields fail where the record is built.