Reading and writing data formats

⏱ 30 min

Hook

Your extract script works perfectly on the 5 MB sample and dies on the real file. The vendor sent 12 GB overnight, and json.load(f) tried to build the whole thing as one Python object in memory before your first line of logic ran. The machine has 8 GB of RAM. Nothing about your logic was wrong — you just read the file the wrong way.

Concept

Three text formats carry most of the data a pipeline ingests, and each has a standard-library reader. The difference that matters at scale is not the syntax — it is whether the format lets you process one record at a time.

CSV is a table as text, one row per line. Python's csv.DictReader reads it lazily, handing you one row dict per iteration, so a 12 GB CSV never has to fit in memory. JSON is a single nested document. json.load is all-or-nothing: it parses the entire structure before returning, which is fine for a small config file and fatal for a huge array. JSONL — JSON Lines — is the format that fixes this for records: one complete JSON object per line, separated by newlines. You read it line by line, parsing each line independently, so it streams like CSV but keeps JSON's nested types.

The tool that makes streaming possible in your own code is the generator: a function that uses yield to produce values one at a time instead of building a list and returning it. A generator holds only the current record in memory, so you can chain reading, transforming, and writing without ever materializing the whole dataset.

def read_lines(path):
    """A generator: yields one line at a time, holding one in memory."""
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

Contrast that with a function that does return [line for line in f]: the list-building version reads every line into memory before returning even the first. The generator version reads the first line, hands it to you, and does not touch the second until you ask. For pipeline files, prefer the generator by default.

Format	Reader	Streams record by record?	Keeps nested types?
CSV	csv.DictReader	Yes	No — all strings
JSON	json.load	No — loads the whole doc	Yes
JSONL	json.loads per line	Yes	Yes

The practitioner default across this school: write intermediate pipeline data as JSONL. It streams, it survives a crash mid-write (each line is independent), and every consumer can read it one record at a time.

🧠 Knowledge check

1. Why does JSONL stream record by record while a single large JSON array does not?

a) JSONL files are always smaller than JSON filesb) JSONL stores one complete JSON object per line, so you parse each line independently; a JSON array must be parsed as one whole documentc) JSON cannot represent nested objectsd) JSONL is a binary format and JSON is text

2. What does using yield instead of building and returning a list give a file reader?

a) It sorts the rows automaticallyb) It holds only the current record in memory instead of the whole filec) It makes the file smaller on diskd) It converts strings to floats for you

Worked example

Let me convert a stream of CSV events into JSONL, transforming each row on the way, without ever holding the whole dataset. I will use a small in-memory string so the example runs on its own, but the exact same code works against a multi-gigabyte file — that is the point of streaming.

import csv
import io
import json

RAW = """event_id,city,amount
e1,pune,120.00
e2,delhi,80.50
e3,pune,200.00
"""


def read_events(text):
    """Stream rows one at a time instead of loading all into memory."""
    yield from csv.DictReader(io.StringIO(text))


def to_jsonl(rows):
    """Transform each row and yield it as a JSON line — still lazy."""
    for row in rows:
        row["amount"] = float(row["amount"])
        yield json.dumps(row)


for line in to_jsonl(read_events(RAW)):
    print(line)

{"event_id": "e1", "city": "pune", "amount": 120.0}
{"event_id": "e2", "city": "delhi", "amount": 80.5}
{"event_id": "e3", "city": "pune", "amount": 200.0}

Trace the laziness, because it is the lesson. read_events yields one row; to_jsonl receives that one row, coerces amount from the string "120.00" to the float 120.0, and yields one JSON line; the for loop prints it. Only then does the next row get read. At no point does a list of all events exist. Swap io.StringIO(text) for open("events.csv") and this pipeline ingests a file far larger than memory without changing a line of logic. Note also that CSV gave me amount as a string — CSV has no types — so the transform's float() is doing real work the JSONL output then preserves as a number.

Hands-on

Your turn, with a different shape of data. The exercise below hands you a JSONL file of raw sale records and asks you to write a generator that reads it line by line, keeps only sales above a threshold, and writes the survivors back out as JSONL — all without loading the file into a list. It runs in Python Arena, so there is nothing to install.

▶ Python Arenade103-filter-jsonl-stream

Open in Python Arena ↗

Success criteria: your reader uses yield (not a list), your output is valid JSONL with one object per line, and only sales above the threshold survive. The Arena checks your output and confirms your reader streams rather than materializing the file.

Recap

• You can read CSV with csv.DictReader, whole JSON documents with json.load, and JSONL one line at a time with json.loads.
• You can explain why a large JSON array blows up memory while JSONL and CSV stream safely.
• You can write a generator with yield that processes one record at a time and chain readers, transforms, and writers without materializing the dataset.

Next up: what to do when a record in that stream is broken — the error handling and logging that keep a script alive and diagnosable when the input misbehaves.

🧠 Knowledge check

1. Which format is the recommended default for intermediate pipeline data in this school, and why?

a) A single large JSON array, because it is one tidy fileb) JSONL, because it streams record by record and each line survives a crash mid-write independentlyc) CSV, because it preserves nested typesd) Plain text, because it needs no parser

📌 Key takeaways

• CSV and JSONL stream record by record; a large JSON array must load whole. - Generators with yield process one record at a time, keeping memory flat. - Default to JSONL for intermediate pipeline data: streamable and crash-tolerant.