Skip to main content

Errors, logging, and robust scripts

⏱ 25 min

Hook​

The extract failed at 3 a.m. You open the terminal and find one line: Traceback ... ValueError. Which of the 40,000 rows broke it? How many made it through before the crash? Did it write a partial file a downstream job is now reading? The script did not say, because it was never told to. A pipeline that cannot explain its own failure costs you the two hours you spend reconstructing what it did.

Concept​

Robust pipeline code makes two decisions on purpose: which errors to catch, and what to record. Get both right and a 3 a.m. failure becomes a five-minute read instead of an investigation.

On catching: the temptation is a bare except: that swallows everything. It is the wrong move, because it hides the bugs you did not anticipate — a typo in your own code gets caught and ignored right alongside a malformed row. Catch the narrowest exception that names the failure you expect: ValueError for a number that will not parse, KeyError for a missing field. Anything you did not name propagates, which is what you want, because an unexpected error should stop the job loudly rather than be silently skipped.

On recording: print is not logging. The logging module gives you severity levels (WARNING for a skipped row, ERROR for a failure, INFO for a summary), timestamps, and one place to control where output goes. A logged message survives in a file long after the terminal scrolls away; a print vanishes.

The pattern that ties them together for row processing is skip and count: wrap the risky per-row work in a narrow try, log and count the rows you skip, and emit a summary at the end. One malformed row should not kill a 40,000-row job — but you must know it happened and how often.

import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
logger = logging.getLogger("orders")

logger.info("this is a summary line")
logger.warning("this row looked wrong but we continued")
INFO this is a summary line
WARNING this row looked wrong but we continued

The rule: catch narrow, log with a level, and always finish with a count. A job that processed 40,000 rows and skipped 3 should say exactly that, so the next person — often you — trusts the output or knows not to.

ChoiceFragile versionRobust version
Catching errorsbare except:except ValueError: (name the failure)
Recording eventsprint(...)logger.warning(...) with a level
Bad rowcrash the whole jobskip, log, and count it
End of runsilencean INFO summary of ok vs. skipped
🧠 Knowledge check
1. Why is catching a narrow exception like ValueError better than a bare except in row-processing code?
2. What does the logging module give you that a print statement does not?

Worked example​

Let me process a batch of order rows where some are broken, and make the script report exactly what it did. Two rows are bad: one has an unparseable amount, one is missing the amount key entirely. I want the good row processed, both bad rows skipped and logged, and a summary at the end.

import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
logger = logging.getLogger("orders")


def parse_amount(row):
    """Raises ValueError on a bad number, KeyError on a missing field."""
    return float(row["amount"])


def process(rows):
    ok, bad = 0, 0
    for row in rows:
        try:
            parse_amount(row)
        except (KeyError, ValueError) as exc:
            bad += 1
            logger.warning("skipping %s: %s", row.get("id"), exc)
            continue
        ok += 1
    logger.info("done: ok=%d bad=%d", ok, bad)
    return ok, bad


rows = [
    {"id": "a", "amount": "10.0"},
    {"id": "b", "amount": "oops"},
    {"id": "c"},
]
process(rows)
WARNING skipping b: could not convert string to float: 'oops'
WARNING skipping c: 'amount'
INFO done: ok=1 bad=2

Read what the log tells the next engineer. Row b failed because "oops" is not a number; row c failed because it had no amount key at all — and the two distinct messages came from one except (KeyError, ValueError) that named both expected failures. Crucially, the job did not stop: it processed the good row, counted the two bad ones, and ended with ok=1 bad=2. If a fourth row had raised something unexpected — say a TypeError from a bug in parse_amount — it would not be caught, and the job would stop loudly, which is exactly the behavior you want for an error you did not plan for.

Hands-on​

Your turn, with a different failure mode. The exercise below hands you rows where some have a malformed event_time, and asks you to write a processor that parses each timestamp, skips and logs the unparseable ones with a WARNING, and returns a (processed, skipped) count. It runs in Python Arena, so there is nothing to install.

▶ Python Arenade103-skip-and-count-timestamps
Open in Python Arena ↗

Success criteria: valid rows are processed, each bad row is skipped rather than crashing the run, and your returned counts match the good and bad totals. The Arena feeds your processor a mix of good and malformed rows and checks the counts.

Recap​

  • You can catch the narrowest exception that names an expected failure and let unexpected errors propagate loudly.
  • You can use the logging module's levels — WARNING for skips, INFO for summaries — instead of print statements that vanish.
  • You can apply the skip-and-count pattern so one bad row never kills the job and every run ends with a trustworthy summary.

Next up: the lab, where you combine all three lessons — typed records, streaming file formats, and this skip-and-count logging — into a runnable ingestion script.

🧠 Knowledge check
1. In the skip-and-count pattern, what should a 40,000-row job that hit 3 malformed rows do?
📌 Key takeaways
  • Catch the narrowest exception that names the failure; let the unexpected propagate. - Use logging levels, not print, so failures persist and carry severity. - Skip and count bad rows, and always end a run with a summary of ok vs. skipped.