Python for Data Engineering (DE-103)
A vendor sends you a 12 GB CSV overnight and your loader dies with
MemoryError before it writes a single row. A nightly extract quietly logs
nothing, so when it drops half its records nobody notices for a week. A colleague
hands you a 300-line script held together by bare dict access, and one renamed
key takes down three dashboards. None of this is a Spark problem or a cloud
problem. It is a Python problem — and this course teaches you to write the
Python that pipelines are actually made of.
DE-103 is the Python engineering course for the School of Data Engineering. It is not a general Python tutorial; it assumes you can already write a loop and a function, and it spends its time on the patterns that separate a throwaway script from pipeline code you can hand to on-call. You will choose the right data structure for a record, read and write CSV, JSON, and JSONL without loading a file into memory all at once, and instrument a script so that when it fails at 3 a.m. the log tells you exactly what broke and how much data it touched.
- Choose the right data structure — dict, set, tuple, or dataclass — for a pipeline record and defend the choice - Read and write CSV, JSON, and JSONL using only the Python standard library - Stream large files lazily with generators instead of loading them into memory - Handle bad rows with narrow exceptions that skip and count rather than crash blindly - Instrument a script with the logging module so failures are diagnosable after the fact - Assemble these pieces into a robust, runnable ingestion script
Who this course is for
DE-103 is one of the three Foundation courses in the school, and it sits beside DE-102 (SQL) as required preparation for every 200-level course. It is written for:
- The analyst crossing over who writes SQL daily, knows a little Python, and needs the file-handling and error-handling habits that pipeline work demands.
- The fresh graduate with coursework Python and no production experience, learning what "robust" means in code that has to run every night.
- The backend engineer comfortable with Python and Git who wants the data-specific patterns — streaming reads, row validation, structured logging — before moving on to PySpark in DE-202.
Prerequisites
You should be comfortable writing basic Python: variables, loops, functions, lists, and dictionaries. DE-101 (Data Engineering Foundations) is recommended first, because this course assumes you already know what a pipeline is and why late data, duplicates, and schema drift matter. You do not need a cloud account, a database, or any third-party package — every example and the lab run on Python 3.10 or newer using only the standard library.
Modules
DE-103 is roughly fourteen hours of effort across seven modules. This first slice delivers three core lessons and your first hands-on lab, covering the data structures, file formats, and robustness patterns the rest of the course builds on.
| # | Module | What you leave with |
|---|---|---|
| 1 | Python engineering hygiene | Virtualenvs, dependencies, and project layout |
| 2 | Files and serialization | CSV, JSON, JSONL, and Parquet trade-offs |
| 3 | Working with APIs and databases | requests, connectors, and retries |
| 4 | Data structures for pipelines | dataclasses, typing, and validation |
| 5 | Error handling and logging | Diagnosable failures instead of silent ones |
| 6 | Writing testable pipeline code | Pure functions and dependency injection |
| 7 | CLI entrypoints and configuration | From script to reusable tool |
The three lessons and the lab below cover the core of Modules 4, 2, and 5 and give you the ingestion skill the rest of the course extends.
Outcomes
By the end of DE-103 you can:
- Model a pipeline record as a typed structure and validate raw input against it.
- Move data between CSV, JSON, and JSONL without holding an entire file in memory.
- Write a script that survives bad input by skipping and counting, not crashing.
- Produce logs that make a 3 a.m. failure diagnosable instead of a mystery.
- Package these habits into an ingestion script you can reuse across the school.
Where this leads: the capstone
Everything in this school converges on DE-350: Production-Grade Pipeline, where you build and operate a complete pipeline against a realistic source scenario seeded with data defects and late-arriving records. The ingestion script you build in this course's lab — reading raw files, validating rows, writing clean output, and logging a summary — is the first stage of exactly that pipeline. The PySpark transformations (DE-202), orchestration (DE-204), and data quality suite (DE-301) layer on top of clean, validated data that code like this produces.
Do the lessons in order. Each one introduces a pattern the lab depends on: the lab's clean record is a typed structure from Lesson 1, its readers stream files using generators from Lesson 2, and its bad-row handling uses the logging pattern from Lesson 3.