Batch Processing with PySpark (DE-202)
Your transformation script ran in 40 seconds against the sample extract and is still running four hours later against last month's full load. Nothing in the code changed — only the size of the data. Somewhere in that job, Spark is moving every row across the network, recomputing the same DataFrame three times, and waiting on one overloaded task while the rest of the cluster sits idle. You wrote all of that without meaning to, because the API hid it.
DE-202 is the PySpark course for the School of Data Engineering. It is not a tour of every function in the API; it is about the handful of ideas that decide whether your batch jobs are fast, correct, and cheap: how Spark actually executes the code you write, which operations move data and which leave it in place, and where the time and memory go when a job is slow. You will finish able to read a Spark job's execution plan, predict its shuffles before you run it, and write DataFrame pipelines that scale from a laptop sample to a production cluster without a rewrite.
- Explain how Spark runs a job — driver, executors, and the split from jobs into stages into tasks — and why nothing computes until an action fires
- Distinguish transformations from actions, and narrow transformations from wide ones, before you run a line of code
- Shape data with the core DataFrame operations: select, filter, withColumn, groupBy, and join
- Identify the shuffle in a physical plan and connect it back to the wide transformation that caused it
- Reduce shuffled data by filtering and projecting early, and cache a DataFrame that is reused instead of recomputing it
- Assemble ingest, clean, aggregate, and write into a runnable PySpark batch pipeline
Who this course is for
DE-202 is a Practitioner course. It assumes you can already write Python and have modelled the data you are about to process, and it spends its time on distributed processing itself. It is written for:
- The analyst crossing over who has outgrown single-node SQL and pandas and needs a processing engine that does not fall over on a large table.
- The backend engineer pivoting who is comfortable with Python and Git and now needs the distributed-systems intuition that makes Spark predictable instead of magical.
- The cohort learner already inside the migrated PySpark material, converting hands-on practice into the full credentialed track.
Prerequisites
This course builds directly on two others, and you should complete both first:
- DE-103 Python for Data Engineering — you will write PySpark in Python, so the typed-record, file-format, and robustness habits from DE-103 are assumed.
- DE-201 Data Modelling and Warehousing — you model data before you process it at scale. The aggregations and joins in this course target the kind of dimensional model DE-201 teaches you to design.
Every lesson's hands-on runs in Spark Forge, Tapasya's PySpark playground, so there is no local cluster to install. The lab runs on your own machine against a tiny dataset in Spark local mode, which needs only Python 3.10 or newer and a Java runtime — the setup section walks you through it.
Modules
DE-202 is roughly twenty-four hours of effort across eight modules. This slice delivers the three core execution-and-API lessons plus your first end-to-end pipeline lab — the foundation the rest of the course tunes and hardens.
| # | Module | What you leave with |
|---|---|---|
| 1 | Spark's execution model | Driver, executors, lazy evaluation, and the DAG |
| 2 | DataFrames and the API surface | select, filter, withColumn, and expressions |
| 3 | Aggregations, joins, and shuffles | groupBy, join, and the shuffle they trigger |
| 4 | Working with real files | Parquet, partitioning, and schema handling |
| 5 | UDFs and when not to use them | Built-ins over UDFs, and the performance reason |
| 6 | Performance tuning | Partitions, caching, broadcast joins, and skew |
| 7 | Structured batch jobs | spark-submit, configuration, and job structure |
| 8 | Debugging Spark | Reading the UI, OOMs, and the failure catalog |
The three lessons and the lab below cover the core of Modules 1, 2, and 3 and give you the batch-pipeline skill the rest of the course extends.
Outcomes
By the end of DE-202 you can:
- Trace a Spark job from the action that triggers it down to the tasks that run it, and say why an earlier transformation printed nothing.
- Predict whether any transformation stays local or forces a shuffle before you run it.
- Write a DataFrame pipeline that ingests raw data, cleans it, aggregates it, and writes partitioned output.
- Find the shuffle in a physical plan and cut the data it moves.
- Cache a reused DataFrame instead of paying to recompute it.
Where this leads: the capstone
Everything in this school converges on DE-350: Production-Grade Pipeline, where you build and operate a complete pipeline against a realistic source scenario seeded with data defects and late-arriving records. The PySpark transformations at the heart of that capstone — ingest, clean, deduplicate, aggregate, and write partitioned output — are exactly what this course's lab has you build. Orchestration (DE-204) and a data quality suite (DE-301) later wrap around PySpark code like the code you write here.
Do the lessons in order. Each one sets up the next: the execution model in Lesson 1 is what makes the transformation-versus-action distinction in Lesson 2 concrete, and both are what let Lesson 3 explain why a shuffle costs what it costs.