Spark's execution model
Hook
You add a .filter() to your PySpark job, run it, and it returns instantly — far
too fast to have read a hundred-million-row table. Convinced something is broken,
you add a .count() and now the same job takes six minutes. Nothing you added
should have made it slower. The filter was not fast because it was cheap; it was
fast because Spark did not run it yet.
Concept
Spark is a distributed engine: your program runs on one coordinating machine while the actual data processing happens across many. Understanding where each piece of your code executes is what turns Spark from magic into something you can reason about.
The coordinating machine runs the driver — the process that holds your
SparkSession, the variable spark, and the logic of your program. The driver
does not process your data. It plans the work and hands it out. The machines that
do process data run executors: worker processes that hold slices of your data
in memory and run computations on them in parallel. When people say a job "ran on
the cluster," they mean the driver split the work into pieces and the executors
ran those pieces.
The unit of data an executor works on is a partition — one chunk of a DataFrame's rows. A DataFrame is not one table sitting in one place; it is a logical view over many partitions scattered across executors. This is the whole source of Spark's speed: a hundred partitions can be processed by a hundred task slots at once.
Now the part that explains the hook. Spark operations come in two kinds:
- A transformation describes a change to a DataFrame —
select,filter,withColumn,groupBy. It does not run. It appends a node to a plan and returns a new DataFrame. - An action asks for a result —
count,collect,show,write. An action is what forces Spark to actually execute the plan built up by every transformation before it.
This is lazy evaluation: transformations build a recipe, and only an action
cooks it. Your .filter() returned instantly because it only recorded an
instruction. Your .count() was slow because it was the action that finally ran
the whole chain.
When an action fires, Spark compiles the accumulated transformations into a DAG — a directed acyclic graph of steps — and executes it as a hierarchy:
| Level | What it is | What creates a boundary |
|---|---|---|
| Job | All the work triggered by one action | Each action starts one job |
| Stage | A run of transformations with no data movement | A shuffle splits one stage from the next |
| Task | One stage's work on one partition | One task per partition per stage |
A job is one action's worth of work. Spark cuts that job into stages wherever the data has to be reshuffled across the cluster; within a stage, every step can run on a partition without looking at any other partition. Each stage then runs as a set of tasks — one task per partition — and those tasks are the things that actually execute on executors, in parallel.
One more distinction matters. Spark's older API is the RDD (Resilient
Distributed Dataset), a low-level collection of objects where you write the
processing logic yourself. The modern API is the DataFrame, a table with named,
typed columns. You should default to the DataFrame API, and not only for
readability: DataFrame operations pass through Catalyst, Spark's query
optimizer, which can reorder filters, prune columns, and choose join strategies
for you. Raw RDD code is opaque to Catalyst, so the engine cannot optimize it. In
this course, spark is always the session and the DataFrame API is always the
default.
Worked example
Let me make the laziness visible instead of just asserting it. I will build a small DataFrame of page-view events, chain two transformations, and watch when Spark does — and does not — do work.
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = SparkSession.builder.appName("execution-model").getOrCreate()
events = spark.createDataFrame(
[
("u-1", "home", 1200),
("u-2", "search", 300),
("u-1", "checkout", 5000),
("u-3", "home", 800),
],
["user_id", "page", "dwell_ms"],
)
# Two transformations. Neither runs here — each returns a new DataFrame.
slow_views = events.filter(F.col("dwell_ms") > 1000)
projected = slow_views.select("user_id", "page")
print(type(projected))
<class 'pyspark.sql.dataframe.DataFrame'>
Up to this point Spark has read no data and processed no rows. The filter and
select calls returned instantly because each one only added a node to a plan.
You can inspect that plan without running it, using the explain() action's
cousin — but the moment you ask for an actual result, Spark executes:
projected.show()
+-------+--------+
|user_id| page|
+-------+--------+
| u-1| home|
| u-1|checkout|
+-------+--------+
show() is the action. It triggered one job. Because this pipeline has no
groupBy or join — nothing that moves data between partitions — that job ran as a
single stage, with one task per partition of events. The filter and the select
were fused into that one stage and executed together in a single pass over the
data. If you had ended the pipeline with a groupBy before showing, Spark would
have cut the job into two stages at the point the data had to be regrouped across
the cluster. That boundary is the subject of Lesson 3.
Hands-on
Your turn, with a different dataset. The exercise below loads a stream of server request logs (not page views) and gives you a pipeline of transformations with no action at the end. Your job is to predict how many jobs run, add the action that triggers execution, and read the resulting stage count back from the plan.
Success criteria: you correctly state that the transformation-only version runs zero jobs, then add one action and confirm from the execution plan that a shuffle-free pipeline runs as a single stage. Spark Forge checks your answer and your plan automatically.
Recap
- You can name where each part of a Spark program runs: the driver plans, the executors process partitions, and tasks are one partition's work in a stage.
- You can explain why a transformation prints nothing and an action is what finally triggers the whole chain.
- You can describe how one action becomes a job, how a job splits into stages, and how a stage runs as one task per partition.
- You can say why the DataFrame API is the default: Catalyst optimizes it, and it cannot optimize raw RDD code.
Next up: the DataFrame API itself — the transformations you will actually write, and the line between the ones that keep data in place and the ones that move it.
- The driver plans work; executors process partitions; a task is one partition's work within a stage.
- Transformations are lazy and build a plan; an action is what triggers execution.
- One action becomes a job, a job splits into stages at shuffle boundaries, and a stage runs as one task per partition.
- Default to the DataFrame API so Catalyst can optimize your job.