Streaming Fundamentals (DE-205)
A dashboard that was "real-time" quietly drifts a full minute behind reality, and nobody can say why. A windowed count that looked right in testing reports different totals every time a job restarts. A sensor that was offline for an hour reconnects, floods the pipeline with old readings, and every aggregate downstream is suddenly wrong. None of these are bugs in the ordinary sense. They are what happens when data never stops arriving and you reason about it as if it did.
DE-205 teaches streaming as a set of trade-offs rather than a tour of Kafka and Spark buttons. The central idea is that in an unbounded stream, time itself becomes ambiguous — an event can happen at one moment and reach you at a very different one — and almost every streaming concept exists to manage that gap. You will learn to separate event time from processing time, close windows correctly with watermarks, handle late and out-of-order events on purpose, and reason about at-least-once versus exactly-once delivery honestly. Throughout, the course is explicit about when a batch job is still the better engineering call.
- Distinguish event time from processing time and explain why the gap between them drives every hard streaming decision - Assign events to tumbling, sliding, and session windows and choose the right window shape for a problem - Use a watermark to decide when a window is complete and what to do with data that arrives after it - Reason about stateful aggregation and the operational cost of keeping state in a long-running job - Compare at-least-once and exactly-once delivery semantics and what each one costs to guarantee
Who this course is for
DE-205 is a Practitioner course in the School of Data Engineering. It is written for engineers who can already process bounded data at scale and now need to reason about data that never ends:
- The backend engineer who has finished the PySpark course and wants the streaming mental models before touching Structured Streaming in anger.
- The cohort learner progressing through the migrated PySpark and AWS material who keeps meeting the words "watermark" and "exactly-once" and wants them pinned down precisely.
- The analyst crossing over who understands batch pipelines and now fields "can we make this real-time?" requests and needs to answer them with a budget, not a guess.
Prerequisites
This course requires DE-202 Batch Processing with PySpark. Streaming is taught here partly by contrast with batch: you will lean on your understanding of bounded datasets, aggregation, and the cost of shuffles to see what changes when the dataset stops having an end. You do not need Kafka installed, a cluster, or any paid service. The three lessons run in your head and the browser; the lab runs on the Python standard library alone.
Modules
DE-205 is roughly eighteen hours of effort across seven modules. This slice of the course delivers the three lessons that form the conceptual spine — time, windows, and processing semantics — plus a hands-on lab that builds a working windowed aggregator with a watermark.
| # | Module | What you leave with |
|---|---|---|
| 1 | Streaming mental models | Events, logs, and what "unbounded" really changes |
| 2 | Kafka fundamentals | Topics, partitions, consumer groups, and offsets |
| 3 | Delivery semantics | At-least-once, exactly-once, and what they cost |
| 4 | Spark Structured Streaming | Micro-batches, triggers, and sinks |
| 5 | Time in streams | Event time, watermarks, and windowed aggregation |
| 6 | Stateful processing | Keeping state in a long-running job, and its weight |
| 7 | Batch vs. streaming decision | Latency budgets and an honest default |
The three lessons and the lab below cover the core of Modules 3, 5, and 6 — the reasoning that the Kafka and Spark modules then make concrete on real tooling.
Outcomes
By the end of DE-205 you can:
- Explain, with a concrete example, why event time and processing time diverge and why grouping by the wrong one silently corrupts results.
- Pick a window shape — tumbling, sliding, or session — for a stated aggregation need and justify the choice.
- Configure a watermark for an allowed-lateness budget and predict which late events it will accept and which it will drop.
- Describe stateful streaming aggregation and name the operational risks of unbounded state.
- Argue at-least-once versus exactly-once for a scenario in terms of cost and correctness, not slogans.
Where this leads: the capstone
Everything in this school converges on DE-350: Production-Grade Pipeline, where one required deliverable is a streaming ingestion leg for a high-velocity source — or a written ADR that justifies keeping that source on batch. The judgment you build here, deciding when seconds actually matter and how to keep a windowed aggregate correct under late data, is exactly what that deliverable grades. The watermarked aggregator you build in this course's lab is a stripped-down model of the same machinery Spark Structured Streaming runs for you in the capstone.
Do the lessons in order. Event time (Lesson 1) is the vocabulary windows and watermarks (Lesson 2) assume, and both feed the processing-semantics reasoning in Lesson 3. The lab then makes watermarks concrete by having you build one.