Skip to main content

Windowing and watermarks

⏱ 32 min

Hook​

"Average temperature" over a stream that never ends is not a question you can answer — there is no last reading to divide by. Analysts do not actually want the average over all time; they want it per minute, or per five-minute sliding view, or per user session. To answer any of those you first have to cut the endless stream into finite chunks, and then decide the harder question: when is a chunk finished, given that a straggler could still show up?

Concept​

A window is a finite slice of an unbounded stream that you aggregate as a unit. Windows turn an unanswerable question ("the total, over everything") into an answerable one ("the total, per window"). Three window shapes cover almost every real need, and they differ in how an event maps to windows.

Tumbling windows are fixed-size and non-overlapping: [0, 10), [10, 20), and so on. Every event belongs to exactly one. Use them for regular periodic reports — counts per minute, revenue per hour.

Sliding windows are fixed-size but advance by a smaller step, so they overlap: with a 10-second size and a 5-second slide, second 12 falls in both [5, 15) and [10, 20). An event belongs to several windows. Use them for smoothed moving metrics — a rolling 10-minute error rate refreshed every minute.

Session windows have no fixed size at all. They grow to fit a burst of activity and close after a gap of inactivity longer than a chosen threshold. Use them for anything organized around bouts of user activity — a browsing session, a period of device chatter.

Windows carve the stream, but they do not solve the late-data problem on their own. That is the job of the watermark: a moving marker asserting "I believe I have now seen all events up to this event time." When the watermark passes a window's end, the system declares that window complete, emits its result, and drops its state. The watermark is a deliberate bet — you trade a little completeness for the ability to ever finish a window and free its memory.

A watermark is usually derived from the largest event time seen so far, minus an allowed lateness budget you choose:

watermark=max⁡(event times seen)−allowed lateness\text{watermark} = \max(\text{event times seen}) - \text{allowed lateness}

Set allowed lateness to 5 seconds and the watermark trails the newest event by 5 seconds, giving records that much grace to arrive before their window closes. The trade-off is stark and worth stating plainly: a larger allowed lateness catches more stragglers but delays every result and holds window state in memory longer; a smaller one emits sooner and cheaper but discards more late data. There is no correct value in the abstract — only the value your latency budget and your tolerance for dropped records justify.

One property matters for correctness: a watermark must never move backward. It advances only when a new maximum event time raises it, so a burst of older records cannot claw a closed window back open. You will see exactly this monotonic behavior in the worked example.

🧠 Knowledge check
1. With a 10-second window size and a 5-second slide, how many sliding windows contain an event at event time 12?
2. What does a watermark do when it passes a window’s end?

Worked example​

Let me build the three window shapes and a watermark as tiny, readable functions, using event times different from the lab dataset so nothing carries over by copy-paste. First, assigning a single event time to tumbling versus sliding windows:

WINDOW, SLIDE = 10, 5


def tumbling(t):
    start = (t // WINDOW) * WINDOW
    return [(start, start + WINDOW)]


def sliding(t):
    windows = []
    start = ((t - WINDOW) // SLIDE + 1) * SLIDE
    while start <= t:
        if start <= t < start + WINDOW:
            windows.append((start, start + WINDOW))
        start += SLIDE
    return windows


print("tumbling:", tumbling(12))
print("sliding: ", sliding(12))
tumbling: [(10, 20)]
sliding:  [(5, 15), (10, 20)]

Event time 12 lands in one tumbling window but two sliding ones — the overlapping views that let a sliding metric refresh without gaps. Next, sessionizing a series of click times with a five-second inactivity gap:

GAP = 5


def sessionize(times):
    times = sorted(times)
    sessions = []
    start = prev = times[0]
    for t in times[1:]:
        if t - prev > GAP:
            sessions.append((start, prev))
            start = t
        prev = t
    sessions.append((start, prev))
    return sessions


print("sessions:", sessionize([1, 3, 4, 20, 22, 40]))
sessions: [(1, 4), (20, 22), (40, 40)]

The clicks cluster into three bouts: the gap from 4 to 20 exceeds five seconds and cuts a session boundary, as does the gap from 22 to 40. The final click at 40 is a session of one. Session windows found their own sizes from the data.

Finally, the watermark itself. Watch it track the maximum event time minus a three-second lateness budget, and note that an older arrival never drags it back:

ALLOWED_LATENESS = 3
watermark = None
for t in [2, 5, 4, 11, 9]:
    candidate = t - ALLOWED_LATENESS
    if watermark is None or candidate > watermark:
        watermark = candidate
    print(f"saw {t:>2} -> watermark {watermark}")
saw  2 -> watermark -1
saw  5 -> watermark 2
saw  4 -> watermark 2
saw 11 -> watermark 8
saw  9 -> watermark 8

When event time 4 arrives after 5, the watermark holds at 2 rather than dropping to 1 — and when 9 arrives after 11, it holds at 8. That monotonicity is what guarantees a window the watermark already closed stays closed. Put these pieces together — assign events to windows, advance a watermark, close windows the watermark has passed — and you have the core of every event-time streaming engine. You will assemble exactly that in this module's lab.

Hands-on​

Your turn, on a fresh stream of API request events. The exercise below gives you a list of event times and asks you to assign them to tumbling windows, compute a watermark with a stated allowed-lateness budget, and report which windows the watermark has closed after each event — the precise bookkeeping the lab then turns into a running processor.

▶ Python Arenade205-windows-and-watermark-tracking
Open in Python Arena ↗

Success criteria: your window assignments match the reference, and your watermark value after each event is correct, including the cases where an older event must leave it unchanged. Python Arena grades both automatically.

Recap​

  • You can describe tumbling, sliding, and session windows and pick the shape that fits a given aggregation.
  • You can compute a watermark as the maximum event time seen minus an allowed lateness budget, and explain why it must never move backward.
  • You can state the watermark trade-off: more lateness catches more stragglers but costs latency and memory; less lateness is cheaper but drops more.

Next up: windows and watermarks decide when to emit, but a real processor also has to hold aggregation state between events and decide what happens to a record that shows up after its window is already gone. That is stateful stream processing, and the semantics it must promise.

🧠 Knowledge check
1. You raise a job’s allowed lateness from 5 seconds to 60 seconds. What is the main trade-off?
2. Which window shape best fits "group each user’s activity into bursts, ending a group after 30 minutes of inactivity"?
📌 Key takeaways
  • Windows carve an unbounded stream into finite, aggregatable slices; tumbling, sliding, and session windows cover most needs. - A watermark is the maximum event time seen minus an allowed-lateness budget, and it advances monotonically so closed windows stay closed. - Allowed lateness trades straggler coverage against latency and memory — a budget decision, not a universal constant.