Event time vs. processing time

⏱ 30 min

Hook

A mobile game logs a purchase the instant a player taps "buy." The phone is in a tunnel, so the event sits in an on-device buffer for ninety seconds, then flushes when signal returns. Your streaming job counts purchases per minute and reports a suspicious dip followed by a spike — a pattern that never happened. The purchases were evenly spaced. Your job grouped them by the moment they arrived, not the moment they occurred, and invented a story out of network weather.

Concept

Every record in a stream carries two timestamps, and telling them apart is the foundation everything else in this course rests on.

Event time is when the thing actually happened, stamped at the source: the tap, the sensor reading, the transaction commit. Processing time is when your system observes the record and gets to work on it. In a batch world you rarely notice the difference, because you process a bounded file long after everything in it happened. In a stream, the gap is live, variable, and the source of nearly every hard problem.

The mental model: event time is the timestamp printed on a letter; processing time is when it lands in your mailbox. Mail can be delayed, and two letters written minutes apart can arrive in the same delivery — or in the wrong order. Precisely: the delay between event time and processing time is unbounded and non-uniform. A record can arrive milliseconds after it happened or, like the tunneled purchase, a minute and a half later. Nothing in the system guarantees records arrive in event-time order.

This is why the choice of clock is not cosmetic. Grouping a stream into per-minute buckets by processing time answers the question "how many records did I receive this minute?" Grouping by event time answers "how many things happened that minute?" Only the second question is usually the one the business asked. The catch is that answering it means accepting you might not have all of a minute's events until well after that minute of wall-clock time has passed — the problem watermarks exist to manage, in the next lesson.

Property	Event time	Processing time
Stamped by	The source, when the event happened	Your engine, when it sees the event
Reflects	Reality	Network and system delays
Order guaranteed?	No — arrivals can be out of order	Yes — it is arrival order itself
Results reproducible?	Yes — same events, same buckets	No — depends on when data showed up

That last row is the quiet killer. A processing-time aggregation gives a different answer every time you replay the same data, because the delays differ on every run. An event-time aggregation is reproducible: the same events always fall in the same windows, no matter when they arrive. Reproducibility is why serious streaming work is event-time work.

🧠 Knowledge check

1. A sensor records a reading at 10:00:03 but, after a network stall, your job receives it at 10:00:47. What are its event time and processing time?

a) Event time 10:00:47, processing time 10:00:03b) Event time 10:00:03, processing time 10:00:47c) Both are 10:00:03d) Both are 10:00:47

2. Why does a per-minute count grouped by processing time give a different answer each time you replay the same stream?

a) Because event time is randomb) Because network and system delays differ on every run, so records land in different minutesc) Because processing time is more accurate than event timed) Because replaying data changes the events themselves

Worked example

Let me make the divergence concrete with four temperature readings from a field device. The device went offline briefly and buffered one reading, then flushed it late. Each record carries its event time and the processing time at which the job saw it, both as whole seconds past a shared start. I will bucket the same four records two ways — by processing time and by event time — using ten-second windows, and watch them disagree.

readings = [
    ("evt-1", 8, 8),
    ("evt-2", 9, 9),
    ("evt-3", 6, 40),   # captured at 6s, arrived at 40s after reconnect
    ("evt-4", 41, 41),
]

WINDOW = 10


def bucket(t):
    return (t // WINDOW) * WINDOW


by_processing, by_event = {}, {}
for name, etime, ptime in readings:
    by_processing.setdefault(bucket(ptime), []).append(name)
    by_event.setdefault(bucket(etime), []).append(name)

print("by processing time:", by_processing)
print("by event time:     ", by_event)

by processing time: {0: ['evt-1', 'evt-2'], 40: ['evt-3', 'evt-4']}
by event time:      {0: ['evt-1', 'evt-2', 'evt-3'], 40: ['evt-4']}

Look at evt-3. It happened at second 6 — squarely inside the first window, [0, 10) — alongside evt-1 and evt-2. But it did not arrive until second 40, so the processing-time view files it in the [40, 50) window with evt-4, a reading it has nothing to do with. The processing-time count for the first window is 2; the true, event-time count is 3. One delayed record, and the "how many readings in the first ten seconds?" answer is wrong by a third.

The event-time grouping is the correct one — but notice what it demands. To put evt-3 in the first window, the job had to still be willing to update that window at second 40, long after wall-clock time left it behind. A real system cannot keep every window open forever waiting for stragglers. Deciding how long to wait is exactly the watermark problem you take on in the next lesson.

Hands-on

Your turn, with a different dataset: a stream of ride-share trip-completion events, each carrying an event time and a processing time, several of them delayed. The exercise below asks you to compute the per-window counts under both clocks, identify every record that lands in a different window depending on which clock you use, and state which grouping the business question requires.

▶ Python Arenade205-event-vs-processing-time-buckets

Open in Python Arena ↗

Success criteria: your event-time and processing-time bucket counts both match the expected output, and you correctly flag the delayed records. Python Arena checks your bucket dictionaries against the reference automatically.

Recap

• You can define event time as when a record happened at the source and processing time as when your system observed it.
• You can explain why their gap is unbounded and variable, and why that makes processing-time aggregations non-reproducible.
• You can show, on a concrete stream, how one delayed record puts the two clocks into disagreement and which clock the business usually means.

Next up: if event-time windows have to stay open for late records but cannot stay open forever, something has to decide when a window is finally complete. That something is a watermark, and windows are what it closes.

🧠 Knowledge check

1. Which streaming goal is the main reason practitioners prefer event-time aggregation over processing-time aggregation?

a) It uses less memoryb) It produces reproducible results — the same events always fall in the same windowsc) It never has to deal with late datad) It avoids the need for timestamps

2. What does choosing event-time windowing force a streaming system to accept?

a) That results are computed instantly with no delayb) That it may need to keep a window open past its wall-clock end to admit late-but-valid recordsc) That records always arrive in orderd) That processing time can be ignored entirely

📌 Key takeaways

• Event time is when an event happened; processing time is when you saw it, and the gap between them is unbounded and variable. - Grouping by processing time is non-reproducible and answers "what did I receive," not "what happened." - Correct event-time windowing must tolerate late records, which sets up the need for watermarks.