ExamFinch.com My Study

Study guide

12.3 Event-Time Processing, Watermarking, and Windowing

Practice this exam

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

12.3 Event-Time Processing, Watermarking, and Windowing

Key Takeaways Explain This Topic AI

Event-time is the embedded timestamp indicating when an event physically occurred at the source, ensuring deterministic aggregation despite network delays or out-of-order delivery.

Tumbling windows create non-overlapping contiguous time intervals, whereas sliding windows produce overlapping intervals defined by window duration and slide interval.

Watermarks are computed as max(event_time) - delay_threshold; the watermark guides stateful operators; data beyond the lateness threshold may or may not be processed, and withWatermark alone is not a global filter.

Watermarking bounds state store growth by permitting Spark to evict expired window aggregates from executor memory once the watermark surpasses the window end time.

Stream-stream inner joins can run without watermarks but retain unbounded state; watermarks and a time constraint enable cleanup. Outer/semi variants have specific side-dependent requirements.

Last updated: October 2026

In real-world streaming architectures, data records rarely arrive at a compute cluster in clean, sequential order. Network partitions, intermittent mobile device connectivity, and varying queuing delays mean events frequently arrive out of order, delayed by minutes or even days. If a stream processing system evaluated records based solely on the server clock time when the record was received, results would be non-deterministic, varying with every cluster restart or network spike.

To solve this challenge, Structured Streaming bases stateful calculations on Event-Time rather than processing time, using Windowing to group data into discrete temporal buckets and Watermarks to drop excessively late records while bounding state store memory consumption.

The Three Notions of Time

Understanding the distinction between time domains is a foundational concept on the Databricks Associate Developer exam:

Event-Time: The timestamp embedded within the data record at the moment the event physically occurred at the originating source (e.g., an IoT sensor measuring temperature, a user clicking a link on a smartphone, or a point-of-sale terminal swiping a credit card).

Ingestion-Time: The timestamp assigned to a record when it is initially ingested into a centralized messaging broker (e.g., the Kafka log append timestamp).

Processing-Time: The local wall-clock timestamp of the Spark executor node at the exact moment the task thread processes the micro-batch record. +-------------------------------------------------------------------------+

  • | EVENT-TIME VS PROCESSING-TIME |
  • +-------------------------------------------------------------------------+
  • 10:00:00 AM 10:05:00 AM 10:14:00 AM
  • [Event Occurs] --------> [Broker Ingest] --------> [Spark Processes]
  • | | |
  • v v v
  • EVENT-TIME INGESTION-TIME PROCESSING-TIME
  • (Inside payload) (Kafka Broker) (Executor Clock)

Important

Why Event-Time Matters: If a mobile app records a user event at 10:00 AM while offline in an airplane, and uploads the record at 2:00 PM when reconnecting to Wi-Fi, an event-time query groups that event into the 10:00 AM analytical window where it factually happened. A processing-time query would erroneously attribute the event to 2:00 PM.

Windowing on Event-Time

Structured Streaming enables temporal aggregations using the pyspark.sql.functions.window() function. Windowing slices continuous streams into discrete time buckets against an event-time timestamp column.

1. Tumbling Windows (Non-Overlapping)

Tumbling windows are contiguous, fixed-duration time intervals that do not overlap. Every event belongs to exactly one window .

Syntax: F.window(timeColumn, windowDuration)

Example: A 10-minute tumbling window divides time into intervals: [10:00, 10:10) , [10:10, 10:20) , [10:20, 10:30) . from pyspark.sql import functions as F

  • # Tumbling 10-minute window aggregation
  • tumbling_counts_df = sensor_stream_df \
  • .groupBy(
  • F.window(F.col("event_time"), "10 minutes"),
  • F.col("device_type")
  • ) \
  • .count()
  • 2. Sliding Windows (Overlapping)

Sliding windows have a fixed duration but slide forward by a user-specified interval that is smaller than the window duration. Because the slide interval is smaller than the duration, windows overlap, and an individual event falls into multiple windows .

Syntax: F.window(timeColumn, windowDuration, slideDuration)

Example: A 10-minute window with a 5-minute slide interval produces windows: [10:00, 10:10) , [10:05, 10:15) , [10:10, 10:20) .

An event occurring at 10:07 falls into both [10:00, 10:10) and [10:05, 10:15) . # Sliding 10-minute window advancing every 5 minutes

  • sliding_counts_df = sensor_stream_df \
  • .groupBy(
  • F.window(F.col("event_time"), "10 minutes", "5 minutes"),
  • F.col("device_type")
  • ) \
  • .agg(F.avg("temperature").alias("avg_temp"))
  • Schema of the Resulting Window Column

When F.window(...) is invoked in a groupBy() , Spark produces a struct column named window containing two TimestampType fields:

window.start : The inclusive start timestamp of the window bucket.

  • window.end : The exclusive end timestamp of the window bucket. # Selecting window boundaries explicitly
  • formatted_df = sliding_counts_df.select(
  • F.col("window.start").alias("window_start"),
  • F.col("window.end").alias("window_end"),
  • F.col("device_type"),
  • F.col("avg_temp")
  • )
  • Handling Late Data with Watermarking
  • The State Store Memory Problem

In a streaming aggregation, Spark maintains intermediate running totals in an in-memory State Store across executors. Without a mechanism to finalize past windows, Spark would have to retain state for every historical time window dating back to the dawn of the application, because a late-arriving event might arrive tomorrow that belongs to last month's window. Over time, executor memory would inevitably be exhausted, crashing the stream with an OOM error.

What is a Watermark?

A Watermark is a moving temporal boundary that tracks how far behind the latest observed event-time Spark is willing to accept late data. It is defined using .withWatermark(timeColumn, delayThreshold) . Watermark = max ⁡ ( event_time seen so far ) − delayThreshold \text{Watermark} = \max(\text{event\_time seen so far}) - \text{delayThreshold} Watermark = max ( event_time seen so far ) − delayThreshold Latest observed event time: 12:45

  • Delay threshold: 15 minutes
  • Candidate watermark: 12:30

An event outside the delay threshold is not guaranteed inclusion.

For window aggregation, inspect the assigned window end and output mode.

How Watermarking Operates in Micro-Batches

Threshold Calculation: During micro-batch N N N , Spark scans all newly ingested records and finds the maximum event timestamp. It subtracts the delayThreshold from this maximum to determine the candidate watermark.

Enforcement in Subsequent Batches: The new watermark is established at the completion of micro-batch N N N and strictly applied during micro-batch N + 1 N+1 N + 1 .

Operator-specific late-data handling: A watermark is not an unconditional row filter. Window aggregation uses its window-based state/finalization rules; records beyond the threshold may or may not be included. The guarantee is one-sided: data within the delay threshold is not dropped as too late under the documented aggregation conditions.

State Store Cleanup: Once watermark >= window.end , Spark finalizes that window under the query's lateness policy. The engine removes the window's intermediate aggregate from the State Store, reclaiming executor memory. # Defining a 10-minute delay watermark on event_time

  • watermarked_stream = sensor_stream_df \
  • .withWatermark("event_time", "10 minutes") \
  • .groupBy(
  • F.window(F.col("event_time"), "1 hour"),
  • F.col("sensor_id")
  • ) \
  • .count()

Caution

Watermark Declaration Rules (Exam Focus):

withWatermark() must be called before the aggregation . Declare it on the input event-time attribute before grouping. A misplaced watermark may fail analysis or fail to enable cleanup; do not assume every such expression raises the same exception.

The timestamp column passed to withWatermark() must be identical to the timestamp column used inside the F.window() expression.

Output Modes and Watermarking Interaction

The output mode fundamentally alters when aggregated window results are delivered to the sink:

In Update Mode: On every micro-batch, Spark emits updated aggregate values for any window whose data changed during that batch. Once the watermark passes window.end , the window is purged from the State Store, but no final record is emitted because the downstream system already received the latest updates.

In Append Mode: Append emits finalized result rows that will not be revised by this query; physical exactly-once delivery depends on the sink. Therefore, in Append mode, Spark will not emit a window's aggregate row until the watermark passes the window end time ( watermark >= window.end ). Until that threshold is crossed, late data could arrive to alter the count, so emitting early would violate Append semantics.

Output Mode When is Window Aggregate Emitted to Sink? State Purged When?

Update Mode Emitted incrementally every micro-batch where the window's count changes When watermark >= window.end

  • Append Mode Emitted only once , after watermark >= window.end Simultaneously upon emission
  • Complete Mode The entire table is emitted on every micro-batch Never purged (watermark cannot drop state)
  • Streaming Joins

Structured Streaming supports joining streaming DataFrames with static DataFrames or with other streaming DataFrames.

1. Stream-Static Joins

Semantics: Joining a real-time streaming DataFrame with a static reference DataFrame (e.g., enriching an IoT telemetry stream with a static customer/device metadata table).

State Management: Stateless on the stream side. Spark joins incoming stream partitions directly against the static DataFrame.

Watermarking: Not required for stream-static inner joins.

Caveat: Do not assume the static side is a permanently frozen snapshot or a guaranteed live dimension. Source type, cache, file listing and transaction-snapshot behavior determine refresh; define the required consistency and test the chosen source/runtime.

2. Stream-Stream Joins

Semantics: Joining two independent streaming DataFrames (e.g., joining an ad_impressions stream with an ad_clicks stream to calculate click-through attribution).

The State Retention Challenge: Both streams are continuously arriving. When an impression arrives, its matching click might not arrive for several minutes. Spark must buffer both streams in its state store to find matches.

State-bounding pattern and outer-join requirements:

Watermarks on Both Streams: Watermarks on both inputs plus an appropriate time range can bound both state stores. An inner join can run without them but may retain unlimited history.

Time-Range Join Condition: To bound state, the join predicate should specify a temporal boundary linking the event timestamps of both streams (e.g., click.time >= impression.time AND click.time <= impression.time + INTERVAL 1 HOUR ).

Outer Join Constraints: In stream-stream left outer, right outer, or full outer joins, the required watermark side(s) and time bounds depend on which side can produce unmatched rows; consult the supported join matrix. These bounds are needed so Spark knows when a row will never receive a match and can safely emit a NULL -padded outer row.

Official reference

Watermark semantics — source checked October 16, 2026. The examples state their runtime assumptions.