ExamFinch.com My Study

Study guide

1.3 Streaming Workloads with Structured Streaming

Practice this exam

Databricks Certified Data Engineer Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Developing Code for Data Processing using Python and SQL - Core Spark SQL & PySpark

1.1 Complex PySpark Transformations & DataFrame API 1.2 Advanced Spark SQL Patterns 1.3 Streaming Workloads with Structured Streaming 1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

2 Chapter 2: Developing Code for Data Processing using Python and SQL - Streaming & Advanced Patterns

2.1 Advanced Delta Lake Features 2.2 Custom & Stateful Streaming Operations 2.3 Python & PySpark Code Modularization 2.4 User-Defined Functions (UDFs) & Performance Impact

3 Chapter 3: Data Ingestion & Acquisition

3.1 Auto Loader (cloudFiles) Production Ingestion 3.2 Change Data Capture (CDC) Ingestion Patterns 3.3 Batch & Streaming Connectors

4 Chapter 4: Data Transformation, Cleansing, and Quality

4.1 Data Validation & Quality Frameworks 4.2 Data Cleansing, Deduplication & Standardization 4.3 Silver and Gold Medallion Layer Refinement 4.4 Handling Schema Drift and Evolution

5 Chapter 5: Data Sharing and Federation

5.1 Delta Sharing Architecture & Governance 5.2 Lakehouse Federation

6 Chapter 6: Monitoring and Alerting & Observability

6.1 System Tables & Observability 6.2 Databricks Workflows & Job Monitoring 6.3 Spark UI & Query Execution Diagnostics 6.4 Data Quality & Pipeline Alerting

7 Chapter 7: Cost & Performance Optimisation - Data Layout & Compute

7.1 Data Storage & Layout Optimization 7.2 Compute Choice & Cluster Sizing 7.3 Photon Engine & Query Acceleration

8 Chapter 8: Cost & Performance Optimisation - Skew & Cost Governance

8.1 Partitioning & Data Skew Remediation 8.2 Cost Governance & DBU Optimization

9 Chapter 9: Ensuring Data Security and Compliance

9.1 Unity Catalog Access Control & Least Privilege 9.2 Dynamic Data Masking & Row/Column-Level Security 9.3 Secrets Management & Network Security 9.4 Compliance, Audit & Data Privacy

10 Chapter 10: Data Governance in Unity Catalog

10.1 Unity Catalog Governance Architecture 10.2 Data Lineage & Tagging in Unity Catalog 10.3 Managed vs External Tables & Storage Governance

11 Chapter 11: Debugging and Deploying - Asset Bundles & CI/CD

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code 11.2 CI/CD Automation & Databricks CLI / REST API 11.3 Troubleshooting Pipeline & Job Failures 11.4 Code Versioning & Databricks Git Integration

12 Chapter 12: Data Modelling & Lakehouse Analytics

12.1 Medallion Architecture Design Patterns 12.2 Dimensional Modeling in the Lakehouse 12.3 Analytics & BI Optimization

1.3 Streaming Workloads with Structured Streaming

Key Takeaways Explain This Topic AI

Structured Streaming uses micro-batch processing by default for high throughput, while Continuous Processing mode achieves sub-millisecond latency but with limited operations.

Trigger.AvailableNow() replaces Trigger.Once() to process all available data across multiple micro-batches before shutting down, ideal for cost-efficient batch-like streaming.

Watermarking (withWatermark()) manages state memory by dropping late data that arrives after the event-time watermark threshold has passed.

Append mode is the default and only outputs new rows, Update mode outputs rows that have changed, and Complete mode outputs the entire result table for aggregations.

Stateful stream processing (e.g., aggregations, dropDuplicates) requires a watermark in Append mode to prevent OutOfMemory errors over time.

Last updated: July 2026

1.3 Streaming Workloads with Structured Streaming

Structured Streaming is the core engine for building fault-tolerant, scalable, end-to-end streaming applications in Databricks. The Professional exam assesses your ability to configure triggers, manage state with watermarks, and select appropriate output modes for different sink constraints. This section provides an exhaustive overview of the internal mechanisms that make Structured Streaming robust for production data engineering.

Structured Streaming Mechanics

Under the hood, Structured Streaming treats a live data stream as a table that is being continuously appended. By default, it operates using a micro-batch processing engine. The engine keeps track of offsets, ensuring exactly-once processing guarantees even when the application crashes or restarts.

Micro-Batch vs Continuous Processing

In default micro-batch mode, Spark groups incoming data into small batches, processes them using the Catalyst Optimizer, and writes out the results. This provides exactly-once fault tolerance and high throughput, with latencies typically around 100 milliseconds.

For applications requiring sub-millisecond latency, Spark introduced Continuous Processing mode. Instead of batching, it launches long-running tasks that continuously read, process, and write data. However, Continuous Processing has strict limitations: it currently only supports map-like operations (select, project, map) and guarantees at-least-once (not exactly-once) semantics. It does not support complex aggregations or stateful processing, which limits its use cases in complex data engineering pipelines.

Trigger Configurations

Triggers define how frequently Spark executes the micro-batches. Proper trigger configuration is critical for balancing latency, cost, and throughput.

Trigger.ProcessingTime('X seconds') : Fires a micro-batch at regular intervals. This is the most common trigger for continuous stream processing.

Trigger.Once() : Processes a single micro-batch of all available data and then stops the query. This is useful for running daily batch jobs using streaming checkpointing logic, but can cause OutOfMemory errors if the backlog of data is too large.

Trigger.AvailableNow() : Introduced in Spark 3.3, this replaces Trigger.Once(). Instead of processing all data in a single massive batch, AvailableNow processes all available data across multiple discrete micro-batches until the backlog is cleared, and then shuts down the cluster. This is the recommended approach for cost-effective, incremental batch processing because it inherently avoids memory overloads while providing the same job-terminating behavior. # Example of using AvailableNow

  • df.writeStream \
  • .format("delta") \
  • .trigger(availableNow=True) \
  • .option("checkpointLocation", "/path/to/checkpoint") \
  • .start("/path/to/table")
  • State Management and Watermarking

When performing stateful operations like windowed aggregations or stream-stream joins, Spark must maintain intermediate state in memory (typically backed by RocksDB in Databricks). If the stream runs indefinitely, this state will grow unbounded, eventually crashing the application.

Understanding Watermarks

A watermark is a threshold defined by the withWatermark(event_time_column, 'delay_threshold') function. It tells Spark how late data is allowed to arrive relative to the maximum event time seen so far.

For example, if the max event time seen is 12:00 PM and the watermark delay is 10 minutes, the watermark threshold is 11:50 AM. If a record arrives with an event time of 11:49 AM, it is considered late and is dropped.

Crucially, the watermark allows Spark to safely clear old state from memory. Once the watermark surpasses a specific time window, Spark knows no more data will be accepted for that window, outputs the final aggregate result, and purges the state. from pyspark.sql.functions import window

  • # Watermarking example
  • streaming_df.withWatermark("event_time", "10 minutes") \
  • .groupBy(window("event_time", "5 minutes")) \
  • .count()
  • Output Modes

Output modes dictate what gets written to the external sink during each micro-batch. The choice of output mode is strictly constrained by the operations performed in the query and the capabilities of the sink.

Append Mode : The default mode. Only new rows appended to the result table are written to the sink. This is strictly required when the downstream sink does not support updates (like writing to raw Parquet files). If you are performing stateful aggregations, Append mode requires a watermark; Spark will only output the aggregation once the watermark passes the window, guaranteeing it will not change.

Update Mode : Only rows that were updated in the result table during the current micro-batch are written to the sink. This is commonly used for dashboards tracking rolling counts, writing to Delta tables that support upserts, or publishing to key-value stores.

Complete Mode : The entire updated result table is written to the sink after every micro-batch. This is only supported for queries with aggregations and is typically used when writing to small, easily overwritten tables or dashboards.

Stream-Stream Joins

Joining two streaming dataframes is one of the most complex operations in Structured Streaming. Because data arrives independently on both streams, Spark must buffer records as state to wait for matching records from the other stream.

To prevent state from growing infinitely, stream-stream joins strictly require watermarking on both streams and a time-range constraint in the join condition.

For example, joining an 'ad_clicks' stream to an 'ad_impressions' stream requires a condition stating that the click must occur within 1 hour of the impression. This bounds the state memory requirement, allowing Spark to discard old impressions that are older than the 1-hour window. # Stream-Stream Join Example

  • impressions_with_watermark = impressions.withWatermark("impression_time", "2 hours")
  • clicks_with_watermark = clicks.withWatermark("click_time", "3 hours")
  • # Join with time constraints
  • impressions_with_watermark.join(
  • clicks_with_watermark,
  • expr("""
  • click_ad_id = impression_ad_id AND
  • click_time >= impression_time AND
  • click_time <= impression_time + interval 1 hour
  • """)
  • )

Without these constraints, the job will fail with an AnalysisException, enforcing safe operational boundaries for your streaming architecture.