ExamFinch.com My Study

Study guide

12.1 Streaming Architecture, Sources, and Sinks

Practice this exam

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

12.1 Streaming Architecture, Sources, and Sinks

Key Takeaways Explain This Topic AI

Structured Streaming unifies batch and real-time computation by modeling continuous streams as append-only unbounded tables.

Micro-batch is the default engine; continuous processing offers lower latency for a restricted operation/source/sink set with at-least-once guarantees. Actual latency depends on the workload.

Streaming file sources require an explicit schema or the configuration spark.sql.streaming.schemaInference=true, and batch ingestion volume is governed using maxFilesPerTrigger.

The Kafka source delivers messages with binary key and value columns; payloads are decoded by casting col("value").cast("string") and parsing with from_json().

foreachBatch() provides an escape hatch to pass micro-batch DataFrames to arbitrary batch writers, external JDBC storage, and Delta Lake MERGE operations.

Last updated: October 2026

Apache Spark's Structured Streaming engine revolutionized real-time data processing by integrating stream processing directly into the Spark SQL ecosystem. Rather than requiring developers to reason about low-level streaming abstractions (such as receivers, DStreams, and sliding RDD intervals), Structured Streaming introduces a declarative, high-level DataFrame API that treats real-time data ingestion as a continuous query running against an ever-growing, unbounded table .

For the Databricks Certified Associate Developer exam, you must master the fundamental mechanics of Structured Streaming: how queries are conceptualized against unbounded tables, how the underlying execution engines operate, how streaming data sources are configured and schema-enforced, and how processed records are dispatched to production storage sinks.

The Unbounded Table Model

The central architectural abstraction in Structured Streaming is the Unbounded Table . In this mental model, incoming data events are viewed as new rows continuously being appended to an input table with an infinite number of rows. =========================================================================

  • STRUCTURED STREAMING MODEL
  • =========================================================================

Incoming Stream: [Event 1] [Event 2] [Event 3] [Event 4] ...

  • |
  • v
  • +-----------------------------------------------------------------------+
  • | UNBOUNDED INPUT TABLE |
  • |-----------------------------------------------------------------------|
  • | Time | Device ID | Temperature | Status |
  • |------|-----------|-------------|--------------------------------------|
  • | 10:01| sensor-A | 68.4 | OK |
  • | 10:02| sensor-B | 72.1 | OK |
  • | 10:03| sensor-A | 85.0 | WARNING <-- Appended as new row |
  • +-----------------------------------------------------------------------+
  • |
  • [ Continuous Query Computation ]
  • |
  • v
  • +-----------------------------------------------------------------------+
  • | RESULT TABLE |
  • |-----------------------------------------------------------------------|
  • | Device ID | Max Temperature | Total Events |
  • |-----------|-----------------|-----------------------------------------|
  • | sensor-A | 85.0 | 2 |
  • | sensor-B | 72.1 | 1 |
  • +-----------------------------------------------------------------------+
  • |
  • v (Trigger Interval)
  • +-----------------------------------------------------------------------+
  • | OUTPUT SINK |
  • | (Delta Lake / Kafka / Cloud Files / JDBC via foreachBatch) |
  • +-----------------------------------------------------------------------+
  • Core Properties of the Model

Unified Batch and Streaming API: Developers write queries against streaming DataFrames using the identical DataFrame, Dataset, and SQL expressions used for static batch data. You can filter, project, aggregate, and join data using familiar operations like df.filter() , df.groupBy() , and df.select() .

Catalyst Optimizer Integration: Streaming queries are parsed, analyzed, optimized, and compiled into physical execution plans by Spark's Catalyst Optimizer and run through Project Tungsten code generation. The engine determines whether transformations can be pipelined or require stateful tracking.

Incremental Execution: At each trigger interval, Spark checks the streaming source for newly arrived data, computes the delta against the previous state, updates the Result Table, and writes the output to the target sink.

Processing Engines: Micro-Batch vs. Continuous Processing

Structured Streaming supports two distinct execution engines under the hood:

1. Micro-Batch Processing Engine (Default)

The micro-batch engine is the default, battle-tested execution model in Apache Spark:

Latency Profile: Can reach low latencies on suitable workloads, but trigger interval, source, state and sink costs determine actual end-to-end latency.

How It Operates: The coordinator periodically queries the source for available offsets, batches newly arrived records together, constructs a standard Spark physical execution DAG, and executes the micro-batch across distributed executor task slots.

Strengths: Maximum throughput, excellent resource utilization, robust dynamic autoscaling, and support for many relational and stateful operations, subject to streaming-specific restrictions on output modes, joins and chained stateful operations. Arbitrary batch SQL operations are not all supported on an unbounded stream.

2. Continuous Processing Engine (Low-Latency Specialized)

Introduced in Spark 2.3, the Continuous Processing engine can achieve latency as low as about 1 ms in suitable cases; this is not a guaranteed service level:

Latency Profile: Low-latency long-running processing; measure actual latency.

How It Operates: Rather than launching discrete tasks per micro-batch, the driver launches long-running tasks on executors that continuously poll the source and process records immediately. Epoch-based checkpointing (using an asynchronous Chandy-Lamport variant) coordinates snapshot commits at fixed intervals.

Critical Limitations (High Exam Probability):

Supports only map-like operations : selections ( select ), projections, filters ( where ), and simple mapping functions.

Does NOT support aggregations (no groupBy , count , sum ).

Does NOT support watermarking or windowing .

Does NOT support stream-stream joins or full outer joins.

Supports only a subset of sources (Kafka, rate) and sinks (Kafka, memory, console).

  • Feature Micro-Batch Engine (Default) Continuous Processing Engine
  • End-to-End Latency Workload-dependent Potentially around 1 ms; workload-dependent
  • Throughput Capacity Very High (bulk vectorization) Moderate to High

Supported Operations Supported streaming transformations, aggregations and joins Map-like operations only (select, filter)

  • Checkpointing Mode Synchronous WAL per micro-batch Asynchronous epoch markers
  • Trigger Syntax Default or .trigger(processingTime="5 seconds") .trigger(continuous="1 second")

Important

Exam Trap: Certification questions frequently ask which operations are supported under the Continuous Processing engine. Remember: no aggregations, no watermarks, and no stateful joins are permitted in continuous mode. If an analytical query requires running counts or windowed averages, you must use the micro-batch engine.

Streaming Sources ( spark.readStream )

Streaming queries begin by invoking spark.readStream , which returns a DataStreamReader instance.

1. File Sources (JSON, CSV, Parquet, ORC, Delta Lake)

Spark can continuously monitor a cloud object storage directory (e.g., S3, ADLS Gen2, DBFS) or distributed file system for newly written files. # Reading streaming JSON files with an explicit schema

from pyspark.sql.types import StructType, StructField, StringType, DoubleType, TimestampType

  • user_schema = StructType([
  • StructField("device_id", StringType(), True),
  • StructField("temperature", DoubleType(), True),
  • StructField("timestamp", TimestampType(), True)
  • ])
  • streaming_df = spark.readStream \
  • .format("json") \
  • .schema(user_schema) \
  • .option("maxFilesPerTrigger", 10) \
  • .load("/mnt/iot/incoming_json/")

Key File Source Behaviors:

Schema Enforcement: By default, Spark refuses to infer schemas on streaming file sources to avoid the massive performance overhead of scanning entire directories. You must supply a schema via .schema(user_schema) or explicitly enable schema inference with spark.conf.set("spark.sql.streaming.schemaInference", "true") .

maxFilesPerTrigger : Controls rate limiting by capping the maximum number of new files processed in a single micro-batch. This prevents driver and executor memory exhaustion when pointing a stream at a directory containing a large backlog of historical files.

Atomicity Requirement: Files placed in the monitored directory must appear atomically. If a file is written partially while Spark reads it, the stream will read corrupted data.

Delta Lake as Source: When reading from Delta Lake ( .format("delta") ), schema inference is supported automatically because Delta maintains a transaction log ( _delta_log ) that defines the exact table schema. Delta also supports tracking row-level modifications via the Change Data Feed (CDF).

2. Apache Kafka Source

Kafka is the industry-standard distributed streaming source. Spark connects to Kafka using .format("kafka") . kafka_stream_df = spark.readStream \

  • .format("kafka") \
  • .option("kafka.bootstrap.servers", "broker1:9092,broker2:9092") \
  • .option("subscribe", "telemetry-events") \
  • .option("startingOffsets", "latest") \
  • .option("failOnDataLoss", "false") \
  • .load()

Kafka Connection Options:

kafka.bootstrap.servers : Comma-separated list of host:port broker endpoints.

subscribe : Comma-separated list of Kafka topics. Alternatively, use subscribePattern for regex matching or assign for specific topic-partitions.

startingOffsets : Controls where reading begins upon the initial stream launch:

"latest" : Read only new messages arriving after query startup (default for streaming).

"earliest" : Read all available messages from the beginning of Kafka topic retention.

JSON string: Specify specific offsets per partition (e.g., """{"topicA":{"0":23,"1":-1}}""" ).

failOnDataLoss : When set to "true" (default), the query fails if offsets have expired out of Kafka retention before Spark consumed them. Setting this to false can allow processing to continue despite missing offsets, at the cost of possible data loss. Keep the default unless that tradeoff is explicitly acceptable; fix retention and lag rather than hiding missing input.

Schema of the Kafka Source DataFrame:

Spark presents Kafka messages with a fixed internal schema:

  • Column Name Data Type Description
  • key BinaryType The partition key of the Kafka message (raw bytes)
  • value BinaryType The message payload (raw bytes)
  • topic StringType The Kafka topic from which the record was read
  • partition IntegerType The partition ID within the topic
  • offset LongType The unique offset of the record in that partition
  • timestamp TimestampType Message timestamp (CreateTime or LogAppendTime)
  • timestampType IntegerType 0 for CreateTime; 1 for LogAppendTime

Tip

Deserializing Kafka Payloads: Because key and value are binary, you must cast them to strings before parsing structured formats such as JSON: from pyspark.sql.functions import col, from_json

  • parsed_df = kafka_stream_df \
  • .select(from_json(col("value").cast("string"), user_schema).alias("payload")) \
  • .select("payload.*")
  • 3. Testing and Synthetic Sources

Rate Source ( .format("rate") ): Generates synthetic data at a specified rate per second. Contains two columns: timestamp (TimestampType) and value (LongType starting at 0). Useful for performance profiling and benchmarking.

Options: rowsPerSecond (default 1), rampUpTime , numPartitions .

Socket Source ( .format("socket") ): Reads UTF-8 text lines from an open TCP socket connection ( host , port ). Strictly for testing; it is not fault-tolerant because TCP sockets cannot replay historical data upon failure.

Streaming Sinks ( df.writeStream )

A streaming query must write its results to an external sink using df.writeStream , which returns a DataStreamWriter . query = processed_df.writeStream \

  • .format("delta") \
  • .outputMode("append") \
  • .option("checkpointLocation", "/mnt/checkpoints/telemetry_gold") \
  • .start("/mnt/tables/telemetry_gold")
  • Summary of Supported Sinks

Delta Lake Sink ( .format("delta") ): The primary recommended storage sink on Databricks. Supports ACID transactions, schema enforcement, schema evolution ( mergeSchema ), unified batch and streaming reading/writing, and transactional streaming writes with checkpoint-based replay protection; custom foreachBatch writes need their own idempotency design.

Kafka Sink ( .format("kafka") ): Publishes streaming output back to a Kafka topic. Requires a value column (string or binary) and an optional key column. Target topic is specified via .option("topic", "output-topic") or as a topic column in the DataFrame.

File Sink ( .format("parquet") , .format("orc") , .format("json") , .format("csv") ): Writes files to storage directories. Only supports Append output mode ; does not support updates or overwrites. Requires a checkpoint directory.

Console Sink ( .format("console") ): Prints micro-batch output directly to the driver stdout/terminal. Options include numRows and truncate . Strictly for interactive debugging.

Memory Sink ( .format("memory") ): Stores output as an in-memory table on the Driver node registered under .queryName("my_table") . Allows querying with spark.sql("SELECT * FROM my_table") . Dangerous for large volumes because it can cause Driver OutOfMemory (OOM) exceptions.

Foreach and ForeachBatch Sinks: Custom output mechanisms for unsupported databases or complex write logic.

ForeachBatch vs. Foreach

foreachBatch(func) : Passes each micro-batch as a standard static DataFrame to a Python function with signature func(batch_df, batch_id) . This is the most versatile sink in modern Spark. It enables:

Writing to databases lacking native streaming sinks (e.g., writing to PostgreSQL or MySQL via .write.jdbc() ).

Fan-out writing to multiple destinations (e.g., saving to Delta and sending metrics to an API) from a single stream.

Executing Delta Lake MERGE INTO (upserts) per micro-batch. def upsert_to_delta(batch_df, batch_id):

  • # Re-using DeltaTable batch API inside foreachBatch
  • delta_table.alias("target").merge(
  • batch_df.alias("source"),
  • "target.device_id = source.device_id"
  • ).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()
  • query = streaming_df.writeStream \
  • .foreachBatch(upsert_to_delta) \
  • .option("checkpointLocation", "/mnt/checkpoints/upsert_stream") \
  • .start()

foreach(writer) : Row-level processing. Requires implementing an object with open(partition_id, epoch_id) , process(row) , and close(error) . Rarely used in PySpark due to Python-to-JVM serialization overhead; foreachBatch is strongly preferred.

Delivery guarantee depends on the sink

Micro-batch checkpointing plus a replayable source and a correctly implemented transactional/idempotent sink can provide end-to-end exactly-once effects. The Kafka sink is at-least-once and can duplicate records. foreachBatch is also at-least-once unless the callback uses batch IDs or stable business keys to suppress retries. Continuous processing has at-least-once guarantees. An append output mode describes which result rows are emitted, not whether the destination can receive duplicates. A MERGE callback must be deterministic and idempotent under replay; writing two systems in one callback is not one cross-system atomic commit.