12.2 Output Modes, Triggers, and Checkpointing

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

12.2 Output Modes, Triggers, and Checkpointing

Key Takeaways Explain This Topic AI

The three output modes are Append (only new rows output), Update (only rows changed since last trigger output), and Complete (entire result table rewritten every trigger).

Complete mode strictly requires an aggregation and throws an AnalysisException if applied to queries without aggregations; it is incompatible with file sinks.

.trigger(availableNow=True) processes all unconsumed data across rate-limited micro-batches and terminates cleanly, superseding the legacy single-batch .trigger(once=True).

The checkpoint directory contains offsets, commits, state, and metadata directories; it is essential for end-to-end exactly-once fault tolerance and state recovery.

Checkpoint compatibility depends on the change: some filters/projections and rate-limit changes are allowed, while state schemas, grouping keys and source identity have strict restrictions.

Last updated: October 2026

When deploying a Structured Streaming query to production, configuring how results are emitted, when micro-batches execute, and where execution state is persisted is critical for application correctness and cost efficiency. Spark manages these behaviors through Output Modes , Triggers , and Checkpoint Locations .

Understanding the precise constraints of each output mode, the operational advantages of modern triggers like AvailableNow() , and the internal directory mechanics of checkpointing is essential for building resilient streaming pipelines and scoring highly on the certification exam.

The Three Output Modes

The output mode defines what data from the internal Result Table is written to the external sink at each trigger interval. Output mode is configured on the DataStreamWriter via .outputMode("append" | "update" | "complete") . +-------------------------------------------------------------------------+

| OUTPUT MODES SUMMARY |
+-------------------------------------------------------------------------+
| Mode | What Gets Written to Sink? | Aggregation Required? |
|----------|--------------------------------|-----------------------------|
| Append | Only brand-new rows added | No (Allowed only with |
| | since the last trigger | watermarking if aggregated) |
|----------|--------------------------------|-----------------------------|
| Update | Only rows updated or added | No (Behaves like Append |
| | since the last trigger | if no aggregation) |
|----------|--------------------------------|-----------------------------|
| Complete | The ENTIRE Result Table | YES (Throws exception |
| | rewritten on every trigger | if no aggregation) |
+-------------------------------------------------------------------------+
1. Append Mode ( .outputMode("append") )

In Append Mode , only the rows that were newly added to the Result Table since the previous micro-batch are output to the storage sink.

Semantics: Only newly appended result rows are emitted; actual delivery semantics depend on the sink. Once written, rows are never modified or updated.

Permitted Queries:

Simple transformational queries without aggregations (e.g., select , filter , map , flat map).

Aggregations only if an event-time watermark is defined. Spark holds back the output row until the watermark passes the window end time ( watermark >= window.end ), finalizing the retained result under the configured lateness policy; later data may still arrive and be discarded.

Prohibited Queries: Streaming aggregations without a watermark are not supported in Append mode. Because un-watermarked aggregations can be updated indefinitely by future data, Spark cannot guarantee that an aggregate row will never change, and therefore refuses to emit it in Append mode.

2. Update Mode ( .outputMode("update") )

In Update Mode , only the rows in the Result Table that were updated (or newly inserted) since the last trigger are emitted to the sink.

Semantics: If an existing group's count or sum changes during a micro-batch, the updated record is output to overwrite or update downstream systems. Unchanged rows in the Result Table are not emitted.

Permitted Queries:

Queries with aggregations (with or without watermarks).

Queries without aggregations: If a query has no aggregations, Update mode behaves identically to Append mode.

Common Sinks: Sinks that support key-based in-place updates or upserts, such as Delta Lake (via foreachBatch ), Kafka, MySQL/PostgreSQL via JDBC, or key-value NoSQL stores.

3. Complete Mode ( .outputMode("complete") )

In Complete Mode , the entire updated Result Table is rewritten to the external sink every time a trigger fires, regardless of how many rows actually changed.

Mandatory Requirement (Frequent Exam Trap): Complete mode strictly requires an aggregation ( groupBy() , window() , etc.). If you attempt to start a streaming query in Complete mode on a DataFrame without aggregations, Spark immediately throws an AnalysisException : AnalysisException: Complete output mode not supported when there are no streaming aggregations on streaming DataFrames/DataSets

Prohibited Sinks: Complete mode is incompatible with file sinks (Parquet, ORC, CSV, JSON). File storage cannot overwrite individual records or cleanly rewrite an entire directory per micro-batch without massive overhead. Complete mode is typically used with Delta Lake, console, or memory sinks.

State Retention: Unlike Append mode, Complete mode does not drop old aggregation state even if watermarks are defined, because the entire table must be output on every iteration.

Output Mode Compatibility Matrix

Query Transformation Type Append Mode Update Mode Complete Mode
Stateless (select, filter, map) Supported Supported (acts as Append) Not Supported (AnalysisException)
Aggregation without Watermark Not Supported (AnalysisException) Supported Supported
Aggregation with Watermark Supported (emitted after window close) Supported Supported (outputs full table)
Compatible with File Sinks Supported Not Supported Not Supported
Streaming Triggers

Triggers define the timing of streaming execution—dictating exactly when Spark checks sources for newly arrived data and executes micro-batch computations. Triggers are configured via .trigger(...) on the DataStreamWriter . # Configuring an AvailableNow trigger

query = df.writeStream \
.format("delta") \
.outputMode("append") \
.trigger(availableNow=True) \
.option("checkpointLocation", "/mnt/checkpoints/etl_stream") \
.start("/mnt/tables/etl_gold")
Available Trigger Types

Default Micro-Batch Trigger (Unspecified):

If no trigger is specified, Spark executes micro-batches in as-fast-as-possible mode. As soon as the previous micro-batch completes execution, the next micro-batch is scheduled immediately.

Fixed Interval Micro-Batch Trigger ( .trigger(processingTime="interval") ):

Executes micro-batches at discrete, user-defined clock intervals (e.g., .trigger(processingTime="10 seconds") or .trigger(processingTime="1 minute") ).

Catch-up Behavior: If a micro-batch takes longer to process than the specified interval (e.g., a batch takes 16 seconds on a 10-second trigger), Spark starts the next micro-batch immediately upon completion without waiting for the next clock boundary.

Once Trigger ( .trigger(once=True) - Legacy):

Ingests all available data from the source in a single micro-batch , commits offsets to the checkpoint directory, and terminates the streaming query cleanly.

AvailableNow was added in Spark 3.3; once=True is deprecated since Spark 3.4: If a substantial backlog of data has accumulated, .trigger(once=True) attempts to ingest all data in one giant batch regardless of maxFilesPerTrigger , frequently causing executor OutOfMemory (OOM) failures.

AvailableNow Trigger ( .trigger(availableNow=True) - Spark 3.3+ Standard):

Replaces .trigger(once=True) . Ingests and processes all outstanding data across multiple micro-batches , strictly honoring rate limits such as maxFilesPerTrigger or Kafka's maxOffsetsPerTrigger .

Once all backlogged data has been processed across sequential micro-batches, the query shuts down cleanly.

The Incremental Batch-as-Stream Pattern: This is the most cost-efficient pattern on Databricks. Data engineers can schedule a Databricks Job to spin up a compute cluster once per day or hour, execute .trigger(availableNow=True) , process the backlog incrementally using streaming logic, and terminate the cluster to save compute costs.

Continuous Trigger ( .trigger(continuous="1 second") ):

Activates the low-latency continuous processing engine with asynchronous epoch-based checkpoint commits.

Fault Tolerance and Checkpointing

Structured Streaming guarantees end-to-end exactly-once processing through the combination of:

Replayable Sources: Sources (like Apache Kafka, Delta Lake, or cloud object files) that can replay data from specified offsets.

Deterministic Engine with Checkpointing: Spark tracks read progress and state changes in a durable write-ahead log.

Idempotent / Transactional Sinks: Sinks (like Delta Lake) that can detect duplicate writes or execute atomic commits.

Checkpointing is configured by passing an external storage path to .option("checkpointLocation", "/path/to/checkpoint") . query = streaming_df.writeStream \

.format("delta") \
.option("checkpointLocation", "/mnt/lakehouse/checkpoints/device_stream") \
.start("/mnt/lakehouse/tables/device_events")
Internal Directory Structure of a Checkpoint

Inside the specified checkpoint location, Spark maintains four critical directories: /mnt/lakehouse/checkpoints/device_stream/

├── offsets/ <-- Write-Ahead Log of ingested source offsets
│ ├── 0
│ ├── 1
│ └── 2
├── commits/ <-- Completion markers for finished micro-batches
│ ├── 0
│ └── 1
├── state/ <-- Serialized state store files (RocksDB or HDFS)
│ └── 0/
│ └── 0/
└── metadata <-- Query UUID and configuration metadata

offsets/ (Write-Ahead Log): Before executing tasks in micro-batch N N N , the driver records the exact starting and ending source offsets (e.g., Kafka topic partition offsets or file paths) in a new file named N . This guarantees Spark knows exactly what data it intended to process.

commits/ (Completion Log): After all executor tasks in micro-batch N N N successfully complete and the sink confirms persistence, the driver writes a versioned commit-log entry named N (not necessarily an empty file). If a micro-batch fails mid-execution, offsets/N will exist, but commits/N will be missing.

state/ (State Store): Houses intermediate key-value state for stateful operations (aggregations, windowing, deduplication, stream-stream joins). Implemented either as HDFS-backed state store (default memory/disk) or RocksDB State Store (recommended in Databricks for large state sizes).

metadata : Stores the persistent query ID. query.id remains stable across checkpoint recovery; query.runId changes at each start/restart. State-store metadata has separate compatibility rules.

Failure Recovery Protocol

When a crashed Structured Streaming application restarts pointing to the same checkpoint directory:

Spark reads the offsets/ and commits/ directories.

If the highest batch in offsets/ (e.g., batch 2) does not have a corresponding file in commits/ , Spark knows batch 2 failed before completing.

Spark reads the exact offsets recorded in offsets/2 , re-executes micro-batch 2 from the replayable source, and writes the output to the sink. If the sink is idempotent (such as Delta Lake, which checks the streaming query ID and batch ID in its transaction log), duplicate writes are prevented.

Once batch 2 commits, Spark proceeds to batch 3.

Caution

Checkpoint compatibility: Keep source identity and state schemas compatible. Changing grouping keys, stateful dedup keys or state schema is generally unsupported on the same checkpoint. Some filter/projection changes and source rate-limit changes are allowed, subject to output-sink schema compatibility. For an incompatible change, plan a controlled migration with a new checkpoint and intentional replay/output policy; deleting a checkpoint can replay source data and create duplicates. Never share a checkpoint between concurrent queries.

Official reference

Structured Streaming guide — source checked October 16, 2026. The examples state their runtime assumptions.