ExamFinch.com My Study

Study guide

10.1 Broadcast Variables, Accumulators, and Closures

Practice this exam

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

10.1 Broadcast Variables, Accumulators, and Closures

Key Takeaways Explain This Topic AI

Changing a normal Python variable inside worker tasks does not update the driver variable.

SparkContext.broadcast creates a read-only shared value for tasks; functions.broadcast is a distinct SQL join hint.

Only the driver reads an accumulator value; task code adds updates rather than using it as shared mutable state.

Accumulator updates inside transformations can be repeated after recomputation and are unsuitable for business-critical exactly-once totals.

Last updated: October 2026

Why ordinary variables are not shared

A distributed task runs in an executor process, often on a different machine from the driver. Spark serializes the function and referenced closure values needed by that task. Mutating a Python list or counter inside a worker does not mutate the original object in the driver's Python interpreter. Local-mode experiments can hide assumptions about placement, so reason from process boundaries rather than from one observed laptop run.

Suppose count starts at zero on the driver and a function increments it for each row. Task copies can each increment their own value while the driver's count stays zero. Even if one could send updates through an external service, retries could repeat them. A shared mutable global is not a valid replacement for DataFrame.agg or a deliberately designed stateful streaming operator.

Read-only broadcast variables

A broadcast variable distributes a read-only value for reuse by tasks rather than repeatedly embedding a large lookup in task closures. In classic PySpark, create it through SparkContext.broadcast and read its value in worker code. sc = spark.sparkContext

rates = sc.broadcast({"standard": 1.0, "reduced": 0.5})

  • rows = sc.parallelize([("standard", 10.0), ("reduced", 10.0)], 2)
  • priced = rows.map(lambda row: row[1] * rates.value[row[0]])

# For this small example only:

print(sorted(priced.collect())) # [5.0, 10.0]

This example uses the classic RDD API to make the variable behavior visible. It is not supported through Spark Connect's SparkContext surface. A DataFrame built-in expression or a small relational lookup join may be a better production solution. A broadcast dictionary remains a local copy on each consuming process, so it must fit the relevant JVM/Python memory. Broadcasting a huge object is not a way to increase available memory.

Treat the broadcast contents as immutable. Modifying rates.value inside a worker does not consistently update the value on other workers or the driver and violates the intended contract. To change lookup policy, create a new broadcast and ensure tasks reference it deliberately. After use, unpersist() removes executor cached copies so they can be resent if needed; destroy() removes broadcast data and metadata and makes further use invalid. Do not destroy a value while tasks still need it.

Broadcast variable versus broadcast join hint

  • API Object returned/purpose Consumer
  • spark.sparkContext.broadcast(python_value) Broadcast wrapper with .value Classic task closure

pyspark.sql.functions.broadcast(dataframe) DataFrame with a broadcast strategy hint SQL optimizer and supported join plan

df.cache() Persisted distributed relation Later DataFrame computations

The join hint does not give the user a dictionary with .value. The shared-variable API does not automatically turn a large DataFrame join into a BroadcastHashJoin. A cache also differs: its rows remain partitioned, rather than every task receiving the whole relation. Choosing among these tools depends on whether the input is a Python lookup object, a relational join build side, or a reused distributed intermediate.

Accumulators collect task updates

An accumulator supports associative, commutative updates such as summation, with the driver reading the final value. In classic PySpark, a numeric example is sc.accumulator(0). Tasks can add; they should not read its current value as if it were synchronized global state. invalid_count = sc.accumulator(0)

def inspect_value(value):

if value < 0:

invalid_count.add(1)

  • sc.parallelize([3, -1, 4, -2], 2).foreach(inspect_value)
  • print(invalid_count.value) # 2 for this completed action example

foreach is an action, so it requests execution before the driver reads the value. Merely placing accumulator.add inside a lazy transformation is not enough to run it. A cached relation can also change whether upstream task code runs on a later action. The counter therefore reflects executed task updates under the API's semantics, not a universally stable description of the source data.

Retries, recomputation and reliable counts

Spark documents action-only accumulator updates as applied once per task despite task retries. Updates inside transformations can be applied more than once if stages or tasks are recomputed. A sequence of separate actions can intentionally run the function again. Consequently, do not use an accumulator as an authoritative payment count, inventory ledger, or streaming customer balance.

For a business count of invalid DataFrame rows, express the validation as columns and aggregate the resulting dataset: df.filter(F.col("amount") < 0).count(). For a grouped summary, use groupBy/agg and persist the intended result through a suitable destination. For state across streaming batches, use a stateful operator and checkpointed state. These choices make the computation part of the distributed data plan rather than an incidental side effect of task execution.

Common scenario checks

If a question asks why a driver list is unchanged, inspect closure copying. If it asks how to reuse a small read-only Python lookup, consider a broadcast variable and its memory cost. If it asks for a diagnostic counter from tasks, an accumulator may fit, with retry limitations. If it asks to send an email or charge a card once per record, none of these variables supplies exactly-once external side effects.

Confirm the API environment before answering: Spark Connect omits SparkContext, so these classic shared-variable snippets require classic Spark. See Spark shared variables and closures .