Study guide
2.2 Memory Management, Fault Tolerance, and Garbage Collection
Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Spark Architecture and Execution
1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage
2 Chapter 2: Shuffles, Memory and Persistence
2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels
3 Chapter 3: Spark SQL Sessions, Tables and Views
3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views
4 Chapter 4: SQL Sources and Stored Layout
4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout
5 Chapter 5: SQL Queries and Analytics
5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics
6 Chapter 6: DataFrame Construction and Columns
6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions
7 Chapter 7: DataFrame Validation and Inspection
7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion
8 Chapter 8: Aggregations and Dates
8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions
9 Chapter 9: DataFrame Joins and File IO
9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning
10 Chapter 10: Shared Variables and Stateful Processing
10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores
11 Chapter 11: Troubleshooting and Tuning
11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins
12 Chapter 12: Structured Streaming
12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks
13 Chapter 13: Spark Connect and Deployment
13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices
14 Chapter 14: Pandas API on Spark
14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization
2.2 Memory Management, Fault Tolerance, and Garbage Collection
Key Takeaways Explain This Topic AI
Spark's Unified Memory Manager divides the JVM heap into Reserved Memory (300 MiB fixed), User Memory (default 40% of heap minus reserved), and Spark Memory (default 60%).
Spark Memory dynamically shares space between Execution Memory (shuffles, sorts, joins) and Storage Memory (cached DataFrames and broadcast variables); Execution can evict Storage, but Storage can never evict Execution.
Off-heap memory leverages Project Tungsten's compact binary format to manage memory directly via sun.misc.Unsafe , bypassing JVM object headers and reducing garbage collection pressure.
Spark achieves fault tolerance through deterministic RDD lineage graph recomputation, persisted shuffle files, and automatic task retries (default 4 attempts governed by spark.task.maxFailures ).
Compact DataFrame representations can reduce GC pressure; choose JVM options from the supported runtime and measured GC behavior rather than a universal collector recipe.
Last updated: October 2026
Efficient distributed data processing requires careful memory accounting. Prior to Spark 1.6, Spark used a static memory manager where execution and storage pools were rigidly separated by fixed boundaries, frequently causing out-of-memory errors when one pool starved while the other sat empty. Modern Apache Spark uses the Unified Memory Manager , which dynamically balances memory across execution and caching workloads alongside Spark's task recovery mechanisms.
1. The Unified Memory Architecture: JVM Heap Breakdown
When an executor starts, it receives a total memory allocation governed by spark.executor.memory . This JVM heap space is partitioned into four major regions: ┌────────────────────────────────────────────────────────────────────────┐
- │ Total Executor JVM Heap Space │
- ├──────────────┬───────────────────┬─────────────────────────────────────┤
- │ Reserved │ User Memory │ Spark Memory (spark.memory.fraction)│
- │ Memory │ (1 - fraction) ├──────────────────┬──────────────────┤
- │ (300 MiB) │ (~40% of Usable) │ Storage Memory │ Execution Memory │
- │ │ │ (~50% pool) │ (~50% pool) │
- └──────────────┴───────────────────┴──────────────────┴──────────────────┘
- Region 1: Reserved Memory (Fixed 300 MiB)
Size : Exactly 300 MiB , hardcoded in Spark ( UnifiedMemoryManager.scala ).
Purpose : An accounting safety margin excluded from unified execution/storage allocation in upstream Spark 3.5.7; it is not an isolated pool that prevents OOM.
Usable Heap : Subtracted before any other pool is calculated: UsableMemory = ExecutorMemory - 300 MiB .
Accounting example: For an 8 GiB heap, usable heap for this calculation is 8192 - 300 = 7892 MiB; at fraction 0.6, the unified pool is about 4735 MiB. This is not the entire container allocation: memory overhead, Python and native allocations must also fit.
Region 2: User Memory
Size : Governed by 1.0 - spark.memory.fraction . With the default fraction of 0.6 , User Memory receives 40% of Usable Memory .
Purpose : Stores non-Spark data structures, user code objects, internal metadata, custom RDD transformations, and user-defined functions (UDFs).
Characteristics : Spark does not monitor or evict objects in User Memory; if user code allocates massive hash tables or large collections, it can trigger JVM heap exhaustion.
Region 3: Spark Memory ( spark.memory.fraction = 0.6 )
Spark Memory receives the remaining 60% of Usable Memory . This pool is dedicated to distributed data processing and caching, split dynamically between two sub-pools:
Execution Memory :
Used for computational buffers required by shuffles, joins, sorts, and hash aggregations.
Short-lived working memory allocated during task execution.
If an operation requires more Execution Memory than is available, records are sorted and spilled to local executor disk.
Storage Memory :
Used for cached DataFrames and RDDs ( df.cache() , df.persist() ).
Stores broadcast variables distributed to executors ( sparkContext.broadcast(...) ).
Stores unrolled data blocks during serialization.
Governed initially by spark.memory.storageFraction = 0.5 (50% of the Spark Memory pool).
2. Dynamic Borrowing and Eviction Policy
The defining feature of the Unified Memory Manager is the dynamic boundary between Execution Memory and Storage Memory. Rather than maintaining hard walls, both pools borrow space from each other under the following rules:
Asymmetric Borrowing Rules
Borrowing Without Contention :
If no cached data exists in Storage Memory, Execution Memory can expand to occupy 100% of Spark Memory .
If no active shuffles or sorts require Execution Memory, Storage Memory can expand to occupy 100% of Spark Memory for cached DataFrames.
Contention (The Asymmetric Eviction Rule) :
Execution Evicts Storage : If Execution Memory requires additional space and Storage Memory is currently borrowing execution capacity, Execution will forcefully evict Storage blocks from memory. Evicted blocks are dropped to disk (if cached with MEMORY_AND_DISK ) or invalidated from memory (if cached with MEMORY_ONLY ).
Storage Cannot Evict Execution : If Storage Memory requires space to cache a new DataFrame, but Execution Memory is currently occupying the space, Storage CANNOT evict Execution blocks . The caching operation either drops records to disk or skips caching them.
Why Is Eviction Asymmetric?
Execution blocks hold active, mid-flight computational state (such as partial aggregations or sort buffers). Evicting an active execution block would cause running tasks to fail or force expensive intermediate disk-thrashing. In contrast, cached storage blocks are immutable and can be recomputed when their required source data and dependencies remain available.
3. Off-Heap Memory and Project Tungsten
Traditional JVM execution stores data as Java objects on the JVM heap. A Java object layout adds representation-dependent overhead:
Java object overhead depends on JVM layout, compressed references, alignment, and data type; many small objects can use much more memory than their payload.
Garbage collection must traverse millions of these tiny object pointers, resulting in severe "Stop-the-World" GC pauses.
Project Tungsten Binary Format
Project Tungsten re-architected Spark's execution engine by treating memory like raw C-style memory buffers:
Binary Row Format : Rows are stored as compact, contiguous binary byte arrays rather than Java objects. Field access uses 64-bit word offsets, eliminating object headers and pointer dereferencing.
Cache-Aware Algorithms : Because rows are packed densely in contiguous memory, CPU cache lines (L1, L2, L3) are saturated with real data rather than pointer references, speeding up memory bandwidth.
Whole-Stage Code Generation : Synthesizes bytecode for an entire stage into a single Java function loop compiled on-the-fly via the Janino compiler, keeping active data in CPU registers.
Off-Heap Memory Configuration
Spark can allocate execution and storage memory outside the JVM heap using native memory allocators via sun.misc.Unsafe : spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=10g
When off-heap memory is enabled, Tungsten writes binary records directly to operating system memory. This memory is completely invisible to the JVM Garbage Collector , reducing GC tracking for those buffers; heap objects, metadata, and Python workers still consume memory and can pause or fail.
4. Fault Tolerance Architecture in Apache Spark
Distributed systems face frequent node failures, hardware stalls, and network interruptions. Spark ensures resilience through four complementary layers of fault tolerance:
1. RDD Lineage Recomputation
DataFrames and RDDs are immutable representations of data.
Each DataFrame retains a lineage graph (metadata recording the exact sequence of transformations and input file sources that created it).
If a worker node crashes and loses an in-memory partition, the DAG Scheduler inspects the lineage graph and re-executes only the missing partition on another available worker node, without restarting the job.
2. Shuffle File Persistence
In wide transformations, map tasks write their shuffle output files to local executor disk.
If a reducer task fails in Stage 2, it simply re-fetches the persisted shuffle files from Stage 1 executors rather than recomputing Stage 1.
If an executor hosting Stage 1 shuffle files dies, only the specific map tasks from that failed executor are rescheduled.
3. Task Retries ( spark.task.maxFailures )
When a task throws an exception (such as a transient network timeout or connection reset), Spark does not immediately abort the job.
The scheduler re-attempts the task up to spark.task.maxFailures times (default is 4 ).
Repeated counted failures of the same task can abort the stage/job at the configured threshold. A failed job does not necessarily terminate a long-lived application; fatal driver failure is different.
4. Speculative Execution ( spark.speculation )
In heterogeneous clusters, some tasks run substantially slower than others due to degraded worker hardware, slow disks, or long GC pauses (known as stragglers ).
When spark.speculation = true , Spark monitors task progress across the stage. If a task runs significantly slower than the median task completion time, Spark launches a speculative duplicate copy on an alternate executor.
Whichever task copy finishes first is committed; the slower copy is killed.
5. JVM Garbage Collection Optimization
When working with large JVM heaps (> 8 GB), garbage collection pauses are a frequent source of executor failure. If a "Stop-the-World" GC pause exceeds spark.network.timeout (default 120 seconds), the driver mistakenly assumes the executor has died, removes it from the cluster, and triggers unnecessary task recomputation.
Recommended Garbage Collector: G1GC
The Garbage-First Garbage Collector (G1GC) is one JVM option. Use the supported runtime defaults and GC logs before changing collector options; this is an illustrative configuration, not a required recipe: spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:MaxGCPauseMillis=200
Regional Division : G1GC breaks the JVM heap into hundreds of small equal regions (1 MB to 32 MB) rather than maintaining massive monolithic Old and Young generations.
Density-Based Reclamation : G1GC prioritizes regions containing the highest density of dead objects (collecting the "garbage first") to meet the configured pause target ( -XX:MaxGCPauseMillis=200 ).
Best Practices to Mitigate GC Pressure
Use DataFrames Instead of RDDs : DataFrames use Tungsten binary row format, reducing per-row JVM object allocation. JVM-object RDDs can create large object graphs; Python-object RDDs also consume Python worker memory, overwhelming the garbage collector.
Avoid Python UDFs : Python UDFs require transferring data across sockets between the JVM and Python worker subprocesses, causing extreme object allocation and serialization churn. Prefer built-in pyspark.sql.functions .
Tune spark.memory.fraction : Increasing the unified-memory fraction leaves less heap for other objects. Diagnose partition sizes, task concurrency, user objects, and GC before adjusting it; a larger fraction can worsen failures elsewhere.
Official reference
Memory and GC tuning — source checked October 16, 2026. The examples state their runtime assumptions.