Study guide
2.3 Caching, Persistence, and StorageLevels
Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Spark Architecture and Execution
1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage
2 Chapter 2: Shuffles, Memory and Persistence
2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels
3 Chapter 3: Spark SQL Sessions, Tables and Views
3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views
4 Chapter 4: SQL Sources and Stored Layout
4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout
5 Chapter 5: SQL Queries and Analytics
5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics
6 Chapter 6: DataFrame Construction and Columns
6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions
7 Chapter 7: DataFrame Validation and Inspection
7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion
8 Chapter 8: Aggregations and Dates
8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions
9 Chapter 9: DataFrame Joins and File IO
9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning
10 Chapter 10: Shared Variables and Stateful Processing
10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores
11 Chapter 11: Troubleshooting and Tuning
11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins
12 Chapter 12: Structured Streaming
12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks
13 Chapter 13: Spark Connect and Deployment
13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices
14 Chapter 14: Pandas API on Spark
14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization
2.3 Caching, Persistence, and StorageLevels
Key Takeaways Explain This Topic AI
In PySpark DataFrames, df.cache() is a convenience shortcut for df.persist(StorageLevel.MEMORY_AND_DISK_DESER) ; if storage capacity cannot hold the cached blocks, excess partitions are safely spilled to disk rather than dropped and recomputed.
Spark provides multiple StorageLevel configurations that control memory vs. disk placement, serialization flags (Python constants differ from Scala names), and multi-node replication ( _2 suffix for fault tolerance without recomputation).
Caching is evaluated lazily; invoking .cache() or .persist() merely annotates the logical execution plan, and no data is materialized into memory until a downstream action (e.g., count() , take() , write ) is executed.
Unneeded cached DataFrames should be promptly released using df.unpersist() ; lingering caches consume storage memory, triggering Least Recently Used (LRU) cache evictions that degrade performance.
Catalyst and Project Tungsten coordinate with caching: Tungsten maintains an optimized in-memory columnar format, while Catalyst can push projection pruning and filter pushdowns directly into cached columnar memory buffers.
Last updated: October 2026
In Apache Spark, DataFrames are evaluated lazily. Without a reusable cache or other retained/reused results, later actions may repeat upstream computation; exact work depends on the optimized plan and available outputs. In complex analytical pipelines, iterative machine learning algorithms, or branching workflows, re-executing identical computations introduces massive CPU, I/O, and network redundancies. Caching and persistence mechanisms allow developers to store intermediate DataFrame partitions in memory or on disk for rapid downstream reuse.
1. Caching Fundamentals: cache() vs. persist()
Spark provides two primary methods on the DataFrame API to retain intermediate datasets across actions: cache() and persist() . # Using cache() shortcut
df_cached = df.cache()
- # Using persist() with an explicit StorageLevel
- from pyspark.storage import StorageLevel
- df.unpersist() # Remove an existing persistence assignment before changing its level
- df_persisted = df.persist(StorageLevel.MEMORY_ONLY)
- The Relationship Between cache() and persist()
df.cache() is simply an alias that calls df.persist() with a default storage level:
In PySpark DataFrames , df.cache() defaults to StorageLevel.MEMORY_AND_DISK_DESER (in Scala/Java DataFrames and Spark SQL, this corresponds to StorageLevel.MEMORY_AND_DISK ).
In low-level RDDs , rdd.cache() historically defaulted to StorageLevel.MEMORY_ONLY .
Because the PySpark DataFrame default MEMORY_AND_DISK_DESER permits disk storage, if the dataset exceeds available executor memory, Spark does not drop the overflow partitions; instead, it automatically serializes and spills them to local executor disks.
Lazy Evaluation of Caching
A critical concept on Databricks certification exams is the lazy evaluation of caching : # Step 1: Tag the DataFrame for caching (NO DATA IS CACHED YET)
clean_df = raw_df.filter("age > 21").cache()
- # Step 2: An action is called (DATA IS COMPUTED AND STORED IN CACHE HERE)
- row_count = clean_df.count()
- # Step 3: Subsequent actions reuse the cached memory buffers directly
- summary_df = clean_df.groupBy("country").avg("income").collect()
When clean_df.cache() is invoked, Spark merely attaches an InMemoryRelation marker to the logical execution plan. No memory is allocated, and no files are read. Only when the first action ( .count() ) executes does Spark read the source data, apply the filter, and materialize the resulting partition blocks into executor storage memory.
2. StorageLevel Configurations and Trade-offs
The pyspark.storage.StorageLevel class provides fine-grained control over how cached partition blocks are stored. A StorageLevel encapsulates four boolean flags and an integer replication count: useDisk , useMemory , useOffHeap , deserialized , and replication .
- PySpark constant Disk Memory Deserialized flag Replication
- MEMORY_ONLY No Yes False 1
- MEMORY_AND_DISK Yes Yes False 1
- MEMORY_AND_DISK_DESER Yes Yes True 1
- DISK_ONLY Yes No False 1
- MEMORY_ONLY_2 No Yes False 2
- MEMORY_AND_DISK_2 Yes Yes False 2
- OFF_HEAP Yes Yes, off heap False 1
These are Python API constants in Spark 3.5.7. Scala names ending in _SER must not be copied into PySpark code: MEMORY_AND_DISK_SER and MEMORY_ONLY_SER are not PySpark constants. DataFrame SQL caching uses cached columnar batches; these flags do not mean each SQL row becomes a Python or Java object. OFF_HEAP needs off-heap configuration and sufficient native/container capacity; it does not eliminate JVM GC. Replication improves availability but does not replace a durable source or checkpoint.
Important
DataFrame Serialization Under Project Tungsten: While low-level RDDs distinguish sharply between serialized ( _SER ) and deserialized objects, PySpark DataFrames are automatically stored in Tungsten's compact, off-heap-capable binary columnar format. Thus, DataFrame caching in Spark SQL is already highly serialized and memory-efficient by design.
3. The Cache Lifecycle: Best Practices and Eviction
Caching is not free. Misusing cache can degrade performance rather than improving it.
When TO Cache
Iterative Algorithms: Workflows that access the same intermediate dataset repeatedly in a loop (e.g., machine learning model training loops such as Logistic Regression, K-Means, or Gradient Boosted Trees).
Multi-Branching Pipelines: When an expensive set of transformations (e.g., parsing raw JSON, deduplicating, and joining with dimension tables) produces a clean DataFrame that is subsequently consumed by multiple independent downstream actions or written to multiple destinations. # Good caching candidate: clean_df is consumed by two separate branches
clean_df = raw_df.join(lookup_df, "id").filter("is_valid == true").cache()
- # Branch 1: Write raw clean data to Parquet
- clean_df.write.parquet("/mnt/lakehouse/clean_orders")
- # Branch 2: Compute aggregation metrics
- metrics_df = clean_df.groupBy("region").agg(F.sum("amount"))
- metrics_df.write.parquet("/mnt/lakehouse/regional_metrics")
- When NOT to Cache
Single-Use DataFrames: If a DataFrame is read, transformed, and written to disk exactly once, calling .cache() adds pure write and read overhead without any performance benefit.
DataFrames That Vastly Exceed Memory: A 2 TB cache on 200 GB RAM may use substantial disk, evict blocks, or perform poorly. Compare avoided recomputation against cache IO rather than assuming all such caches are beneficial or harmful.
Caching Prior to Narrow Filters: Caching a DataFrame before filtering restricts the Catalyst optimizer from pushing filter predicates down into the underlying Parquet/Delta storage reader.
Explicit Eviction with unpersist()
When a cached DataFrame is no longer needed in subsequent stages of a pipeline, it should be eagerly removed from memory using df.unpersist() : # Free cached memory buffers
clean_df.unpersist(blocking=True)
The blocking Argument: Setting blocking=True halts the calling thread until all executor nodes have completely evicted the partition blocks from memory and disk. By default, blocking=False initiates asynchronous eviction in the background.
Automatic LRU Eviction: If memory becomes saturated and new data must be cached, Spark automatically evicts the oldest cached blocks using a Least Recently Used (LRU) policy. However, relying strictly on LRU eviction can lead to unpredictable memory pressure and unexpected recomputations.
4. Catalyst Optimizer and Project Tungsten Memory Coordination
Caching interacts directly with Spark's underlying query optimizer (Catalyst) and memory execution engine (Project Tungsten).
Caching as a Lineage Boundary
When a DataFrame is cached, Catalyst treats the cached relation as an InMemoryRelation :
It substitutes a cached relation in the execution plan without deleting recovery lineage for downstream queries, reading pre-computed columnar buffers rather than re-evaluating upstream operators.
In-Memory Optimization: Although the cached scan substitutes for upstream computation, Catalyst can still optimize queries that read from the cache. For example, if a query selects only two columns from a 50-column cached DataFrame, Catalyst performs column pruning on the in-memory columnar store, reading only the two required column byte streams.
Project Tungsten Columnar Storage
Under Project Tungsten, cached DataFrames are stored in a compact in-memory columnar representation:
Columnar In-Memory Format: Values from the same column are stored contiguously in memory, enabling hardware-level Single Instruction Multiple Data (SIMD) CPU vectorization.
Compact buffers: Columnar cached batches reduce per-row object overhead. They are not automatically all off heap, and native memory configuration is separate.
Whole-Stage Code Generation: Physical plan operators reading from cached memory buffers continue to leverage compiled bytecode loops (indicated by the * prefix in df.explain() ), reducing interpretation overhead where the operators support code generation; throughput depends on the workload.
Official reference
PySpark storage levels — source checked October 16, 2026. The examples state their runtime assumptions.