Study guide
9.2 Data Sources, SaveModes, and Partitioning
Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Spark Architecture and Execution
1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage
2 Chapter 2: Shuffles, Memory and Persistence
2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels
3 Chapter 3: Spark SQL Sessions, Tables and Views
3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views
4 Chapter 4: SQL Sources and Stored Layout
4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout
5 Chapter 5: SQL Queries and Analytics
5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics
6 Chapter 6: DataFrame Construction and Columns
6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions
7 Chapter 7: DataFrame Validation and Inspection
7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion
8 Chapter 8: Aggregations and Dates
8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions
9 Chapter 9: DataFrame Joins and File IO
9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning
10 Chapter 10: Shared Variables and Stateful Processing
10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores
11 Chapter 11: Troubleshooting and Tuning
11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins
12 Chapter 12: Structured Streaming
12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks
13 Chapter 13: Spark Connect and Deployment
13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices
14 Chapter 14: Pandas API on Spark
14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization
9.2 Data Sources, SaveModes, and Partitioning
Key Takeaways Explain This Topic AI
The DataFrameReader ( spark.read ) and DataFrameWriter ( df.write ) interfaces provide a unified, pluggable API supporting formats including CSV, JSON, Parquet, ORC, and Delta Lake.
Parquet is Spark's default storage format, featuring columnar layout, binary dictionary encoding, Snappy compression, and a metadata footer that enables predicate pushdown and column pruning.
DataFrameWriter.mode() defaults to "error" (or "errorifexists" ), throwing an AnalysisException if destination files or tables already exist.
Directory partitioning via partitionBy() creates subdirectories ( col=value/ ) that allow downstream queries to skip irrelevant data (partition pruning), but high-cardinality keys must be avoided to prevent the small file problem.
Bucketing via bucketBy() and sortBy() hashes and sorts records into a fixed bucket assignment when saving a supported catalog table with saveAsTable(); compatible bucket metadata can allow a join exchange to be avoided.
Last updated: October 2026
Data ingestion and data persistence represent the boundaries of any Apache Spark application. Spark provides a unified, format-agnostic abstraction for reading data from external storage into DataFrames via the DataFrameReader interface, and writing DataFrames back to storage via the DataFrameWriter interface. This section covers the read/write APIs, storage format trade-offs, write modes, and physical layout optimization techniques including directory partitioning and table bucketing.
1. The Unified Data Source API
Apache Spark standardizes all data ingestion and persistence through a consistent, fluent builder pattern: spark.read.format(...).option(...).load(...) --> DataFrameReader
- df.write.format(...).mode(...).option(...).save(...) --> DataFrameWriter
- The DataFrameReader Interface # Generic DataFrameReader syntax
- df = spark.read \
- .format("csv") \
- .option("header", "true") \
- .option("inferSchema", "true") \
- .option("sep", ",") \
- .load("dbfs:/data/raw_transactions.csv")
- # Equivalent format-specific shortcut syntax
- df = spark.read.csv(
- "dbfs:/data/raw_transactions.csv",
- header=True,
- inferSchema=True,
- sep=","
- )
- The DataFrameWriter Interface # Generic DataFrameWriter syntax
- df.write \
- .format("parquet") \
- .mode("overwrite") \
- .option("compression", "snappy") \
- .save("dbfs:/data/processed_transactions.parquet")
- # Equivalent format-specific shortcut syntax
- df.write.mode("overwrite").parquet("dbfs:/data/processed_transactions.parquet")
- 2. Storage Formats: Row-Oriented, Columnar, and Delta Lake
Choosing the appropriate file format fundamentally dictates storage footprint, read latency, and cluster compute costs.
Format Storage Layout Schema Handling Compression Best Use Case
CSV Row-oriented text No embedded schema; requires inferSchema (two scans) or explicit schema Text compression (gzip) Legacy data exchange, external third-party feeds
JSON Row-oriented semi-structured Self-describing keys per record; schema inferred or supplied Text compression (gzip) API payloads, web event logs (use multiLine=True for pretty-printed JSON)
Parquet Binary columnar Embedded schema and statistics in file footer Column-level compression (Snappy, Zstandard) Spark default format ; high-performance analytical queries and BI
ORC Binary columnar Embedded schema and stripe statistics Column-level compression (ZLIB, Snappy) Highly optimized for Apache Hive environments
Delta Lake Parquet files + ACID log Embedded transaction log ( _delta_log/ ) Snappy (inherited from Parquet) Production lakehouses requiring ACID transactions, time travel, and schema evolution
Why Parquet is the Analytical Gold Standard
Parquet is Apache Spark's default storage format ( spark.sql.sources.default=parquet ). Unlike CSV or JSON, which store data row-by-row as plain text, Parquet stores data in binary columnar format :
Column Pruning: When a query executes df.select("user_id", "amount") , Spark's Parquet reader reads only the byte ranges corresponding to those two columns from disk or cloud storage. All other columns are completely bypassed, drastically reducing network and disk I/O.
Predicate Pushdown: Parquet files organize records into horizontal blocks called Row Groups (typically 128 MB each). The Parquet file footer stores metadata statistics—specifically minimum and maximum values—for every column chunk within each row group. When a query contains a filter like .filter("amount > 1000") , Spark inspects the footer metadata first. If a row group's maximum amount is 450, Spark skips reading the entire 128 MB row group entirely.
Efficient Encodings: Because columns contain homogeneous data types, Parquet applies dictionary encoding, bit packing, and run-length encoding, often reducing size relative to text; actual ratios depend on data, encoding and codec.
3. SaveModes: Controlling Destination Write Behavior
When writing a DataFrame to a destination path or table that already exists, the SaveMode controls how Spark handles the conflict. Specify the mode via df.write.mode(mode_string) :
SaveMode String Java Enum Equivalent Behavior if Target Exists Behavior if Target Does Not Exist
"error" / "errorifexists" SaveMode.ErrorIfExists Default. Throws an AnalysisException halting execution Creates target directory and writes data
"append" SaveMode.Append Appends new files alongside existing files Creates target directory and writes data
"overwrite" SaveMode.Overwrite Deletes existing contents and replaces with new data Creates target directory and writes data
"ignore" SaveMode.Ignore Silently does nothing; does not overwrite or write new data Creates target directory and writes data # Example of setting SaveMode
df.write.mode("append").parquet("dbfs:/mnt/datalake/events")
Caution
Non-Atomic Cloud Overwrites: On standard cloud object stores (Amazon S3, Azure Blob Storage), a standard Parquet overwrite operation is not atomic . Spark deletes files in the bucket and uploads new ones; if a failure occurs halfway through, the dataset is left corrupted or empty. In contrast, Delta Lake writes new files first and commits atomically via its transaction log, guaranteeing ACID durability.
4. Physical Storage Layout: Partitioning vs. Bucketing
Organizing physical files on disk to match common query filter and join patterns is the most effective way to eliminate unnecessary I/O.
Directory Partitioning ( partitionBy )
df.write.partitionBy(*cols) divides output data into physical subdirectories based on the distinct values of the specified partitioning columns: # Partitioning transactions by year and month
- df.write \
- .partitionBy("year", "month") \
- .parquet("dbfs:/data/transactions")
- This creates a nested directory structure on the filesystem: dbfs:/data/transactions/
- ├── year=2026/
- │ ├── month=09/
- │ │ ├── part-00000.snappy.parquet
- │ │ └── part-00001.snappy.parquet
- │ └── month=10/
- │ ├── part-00000.snappy.parquet
- │ └── part-00001.snappy.parquet
When downstream queries run df.filter("year = 2026 AND month = 10") , Spark performs partition pruning , reading only files within the year=2026/month=10/ directory and ignoring all other folders.
Important
The Small File Problem (Anti-Pattern): Never partition by a high-cardinality column (such as timestamp , user_id , or order_id ). Partitioning across millions of unique keys creates millions of tiny subdirectories containing files of only a few kilobytes. This exhausts cloud storage API rate limits, degrades listing performance, and causes massive driver memory overhead.
Table Bucketing ( bucketBy and sortBy )
For high-cardinality columns where directory partitioning is unsuitable, use Bucketing : # Bucketing data by customer_id into 16 hash buckets
- df.write \
- .bucketBy(16, "customer_id") \
- .sortBy("transaction_date") \
- .saveAsTable("bucketed_transactions")
Key rules and benefits of bucketing:
Potential Join Exchange Reuse: When two tables are bucketed on the same join key with the same number of buckets (or multiples thereof), Spark may avoid a join shuffle when compatible bucketing is recognized and enabled; sort requirements and provider support still apply. Bucket count need not equal physical file count. It can perform a SortMergeJoin with fewer exchanges. Each bucket from Table A joins directly with the corresponding bucket from Table B locally.
Persistent Table Requirement: bucketBy() works only with saveAsTable() . It cannot be used with raw file path writes ( .save("path") ) because bucketing metadata must be tracked in the Spark metastore/catalog.