ExamFinch.com My Study

Study guide

4.1 JDBC Reads, Writes, and Parallelism

Practice this exam

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

4.1 JDBC Reads, Writes, and Parallelism

Key Takeaways Explain This Topic AI

JDBC needs the appropriate database driver and connectivity from the processes that access the database.

partitionColumn, lowerBound, upperBound and numPartitions configure parallel reads; bounds are not a WHERE filter.

query and dbtable are alternatives; query cannot be combined with partitionColumn in the standard reader.

JDBC append is not an automatic upsert or a transaction spanning all Spark partitions.

Last updated: October 2026

JDBC as a Spark SQL data source

Java Database Connectivity (JDBC) lets Spark read a relational database into a DataFrame and write tabular results back. It differs from a Spark SQL JDBC server: the data-source connector makes Spark the client of another database, while a SQL server endpoint lets an external client query Spark. Certification scenarios often test the correct direction and where parallelism is configured.

The source-specific JDBC driver must be available on Spark's classpath, and the driver/executor environment needs appropriate network access and authorization. Supplying a PostgreSQL URL does not install its driver. In the examples below, jdbc_url, jdbc_user and jdbc_password are assumed to be supplied securely by the environment. They are placeholders for already authorized connection settings, not values to paste into shared source or logs. orders = (spark.read.format("jdbc")

  • .option("url", jdbc_url)
  • .option("dbtable", "sales.orders")
  • .option("user", jdbc_user)
  • .option("password", jdbc_password)
  • .option("driver", "org.postgresql.Driver")
  • .load())
  • orders.createOrReplaceTempView("jdbc_orders")
  • report = spark.sql("""
  • SELECT region, SUM(amount) AS revenue
  • FROM jdbc_orders
  • WHERE status = 'COMPLETE'
  • GROUP BY region
  • """)

The relational operations use Spark's SQL engine, with eligible filters pushed to the source according to connector support. Inspect the plan and database workload rather than assuming the whole query runs exclusively on either system. Schema discovery itself can contact the database before a later row action.

Reading a query or a table

The dbtable option identifies a table or a parenthesized, aliased query usable in a SQL FROM clause. The query option supplies a SELECT which Spark wraps as a subquery. Do not specify both. A database can impose restrictions on the SQL accepted inside that wrapper. The standard reader also forbids combining query with partitionColumn; put an aliased subquery into dbtable when parallel partition reads are required. active = (spark.read.format("jdbc")

  • .option("url", jdbc_url)
  • .option("dbtable", "(SELECT id, region, amount FROM sales.orders WHERE status = 'COMPLETE') AS src")
  • .option("user", jdbc_user)
  • .option("password", jdbc_password)
  • .option("partitionColumn", "id")
  • .option("lowerBound", 0)
  • .option("upperBound", 1000000)
  • .option("numPartitions", 4)
  • .load())

The explicit WHERE inside dbtable filters the population. By contrast, lowerBound and upperBound only calculate partition strides. Rows outside those values are not thereby excluded. If a question requires only IDs from 0 through 999999, encode that predicate in the source query or an appropriate filter; do not answer that the partition bounds enforce it. The partitioning column must be numeric, date or timestamp, and all required partition options must be supplied together.

Balance parallelism against source pressure

numPartitions limits read/write parallelism and concurrent JDBC connections for the connector. More connections can improve throughput until the database, network or connection pool saturates. They can also hurt a transactional source. A small table may not benefit from dozens of readers; a highly skewed ID range can leave one read partition much larger than the rest.

  • Option Direction Main purpose
  • fetchsize Read Rows fetched per database round trip, subject to driver behavior
  • batchsize Write Rows inserted per JDBC batch
  • numPartitions Read/write Bound connector parallelism and connections
  • partitionColumn and bounds Read Divide value range into parallel read strides
  • queryTimeout Driver-dependent Limit statement execution time

For a worked capacity decision, suppose a source owner permits at most four concurrent extraction connections. A requested numPartitions of 32 violates that budget even if the Spark cluster has 32 cores. Start with four and measure. Increasing Spark executors does not override database limits. If one partition dominates, inspect value distribution or use deliberately chosen predicate partitions rather than increasing load blindly.

  • Writing results and save modes (report.write.format("jdbc")
  • .option("url", jdbc_url)
  • .option("dbtable", "analytics.region_revenue")
  • .option("user", jdbc_user)
  • .option("password", jdbc_password)
  • .option("numPartitions", 4)
  • .option("batchsize", 1000)
  • .mode("append")
  • .save())

Append inserts rows; it does not deduplicate a previous load or update an existing primary key automatically. Re-running this code can insert duplicate reports or fail a database constraint. Overwrite can drop/recreate a table depending on dialect and options, potentially affecting indexes and grants. A supported truncate option has different behavior, but is not portable across all databases or schema changes. Verify source-specific behavior before using overwrite for a production table.

A write is distributed across partitions/connections. A transaction isolation setting applies to a connection, not an automatic global transaction over every partition and external system. Retries and partial success need a deliberate ingestion design, such as staging data under a load identifier and applying a database-side transactional merge. The connector alone does not supply business-key upsert semantics.

Choosing the answer in a scenario

Separate three questions: which rows should be read, how many connections may read them, and where the output should go. Use a WHERE predicate for selection, partition options for read strides, and a deliberate writer mode for destination behavior. Keep credentials outside code, confirm the driver/classpath, and inspect pushed filters and per-partition workload. A SQL temporary view registered from the JDBC DataFrame remains session-scoped and does not create a new persistent database table.

See the official JDBC source documentation for supported options and dialect restrictions.