ExamFinch.com My Study

Study guide

3.1 SparkSession, Catalogs, Databases, and Namespaces

Practice this exam

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

3.1 SparkSession, Catalogs, Databases, and Namespaces

Key Takeaways Explain This Topic AI

SparkSession is the unified driver entry point introduced in Spark 2.0, encapsulating SparkContext, SQLContext, HiveContext, and RuntimeConfig into a single cohesive interface instantiated via SparkSession.builder.getOrCreate().

The programmatic Catalog API accessible at spark.catalog allows runtime interrogation and management of databases, tables, columns, functions, and in-memory caching without executing SQL strings.

Calling spark.newSession() spawns an isolated session sharing the existing SparkContext, cluster manager connection, and cached data, while maintaining independent SQL configuration settings, temporary views, and current database pointers.

Modern Spark 3.0+ architectures and Databricks Unity Catalog implement a three-tier namespace structure (catalog.schema.table), where unqualified identifiers resolve dynamically against current_catalog() and current_database().

Last updated: October 2026

Apache Spark SQL provides a relational programming abstraction over distributed datasets. To execute structured queries, define tables, or manipulate metadata, applications interact with the driver through foundational entry points and catalog interfaces.

1. SparkSession: The Unified Driver Entry Point

Prior to Apache Spark 2.0, applications required separate entry points depending on the intended workload: SparkContext for core RDD operations, SQLContext for standard Spark SQL, HiveContext for querying metadata from an external Apache Hive metastore, and StreamingContext for real-time DStreams. This fragmented design led to configuration synchronization issues and boilerplate code.

Spark 2.0 introduced SparkSession , consolidating these disparate contexts into a single unified driver interface. In modern PySpark and Databricks environments, SparkSession is the primary entry point for reading structured files, executing SQL queries, configuring engine parameters, and interacting with catalog metadata.

The Builder Pattern

SparkSession instances are constructed using the builder pattern via SparkSession.builder . The .getOrCreate() method retrieves an existing active SparkSession within the driver JVM or instantiates a new one if none exists. from pyspark.sql import SparkSession

  • # Configure and instantiate a SparkSession
  • spark = SparkSession.builder \
  • .appName("DatabricksCatalogManagement") \
  • .config("spark.sql.shuffle.partitions", "16") \
  • .config("spark.sql.adaptive.enabled", "true") \
  • .enableHiveSupport() \
  • .getOrCreate()
  • Core Properties and Encapsulated Sub-Contexts

Once instantiated, the spark object provides direct access to lower-level subsystems and helper utilities:

spark.sparkContext : The underlying low-level SparkContext responsible for task scheduling, cluster manager coordination, RDD creation, and hardware metric collection.

spark.conf : The runtime configuration interface ( RuntimeConfig ), used to inspect and set dynamic SQL parameters (e.g., spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "10485760") ).

spark.catalog : The catalog interface ( Catalog ) exposing programmatic metadata operations over databases, tables, functions, and columns.

spark.read and spark.readStream : The DataFrameReader and DataStreamReader entry points for loading static and streaming datasets.

spark.udf : The registration interface for SQL-callable user-defined functions; grouped pandas APIs have their own interfaces.

Isolated Sessions: spark.newSession()

A critical exam concept is session isolation. Calling spark.newSession() instantiates a new SparkSession sharing the exact same underlying SparkContext , executor resources, and cluster manager allocation. However, the new session maintains:

An isolated SQL configuration environment (independent spark.conf settings).

An isolated catalog of standard temporary views.

  • An isolated default database/schema pointer. # Create an isolated child session
  • session_analytics = spark.newSession()
  • # Changes to configuration in session_analytics do NOT alter the parent session
  • session_analytics.conf.set("spark.sql.shuffle.partitions", "64")
  • print(spark.conf.get("spark.sql.shuffle.partitions")) # Outputs: 16
  • print(session_analytics.conf.get("spark.sql.shuffle.partitions")) # Outputs: 64

Cached tables and physical data cached in memory/disk by the parent session remain accessible across both sessions because they reside in the shared executor memory managed by the single underlying SparkContext .

2. Programmatic Catalog Management: spark.catalog

The spark.catalog object exposes metadata management methods without requiring SQL string compilation. In PySpark, list methods return local Python lists of metadata objects, not DataFrames or Scala Datasets.

Metadata Inspection Methods

spark.catalog.listDatabases() : Returns a Python list of Database objects containing name , catalog , description , and locationUri .

spark.catalog.listTables(dbName=None) : Returns a Python list of Table objects containing name , catalog , namespace , description , tableType , and isTemporary . If dbName is omitted, it lists tables and temporary views in the current database.

spark.catalog.listColumns(tableName, dbName=None) : Returns column metadata including name , description , dataType , nullable , isPartition , and isBucket .

spark.catalog.listFunctions(dbName=None) : Returns registered SQL functions (built-in, UDFs, and external metastore functions) with name , className , and isTemporary . # Inspect databases currently available in the metastore

databases = spark.catalog.listDatabases()

for db in databases:

print(f"Database: {db.name} | Location: {db.locationUri}")

  • # Inspect tables in the current active database
  • tables = spark.catalog.listTables()

for tbl in tables:

  • print(f"Table: {tbl.name} | Type: {tbl.tableType} | Temporary: {tbl.isTemporary}")
  • Active Context Getters and Setters
  • To inspect or change the active database programmatically: # Retrieve current database name
  • current_db = spark.catalog.currentDatabase()
  • # Change current database context programmatically
  • spark.catalog.setCurrentDatabase("finance_analytics")
  • Cache Administration

spark.catalog provides targeted cache control over specific registered entities:

spark.catalog.cacheTable("table_name") : Lazy or eager caching of a relational table or view into executor memory.

spark.catalog.uncacheTable("table_name") : Evicts the specified table from executor memory and storage.

spark.catalog.isCached("table_name") : Returns True if the table is currently cached.

spark.catalog.clearCache() : Removes cached table data in the shared cache manager; it does not delete table definitions or all query plans.

Metadata Synchronization

When external systems (such as Hive, Presto, or an external ETL pipeline) write new files or drop partitions directly in object storage, Spark's in-memory catalog cache becomes stale. spark.catalog provides programmatic invalidation:

spark.catalog.refreshTable("table_name") : Invalidates cached metadata, file block locations, and physical schema definitions for the specified table.

spark.catalog.refreshByPath("/path/to/data") : Invalidates cached metadata for any table or relation backed by the designated storage path.

spark.catalog.recoverPartitions("table_name") : Scans the underlying directory structure of a partitioned table and registers any newly detected partition directories into the catalog metastore (equivalent to SQL MSCK REPAIR TABLE ).

3. Declarative SQL Namespace and Database Commands

In addition to the programmatic Python API, Spark SQL supports ANSI-compliant Data Definition Language (DDL) statements executed through spark.sql("...") . -- Retrieve the active catalog and database

SELECT current_catalog(), current_database();

  • - List available databases matching a prefix pattern
  • SHOW DATABASES LIKE 'prod_*';
  • SHOW SCHEMAS;
  • - Create a new database with explicit location and metadata properties
  • CREATE DATABASE IF NOT EXISTS retail_analytics
  • COMMENT 'Centralized retail transactions and customer analytics'
  • LOCATION 's3://company-lakehouse-warehouse/retail_analytics.db'
  • WITH DBPROPERTIES ('environment' = 'production', 'tier' = 'gold');
  • - Switch active database
  • USE retail_analytics;
  • - Inspect extended metadata, parameters, and storage location
  • DESCRIBE DATABASE EXTENDED retail_analytics;
  • Dropping Databases: RESTRICT vs CASCADE

When dropping a database, Spark SQL defaults to RESTRICT mode:

DROP DATABASE IF EXISTS retail_analytics RESTRICT; : Throws an AnalysisException if the database contains any tables, views, or functions.

DROP DATABASE IF EXISTS retail_analytics CASCADE; : Recursively drops all child tables and views contained within the database before dropping the database entity itself. For managed tables, physical storage directories are permanently deleted.

4. Modern Multi-Catalog Architecture & Three-Tier Namespaces

Traditional Spark deployments relied on a single two-tier namespace: <database/schema>.<table> . Spark 3.0 introduced pluggable catalog provider interfaces ( org.apache.spark.sql.connector.catalog.CatalogPlugin ), establishing a hierarchical three-tier namespace: catalog ⟶ schema (database) ⟶ table / view / volume \text{catalog} \;\longrightarrow\; \text{schema (database)} \;\longrightarrow\; \text{table / view / volume} catalog ⟶ schema (database) ⟶ table / view / volume

Databricks Unity Catalog Architecture

In modern Databricks environments powered by Unity Catalog, governance is organized into three distinct tiers:

Catalog (Top Tier) : The highest organizational boundary for data assets. Organizations frequently separate catalogs by environment ( dev , staging , prod ) or business functional units ( finance , marketing , supply_chain ).

Schema / Database (Middle Tier) : Logical groupings of related tables, views, volumes, and functions within a catalog.

Table / View (Bottom Tier) : The actual structured data entities or saved SQL queries.

Identifier Resolution Mechanics

When a query executes, Spark resolves object references based on the number of specified namespace tokens:

Three-part identifier ( prod_catalog.finance.transactions ): Fully qualified. Spark directs the query directly to prod_catalog , navigates to finance , and resolves transactions .

Two-part identifier ( finance.transactions ): Partially qualified. Spark resolves the schema finance within the currently active catalog returned by current_catalog() .

One-part identifier ( transactions ): Unqualified. Spark resolves the entity within the currently active catalog and active database returned by current_database() . -- Databricks Unity Catalog example (commands are platform-specific)

  • - View registered catalogs
  • SHOW CATALOGS;
  • - Switch active catalog context
  • USE CATALOG prod_catalog;
  • - Switch active schema within the active catalog
  • USE SCHEMA finance;
  • - This unqualified query resolves to prod_catalog.finance.transactions
  • SELECT * FROM transactions;
  • 5. Catalog API vs. Spark SQL Command Reference

The following reference table summarizes the programmatic spark.catalog methods alongside their exact Spark SQL declarative equivalents:

  • Operation Description Programmatic Catalog API ( spark.catalog ) Declarative Spark SQL Equivalent
  • List available databases spark.catalog.listDatabases() SHOW DATABASES or SHOW SCHEMAS
  • List tables in database spark.catalog.listTables(dbName="finance") SHOW TABLES IN finance
  • List columns of table spark.catalog.listColumns("orders", "finance") DESCRIBE TABLE finance.orders
  • List registered functions spark.catalog.listFunctions() SHOW FUNCTIONS
  • Inspect current database spark.catalog.currentDatabase() SELECT current_database()
  • Set current database spark.catalog.setCurrentDatabase("finance") USE finance
  • Cache table in memory spark.catalog.cacheTable("orders") CACHE TABLE orders
  • Evict table from cache spark.catalog.uncacheTable("orders") UNCACHE TABLE orders
  • Check table cache state spark.catalog.isCached("orders") None (inspect spark.catalog.isCached or Spark UI)
  • Refresh metadata & files spark.catalog.refreshTable("orders") REFRESH TABLE orders
  • Recover partition paths spark.catalog.recoverPartitions("orders") MSCK REPAIR TABLE orders
  • Session lifecycle is not application duplication

getOrCreate() can reuse an existing session rather than start another application. newSession() creates isolated SQL configuration and temporary-view state while sharing the SparkContext and cache manager. In classic Spark, SparkSession.stop() stops that underlying context, so stopping one shared-context session can end work for the others; it is not a way to close just one isolated SQL namespace while leaving the application untouched. Let managed notebook/platform lifecycle rules control shared compute. Spark Connect has separate remote-session semantics, so do not generalize classic context access to a thin client.