6.3 Spark UI & Query Execution Diagnostics

Databricks Certified Data Engineer Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Developing Code for Data Processing using Python and SQL - Core Spark SQL & PySpark

1.1 Complex PySpark Transformations & DataFrame API 1.2 Advanced Spark SQL Patterns 1.3 Streaming Workloads with Structured Streaming 1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

2 Chapter 2: Developing Code for Data Processing using Python and SQL - Streaming & Advanced Patterns

2.1 Advanced Delta Lake Features 2.2 Custom & Stateful Streaming Operations 2.3 Python & PySpark Code Modularization 2.4 User-Defined Functions (UDFs) & Performance Impact

3 Chapter 3: Data Ingestion & Acquisition

3.1 Auto Loader (cloudFiles) Production Ingestion 3.2 Change Data Capture (CDC) Ingestion Patterns 3.3 Batch & Streaming Connectors

4 Chapter 4: Data Transformation, Cleansing, and Quality

4.1 Data Validation & Quality Frameworks 4.2 Data Cleansing, Deduplication & Standardization 4.3 Silver and Gold Medallion Layer Refinement 4.4 Handling Schema Drift and Evolution

5 Chapter 5: Data Sharing and Federation

5.1 Delta Sharing Architecture & Governance 5.2 Lakehouse Federation

6 Chapter 6: Monitoring and Alerting & Observability

6.1 System Tables & Observability 6.2 Databricks Workflows & Job Monitoring 6.3 Spark UI & Query Execution Diagnostics 6.4 Data Quality & Pipeline Alerting

7 Chapter 7: Cost & Performance Optimisation - Data Layout & Compute

7.1 Data Storage & Layout Optimization 7.2 Compute Choice & Cluster Sizing 7.3 Photon Engine & Query Acceleration

8 Chapter 8: Cost & Performance Optimisation - Skew & Cost Governance

8.1 Partitioning & Data Skew Remediation 8.2 Cost Governance & DBU Optimization

9 Chapter 9: Ensuring Data Security and Compliance

9.1 Unity Catalog Access Control & Least Privilege 9.2 Dynamic Data Masking & Row/Column-Level Security 9.3 Secrets Management & Network Security 9.4 Compliance, Audit & Data Privacy

10 Chapter 10: Data Governance in Unity Catalog

10.1 Unity Catalog Governance Architecture 10.2 Data Lineage & Tagging in Unity Catalog 10.3 Managed vs External Tables & Storage Governance

11 Chapter 11: Debugging and Deploying - Asset Bundles & CI/CD

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code 11.2 CI/CD Automation & Databricks CLI / REST API 11.3 Troubleshooting Pipeline & Job Failures 11.4 Code Versioning & Databricks Git Integration

12 Chapter 12: Data Modelling & Lakehouse Analytics

12.1 Medallion Architecture Design Patterns 12.2 Dimensional Modeling in the Lakehouse 12.3 Analytics & BI Optimization

6.3 Spark UI & Query Execution Diagnostics

Key Takeaways Explain This Topic AI

The SQL/DataFrame tab visualizes the physical execution DAG from bottom (data sources) to top (output).

A SortMergeJoin operator in the Spark UI indicates that data is being shuffled and sorted across the cluster.

Data skew can be identified in the Event Timeline when a single task's duration vastly exceeds the median task duration.

Memory Spill to disk occurs when a data partition exceeds the executor's available RAM, degrading performance.

High Garbage Collection (GC) pause times often indicate memory pressure and can be observed in the Stage tab.

Last updated: July 2026

6.3 Spark UI & Query Execution Diagnostics

While Databricks abstracts away much of the complexity of cluster management, optimizing large-scale data pipelines still requires a deep understanding of Apache Spark's execution engine. The Spark UI is the ultimate diagnostic tool for identifying performance bottlenecks, data skew, and inefficient query plans. This section explores how to navigate the Spark UI, interpret the SQL/DataFrame DAG, analyze the Event Timeline for stragglers, and detect memory spill issues. Becoming proficient with the Spark UI elevates you from simply running code to truly tuning distributed compute engines.

Overview of the Spark UI

Every cluster in Databricks hosts a Spark UI accessible from the compute tab or directly from a job run's details. The UI provides several critical tabs that offer different perspectives on the execution of your workloads:

Jobs: High-level view of all Spark actions (like count() , write() , collect() ). This is the entry point for understanding which overarching operation is consuming the most time in your application.

Stages: Detailed metrics on the stages that make up a job, including shuffle data and task durations. Since Spark establishes stage boundaries across wide transformations (shuffles), analyzing this tab helps pinpoint network and data movement bottlenecks.

SQL / DataFrame: Visual representation of the physical execution plan for Spark SQL and DataFrame API queries. This is arguable the most important tab for modern Databricks users leveraging the Catalyst Optimizer.

Executors: Memory, disk, and core usage statistics for each executor node in the cluster. Reviewing this tab helps determine if your cluster is over-provisioned or under-provisioned for the workload.

The SQL/DataFrame Tab and DAG Visualization

When you execute a query, Catalyst (Spark's query optimizer) generates a physical execution plan. The SQL tab visualizes this Directed Acyclic Graph (DAG). Understanding this visual representation allows you to verify if Spark is executing your query as efficiently as you intended.

Reading the DAG: The DAG is read from bottom to top . The leaf nodes at the bottom represent data sources (e.g., Scan parquet main.sales ). As you move up, you see transformations like Filter , Project (selecting columns), and aggregations. The top node represents the final output of the action. By tracing the path upward, you can see exactly where data volumes expand or contract during processing.

Identifying Joins in the DAG

Joins are frequent sources of performance issues. The DAG visualizes the type of join Spark selected, which provides immediate insight into performance characteristics:

BroadcastExchange / BroadcastHashJoin: Spark broadcasts the smaller table to all worker nodes. This is extremely fast because it avoids shuffling the larger table across the network. If you expect a broadcast join but don't see one, you may need to increase the spark.sql.autoBroadcastJoinThreshold or use a broadcast hint.

SortMergeJoin: Used for joining two large tables. The DAG will show an Exchange (shuffle) followed by a Sort on both branches before the SortMergeJoin . This requires moving massive amounts of data across the network, which is expensive in terms of CPU, memory, and I/O. Spotting an unexpected SortMergeJoin is often the first step in optimizing a slow query.

Stage Tab and Event Timeline: Detecting Stragglers

When a job feels sluggish, the Stage tab is the first place to investigate. Within a stage, Spark breaks work down into Tasks (one task per data partition). The efficiency of a Spark job relies on these tasks running concurrently and completing at roughly the same time.

The Event Timeline provides a visual Gantt chart of task execution across executors. In an optimized workload, tasks are distributed evenly, and the timeline looks like a solid block of green processing bars, indicating that the cluster is fully utilized.

Identifying Data Skew

Data skew occurs when data is unevenly distributed across partitions. For example, if you group by customer_id and one customer represents 40% of your sales, the task processing that customer's partition will take significantly longer than the others. Distributed processing is only as fast as its slowest task.

In the Event Timeline, skew manifests as a straggler task —a single green bar that extends far to the right while all other tasks have finished quickly. You can also view this in the Summary Metrics table for the stage: if the Max task duration is 20 minutes, but the Median (50th percentile) is 30 seconds, you have severe data skew. The cluster spends 19.5 minutes mostly idle, waiting for one node to finish.

Mitigation strategies include salting keys (adding random prefixes to distribute the skewed key), increasing the number of partitions, or ensuring that Databricks Adaptive Query Execution (AQE) skew hints are active. Modern Databricks runtimes often handle skew automatically, but manual intervention is sometimes required for extreme cases.

Garbage Collection (GC) Pauses

The Event Timeline also highlights Garbage Collection time in red. High GC time indicates that the JVM is struggling to manage memory, often because objects are being created too rapidly or partitions are too large. If GC time exceeds 10% of total task time, you are facing memory pressure. This can lead to tasks failing or the entire executor dying with an OutOfMemoryError, necessitating a deep dive into memory management or data serialization techniques.

Identifying Memory Spill

Spark performs operations primarily in RAM. However, if a single partition of data is too large to fit in the memory allocated to a task (the execution memory), Spark will pause processing, serialize the data, and write it to the local disk of the worker node. This is called Memory Spill .

Spilling to disk prevents immediate OutOfMemory (OOM) errors, but it absolutely devastates performance because disk I/O is orders of magnitude slower than RAM access. Workloads that frequently spill will experience severe degradation, often taking hours instead of minutes to complete.

Spotting Spill in the UI

In the Stage tab's summary table, look for two crucial columns:

Spill (Memory): The size of the data as it existed in RAM before spilling. This number is often larger due to the overhead of Java objects.

Spill (Disk): The size of the serialized, compressed data written to disk.

If these columns contain values greater than 0, your workload is spilling. A small amount of spill might be acceptable, but gigabytes or terabytes of spill demands immediate attention.

How to Fix Memory Spill

Increase Partitions: By increasing spark.sql.shuffle.partitions (default is 200), you divide the same amount of data into smaller, more manageable chunks that fit into the available RAM per core. This is often the quickest fix.

Handle Skew: If only one task is spilling while the others are not, it's highly likely due to data skew (one massive partition). Address the skew (as mentioned previously) to inherently fix the spill issue.

Scale Up Compute: Use cluster node types with more RAM per core (e.g., memory-optimized instances in AWS or Azure) to provide larger execution memory pools. Sometimes, throwing hardware at the problem is the most cost-effective solution when engineering time is expensive.