Study guide

13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts

Microsoft Certified: Azure Databricks Data Engineer Associate DP-750

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Azure Databricks Architecture & Compute Selection

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

2 Chapter 2: Compute Configuration, Performance & Cost Management

2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup

3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown

4 Chapter 4: Unity Catalog Privilege Model & Access Control

4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation

5 Chapter 5: Advanced Data Security, Masking & ABAC

5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks

6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing

6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming

7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment

10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables

10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs

11 Chapter 11: Workflow Orchestration with Lakeflow Jobs

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles

12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines

13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning

13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts

Key Takeaways Explain This Topic AI

Driver OutOfMemoryError (OOM) typically stems from pulling large datasets into the driver process via .collect(), .toPandas(), collecting massive metadata from millions of tiny files, or broadcasting datasets that exceed driver memory.

Executor OutOfMemoryError occurs when executor execution and storage memory are exhausted by memory-intensive operations, oversized partition processing, non-vectorized Python UDF buffers, or extreme join skew.

Spill (Memory) represents the deserialized in-memory footprint of data evicted from RAM, while Spill (Disk) represents the serialized, compressed bytes written to local NVMe storage; high disk spill severely degrades pipeline throughput due to disk I/O and CPU serialization penalties.

Data skew can be remediated manually by salting join/grouping keys with random integers (0..N-1) or automatically by enabling Adaptive Query Execution (AQE) skew join optimization.

Azure Spot VM evictions trigger executor loss and shuffle fetch failures (FetchFailedException); Databricks mitigates this through automatic stage resubmission, local shuffle tracking, and Spot fallback to on-demand nodes.

Last updated: August 2026

13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts

Distributed data processing across elastic cloud infrastructure introduces complex failure modes. When an Azure Databricks pipeline fails or degrades, data engineers must rapidly identify whether the root cause stems from architectural antipatterns (such as collecting data to the driver), memory exhaustion, execution disk spill, data skew, or cloud infrastructure disruptions (such as Azure Spot VM evictions).

Understanding JVM memory architecture, shuffle mechanics, and error signatures is essential for resolving production incidents.

1. Anatomy of OutOfMemory (OOM) Errors: Driver vs. Executor

Apache Spark separates memory management into two distinct JVM processes: the Driver (master orchestrator) and Executors (worker compute engines). An OutOfMemoryError (OOM) manifest in either process has distinct root causes and remediations. +-------------------------------------------------------------------------+

  • | DRIVER OOM VS. EXECUTOR OOM |
  • +-------------------------------------------------------------------------+
  • | |
  • | DRIVER OUTOFMEMORYERROR EXECUTOR OUTOFMEMORYERROR |
  • | * Error: java.lang.OutOfMemoryError * Error: ExecutorLostFailure |
  • | Java heap space * Container killed by YARN/K8s |
  • | * Location: Driver JVM * Location: Worker Node JVM |
  • | |
  • | COMMON CAUSES: COMMON CAUSES: |
  • | 1. Calling df.collect() or .toPandas() 1. Giant skewed partitions |
  • | 2. Broadcasting table > driver RAM 2. Low partition count (few |
  • | 3. Excessive unpartitioned file gigantic tasks per core) |
  • | metadata (millions of files) 3. Memory-heavy Python UDFs |
  • | 4. Heavy SparkContext logging 4. Insufficient executor heap |
  • | |
  • | REMEDIATION: REMEDIATION: |
  • | - Write directly to Delta tables - Increase shuffle partitions |
  • | - Use .take(n) or .limit(n) - Upsize worker VM memory |
  • | - Upsize Driver VM instance - Salt skewed join keys |
  • +-------------------------------------------------------------------------+
  • 1. Driver OutOfMemoryError Diagnosis & Fixes

When a driver runs out of memory, the entire Spark session crashes, aborting all active jobs.

Antipattern: .collect() or .toPandas() on Large Datasets : # DANGEROUS: Pulls 50 million rows across network into driver JVM heap!

  • all_data = df.collect()
  • pandas_df = df.toPandas()
  • # RECOMMENDED: Filter, aggregate, or sample on workers before collecting
  • sample_data = df.limit(100).toPandas()

# Or write directly to cloud storage without touching the driver:

df.write.format("delta").mode("append").saveAsTable("silver.cleansed_events")

Antipattern: Oversized Broadcast Joins :

Broadcasting a DataFrame via broadcast(df) forces the driver to collect the entire dataset into driver RAM before serializing and broadcasting it to executors. If the broadcast table exceeds spark.driver.memory (or spark.sql.autoBroadcastJoinThreshold ), the driver crashes with OOM.

2. Executor OutOfMemoryError Diagnosis & Fixes

When an executor exhausts its heap or off-heap memory, the cluster manager terminates the container, logging ExecutorLostFailure (exit code 137 / OOMKilled) .

Root Cause: Partition Sizing & High Concurrency :

If a cluster has 8 cores per worker and only 8 partitions for a 100 GB dataset, each task attempts to process ~12.5 GB of data simultaneously within the worker's shared JVM heap.

Remediation :

Increase partition count so that individual tasks process 100 MB–200 MB chunks.

Switch worker node types from compute-optimized VMs ( Standard_F series) to memory-optimized VMs ( Standard_E series, e.g., Standard_E8ds_v5 ).

2. Memory Spill vs. Disk Spill: The Hidden Performance Killer

Spark allocates executor memory into two primary pools: Execution Memory (used for shuffle, joins, sorts, and aggregations) and Storage Memory (used for cached data and broadcast variables).

When execution memory is exhausted during a memory-intensive operation (such as sorting or hashing), Spark evicts data from RAM to disk. This process is called Spill . +-------------------------------------------------------------------------+

  • | SPARK MEMORY SPILL LIFECYCLE |
  • +-------------------------------------------------------------------------+
  • | |
  • | EXECUTOR JVM HEAP |
  • | ┌────────────────────────────────────────────────────────┐ |
  • | │ Spark Memory Pool (Execution + Storage) │ |
  • | │ [ In-Memory Data Structures (Hash Tables, Sort Buffers) ] |
  • | └───────────────────────────┬────────────────────────────┘ |
  • | │ Execution Memory Full! |
  • | ▼ |
  • | SPILL (MEMORY): Deserialized Java objects in RAM (~24 GiB) |
  • | │ |
  • | │ Compress & Serialize to Disk |
  • | ▼ |
  • | SPILL (DISK): Compressed binary bytes on local NVMe SSD (~6 GiB) |
  • | |
  • +-------------------------------------------------------------------------+
  • Spill (Memory) vs. Spill (Disk) Explained

Metric in Spark UI Definition Performance Impact

Spill (Memory) The size of the evicted data as it existed in deserialized memory within the JVM heap before spilling. High number indicates large in-memory object overhead (e.g., Scala/Java object headers and pointer graphs).

Spill (Disk) The actual size of the data written to local worker NVMe/SSD storage after serialization and block compression (LZ4/Snappy). Indicates heavy disk I/O, CPU serialization overhead, and future read penalty when reading spilled blocks back into RAM.

Exam Tip : Spill (Memory) is almost always significantly larger than Spill (Disk) (often 3x to 5x larger) because in-memory JVM objects consume far more space than compressed, serialized on-disk bytes.

Eliminating Spill in Production Pipelines

Increase Shuffle Partition Count : Ensure partitions are small enough (< 200 MB) to fit entirely within execution memory.

Enable AQE Partition Coalescing & Skew Join : Dynamically handles partition sizing at runtime.

Upsize Worker RAM : Utilize memory-optimized Azure VM series ( Standard_E family).

Prune Unused Columns : Drop unnecessary columns before joins and aggregations to reduce in-memory row widths.

3. Data Skew Diagnosis & Remediation via Salting

Data skew occurs when data is distributed unevenly across partitions due to high cardinality imbalances in join or grouping keys (for example, millions of web click events with user_id = NULL or store_id = 9999 ).

Remediation 1: Manual Key Salting

When Adaptive Query Execution (AQE) is disabled or unable to resolve extreme skew, data engineers implement Key Salting . Salting adds a pseudo-random integer ( 0 to N-1 ) to the skewed key of the large DataFrame and replicates the matching key on the smaller DataFrame across all N values using an explode operation. # Manual Key Salting Implementation in PySpark

from pyspark.sql import functions as F

SALT_FACTOR = 4 # Split skewed keys into 4 distinct sub-buckets

# Step 1: Add a random salt (0..3) to the large skewed DataFrame

large_skewed_df = large_df.withColumn("salt_key", F.concat(F.col("join_key"), F.lit("_"), (F.rand() * SALT_FACTOR).cast("int")))

  • # Step 2: Replicate the lookup DataFrame by exploding an array of [0, 1, 2, 3]
  • small_replicated_df = small_df.withColumn("salt_array", F.array([F.lit(i) for i in range(SALT_FACTOR)])) \
  • .withColumn("salt_val", F.explode("salt_array")) \
  • .withColumn("salt_key", F.concat(F.col("join_key"), F.lit("_"), F.col("salt_val")))
  • # Step 3: Execute the join on the salted key (Distributes identical join keys across 4 tasks!)
  • balanced_joined_df = large_skewed_df.join(small_replicated_df, on="salt_key", how="inner") \

.drop("salt_key", "salt_array", "salt_val") +-------------------------------------------------------------------------+

  • | KEY SALTING ARCHITECTURE |
  • +-------------------------------------------------------------------------+
  • | |
  • | WITHOUT SALTING (100% of Key 'A' goes to Partition 0): |
  • | Large DF: [A, A, A, A, A, A, A, A] ===> [ Task 0: 8 rows (STRAGGLER) ] |
  • | Small DF: [A] ===> [ Task 0: 1 row ] |
  • | |
  • | WITH SALTING (Key 'A' salted with random 0..3): |
  • | Large DF: [A_0, A_1, A_2, A_3, A_0, A_1, A_2, A_3] |
  • | Small DF: Exploded to [A_0, A_1, A_2, A_3] |
  • | |
  • | Task 0: [A_0, A_0] joins [A_0] (2 rows) -> Perfectly Balanced! |
  • | Task 1: [A_1, A_1] joins [A_1] (2 rows) -> Perfectly Balanced! |
  • | Task 2: [A_2, A_2] joins [A_2] (2 rows) -> Perfectly Balanced! |
  • | Task 3: [A_3, A_3] joins [A_3] (2 rows) -> Perfectly Balanced! |
  • +-------------------------------------------------------------------------+
  • 4. Azure Spot VM Evictions & Network Fetch Failures

Azure Databricks allows worker nodes to run on Azure Spot Virtual Machines to reduce compute costs by up to 80%. However, Azure can evict Spot VMs at any time when compute capacity is reclaimed.

Spot Eviction Failure Cascades & FetchFailedException

Executor Loss : When Azure evicts a Spot VM, the executor running on that VM terminates abruptly.

Shuffle Fetch Failure : If the evicted executor was hosting intermediate shuffle files on its local disk, downstream tasks running on other workers fail when attempting to read those shuffle files over the network ( org.apache.spark.shuffle.FetchFailedException ).

Automatic Stage Resubmission : Spark handles FetchFailedException natively: the DAG scheduler marks the lost shuffle data as missing and automatically resubmits the upstream stage to recompute the missing partitions. +-------------------------------------------------------------------------+

  • | SPOT EVICTION & STAGE RETRY WORKFLOW |
  • +-------------------------------------------------------------------------+
  • | |
  • | 1. Worker VM 2 (Spot) Evicted by Azure! |
  • | 2. Downstream Task on Worker 1 attempts to read Shuffle Block from VM 2|
  • | 3. Network Fetch Fails -> Spark throws FetchFailedException |
  • | 4. DAG Scheduler marks Stage 0 Shuffle Output as missing |
  • | 5. Spark resubmits missing tasks of Stage 0 on healthy Worker 1 & 3 |
  • | 6. Downstream Stage 1 resumes successfully without job failure |
  • +-------------------------------------------------------------------------+

Production Best Practices for Spot Compute:

Never Run Driver Nodes on Spot VMs : If the driver VM is evicted, the entire Spark application terminates immediately and cannot recover. Azure Databricks enforces this by placing the Driver on an On-Demand VM by default.

Use Spot Fallback : Enable Databricks Spot Fallback so that if Azure Spot capacity is exhausted, worker nodes automatically provision as On-Demand VMs to prevent job failure.

Baseline On-Demand + Burst Spot : Configure clusters with a minimum count of On-Demand workers to guarantee baseline progress, using Spot VMs only for autoscaling burst capacity.

Loading diagram... Data Skew Resolution: Standard Shuffle vs Salting vs AQE