Study guide

8.2 ACID Transactions, Concurrency Control, & Conflict Resolution

Microsoft Certified: Azure Databricks Data Engineer Associate DP-750

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Azure Databricks Architecture & Compute Selection

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

2 Chapter 2: Compute Configuration, Performance & Cost Management

2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup

3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown

4 Chapter 4: Unity Catalog Privilege Model & Access Control

4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation

5 Chapter 5: Advanced Data Security, Masking & ABAC

5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks

6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing

6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming

7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment

10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables

10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs

11 Chapter 11: Workflow Orchestration with Lakeflow Jobs

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles

12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines

13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning

13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

8.2 ACID Transactions, Concurrency Control, & Conflict Resolution

Key Takeaways Explain This Topic AI

Delta Lake provides full ACID transactions on cloud object storage using Optimistic Concurrency Control (OCC) and Multi-Version Concurrency Control (MVCC).

The OCC protocol follows three distinct phases: Read (obtain table snapshot version), Write (stage data files to storage), and Validate & Commit (attempt atomic commit and resolve conflicts).

Delta Lake supports two isolation levels: WriteSerializable (default, ensuring writers do not conflict on the same underlying files) and Serializable (strictest ANSI SQL isolation).

When concurrent transactions attempt to modify the same partitions or files, the second transaction fails validation with a ConcurrentModificationException and initiates automatic commit retries.

Partitioning, Liquid Clustering, and fine-grained MERGE predicates minimize file overlap between concurrent writers, drastically reducing concurrency conflicts.

Last updated: August 2026

8.2 ACID Transactions, Concurrency Control, & Conflict Resolution

Enterprise lakehouses frequently support dozens of concurrent data pipelines, streaming micro-batches, and interactive analytical queries operating simultaneously on shared Delta tables. Without robust concurrency control, concurrent write operations on cloud object storage would suffer from race conditions, lost updates, and dirty reads.

Delta Lake solves this by implementing Optimistic Concurrency Control (OCC) coupled with Multi-Version Concurrency Control (MVCC) to provide full ACID transaction guarantees on top of Azure Data Lake Storage Gen2.

1. ACID Guarantees in Delta Lake

Delta Lake enforces all four pillars of ACID transactions directly on cloud object storage:

ACID Property Implementation Mechanism in Delta Lake

Atomicity All file additions ( add ) and deletions ( remove ) are committed as a single atomic JSON file in _delta_log/ . If a job fails mid-write, the staged Parquet files are never referenced in the log and the table remains intact.

Consistency Schema enforcement, NOT NULL constraints, and CHECK constraints are validated before commit. If any invariant fails, the transaction is aborted.

Isolation Readers always query a consistent, immutable point-in-time snapshot (MVCC) without being blocked by concurrent writers. Writers coordinate via Optimistic Concurrency Control (OCC).

Durability Once a commit JSON file is written to ADLS Gen2, the transaction is permanent and resilient to cluster restarts or compute node crashes.

2. Optimistic Concurrency Control (OCC) Workflow

Delta Lake uses Optimistic Concurrency Control, which operates on the assumption that concurrent transactions rarely modify the exact same files or rows. Rather than locking tables upfront with pessimistic locks, transactions proceed optimistically and validate conflicts at the moment of commit. +-------------------------------------------------------------------------+

  • | THE 3-PHASE OCC TRANSACTION PROTOCOL |
  • +-------------------------------------------------------------------------+
  • | |
  • | PHASE 1: READ SNAPSHOT |
  • | - Engine reads _delta_log to obtain the current table version (e.g. v5)|
  • | - Identifies active Parquet files for query processing |
  • | |
  • | PHASE 2: COMPUTE & WRITE (STAGE) |
  • | - Spark executes transformation logic across worker nodes |
  • | - Writes new Parquet files to ADLS Gen2 with unique UUID filenames |
  • | |
  • | PHASE 3: VALIDATE & COMMIT |
  • | - Engine attempts to write commit file 00000000000000000006.json |
  • | - If version 6 does not exist -> Commit SUCCESS |
  • | - If version 6 already committed by another writer -> CONFLICT CHECK |
  • | * If operations are non-conflicting -> Automatically replay & commit |
  • | * If operations conflict -> Throw ConcurrentModificationException |
  • +-------------------------------------------------------------------------+
  • Atomic Commit Mechanics on ADLS Gen2

On ADLS Gen2 (which provides true atomic directory renames and POSIX multi-action file creation via the ABFSS driver), Delta Lake ensures mutual exclusion during Phase 3. Only one writer can successfully write commit 00000000000000000006.json . Any secondary writer attempting the same file name receives a file collision error from storage and enters conflict resolution.

3. Delta Lake Isolation Levels: WriteSerializable vs. Serializable

Delta Lake supports two formal transaction isolation levels, configurable via table properties: ALTER TABLE sales_silver

  • SET TBLPROPERTIES ('delta.isolationLevel' = 'WriteSerializable'); -- or 'Serializable'
  • Comparison of Isolation Levels

Feature / Behavior WriteSerializable (Default) Serializable (Strictest)

Definition Guarantees that the final state of written files is consistent with some serial execution order. Guarantees that the entire sequence of operations (reads + writes) is strictly serializable.

Phantom Reads Allows phantom reads (a transaction may commit even if a concurrent append added rows matching its read predicate). Prevents phantom reads (if a concurrent write adds rows that would match the read filter, the transaction fails).

Append Concurrency Highly concurrent; multiple concurrent INSERT / Append operations never conflict with each other. Concurrent appends may conflict if an operation reads the table and makes decisions based on read state.

Primary Use Case High-throughput data ingestion, append-only streams, Medallion Bronze/Silver ETL pipelines. Strict financial ledgering, inventory reservation systems requiring absolute sequential consistency.

Exam Tip : WriteSerializable is the default isolation level in Azure Databricks because it maximizes write throughput by allowing concurrent append operations to proceed without false-positive conflict failures.

4. Conflict Detection Matrix

When two transactions execute concurrently on the same Delta table, Delta Lake validates whether their file-level and partition-level modifications overlap:

Transaction 1 (Committed First) Transaction 2 (Attempting Commit) Conflict Status ( WriteSerializable ) Explanation / Resolution

Append ( INSERT INTO ) Append ( INSERT INTO ) No Conflict Both transactions simply add new files. Transaction 2 automatically rebases its commit to the next version.

Append ( INSERT INTO ) Update / Delete / Merge Conditional Conflict Conflict occurs only if the files appended by Transaction 1 match the WHERE / ON predicate of Transaction 2. If disjoint, Transaction 2 rebases successfully.

Update / Delete / Merge Append ( INSERT INTO ) No Conflict Transaction 1 modified existing files; Transaction 2 is purely adding new files. Rebase succeeds.

Update / Delete / Merge Update / Delete / Merge Conflict (if overlapping) If both transactions read and attempt to rewrite or remove the same Parquet files , Transaction 2 fails with ConcurrentModificationException .

OPTIMIZE / Compaction Append ( INSERT INTO ) No Conflict OPTIMIZE bin-packs existing files; concurrent appends write new files. Rebase succeeds automatically.

OPTIMIZE / Compaction Update / Delete / Merge Conditional Conflict If an UPDATE modifies a file that OPTIMIZE is currently compacting, the update transaction wins and OPTIMIZE will safely retry or abort.

5. Handling ConcurrentModificationException & Retry Protocols

When a conflict is detected during Phase 3, Delta Lake checks if the conflict is reconcilable : CONFLICT EVALUATION LOGIC

  • [ Conflict Detected at Commit ] ---> ( Did Tx1 touch files read by Tx2? )
  • |
  • +---------------------------+---------------------------+
  • | NO (Files are disjoint) | YES (Overlapping files)
  • v v
  • [ Automatic Rebase & Commit ] [ Automatic Retry Loop ]
  • Update Tx2 start version to v+1 - Roll back staged files
  • Write JSON commit log - Re-execute query logic
  • Return SUCCESS - Exhausted? Throw Exception
  • Automatic Retry Loop Configuration

Delta Lake automatically retries conflicting transactions on the Spark driver up to a configurable maximum count before surfacing an exception to the user application: -- Configure maximum driver commit retry attempts (default: 10 million)

SET spark.databricks.delta.maxCommitAttempts = 100;

If the maximum retry attempts are exhausted, or if the concurrent operation fundamentally invalidates the read dataset, the driver raises: org.apache.spark.sql.delta.ConcurrentModificationException:

This table was modified by a concurrent update. Please try the operation again.

6. Architectural Best Practices to Prevent Concurrency Conflicts

Data engineers must design pipelines to minimize concurrent write collisions:

Use Partitioning or Liquid Clustering : Ensure concurrent writers target disjoint partitions or clustering keys (e.g., pipeline A writes region = 'US-East' while pipeline B writes region = 'EU-West' ).

Narrow MERGE Predicates : Avoid broad merge predicates (e.g., ON 1=1 ); always specify partition filters and specific primary keys (e.g., ON target.date = source.date AND target.id = source.id ) so Delta Lake only locks necessary files.

Separate Compaction from Real-Time Writers : Schedule OPTIMIZE jobs during lower-traffic windows or rely on Auto-Compaction to minimize file churn.

Group Streaming Micro-Batches : In streaming workloads, increase the trigger interval (e.g., processingTime = '30 seconds' ) to reduce the frequency of rapid-fire commit contention on the transaction log.

Loading diagram... Optimistic Concurrency Control (OCC) Commit & Conflict Resolution Loop