Study guide

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log

Microsoft Certified: Azure Databricks Data Engineer Associate DP-750

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Azure Databricks Architecture & Compute Selection

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

2 Chapter 2: Compute Configuration, Performance & Cost Management

2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup

3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown

4 Chapter 4: Unity Catalog Privilege Model & Access Control

4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation

5 Chapter 5: Advanced Data Security, Masking & ABAC

5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks

6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing

6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming

7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment

10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables

10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs

11 Chapter 11: Workflow Orchestration with Lakeflow Jobs

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles

12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines

13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning

13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log

Key Takeaways Explain This Topic AI

Delta Lake is an open-source, columnar storage layer that brings ACID transactions, scalable metadata handling, and unified batch and streaming data processing to Apache Parquet files on ADLS Gen2.

The _delta_log/ transaction log is the single source of truth, recording atomic commits as monotonically increasing, 20-digit zero-padded JSON files (e.g., 00000000000000000000.json ).

Every 10 commits, Delta Lake automatically generates a compacted .checkpoint.parquet file that consolidates the entire table state, eliminating the need to replay hundreds of historical JSON logs.

Delta Lake stores column-level min/max statistics and null counts directly in the add action of the transaction log (indexing the first 32 columns by default), enabling powerful file skipping without reading Parquet data files.

Schema enforcement prevents data corruption by rejecting writes with mismatched columns or incompatible data types, whereas schema evolution can be explicitly enabled using .option("mergeSchema", "true") or table properties.

Last updated: August 2026

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log

Delta Lake is the core storage technology powering the Azure Databricks Lakehouse architecture. Traditional cloud data lakes built on raw formats (such as raw CSV, JSON, or vanilla Apache Parquet) suffer from significant operational limitations: lack of ACID transactions, inability to handle concurrent append and update operations, painful metadata bottlenecks during directory listings on cloud object storage, and frequent data corruption from failed pipeline jobs.

Delta Lake resolves these challenges by coupling standard Apache Parquet payload files with a structured, transactionally guaranteed metadata layer housed in the _delta_log/ directory. Understanding this underlying storage architecture is essential for designing resilient data engineering pipelines and excelling on the DP-750 exam.

1. Delta Lake Physical Storage Architecture

A Delta Lake table stored in Azure Data Lake Storage Gen2 (ADLS Gen2) consists of two distinct components residing in the table root directory: abfss://container@storageaccount.dfs.core.windows.net/tables/sales/

  • ├── _delta_log/
  • │ ├── 00000000000000000000.json
  • │ ├── 00000000000000000001.json

│ ├── ...

  • │ ├── 00000000000000000010.json
  • │ ├── 00000000000000000010.checkpoint.parquet
  • │ └── _last_checkpoint
  • ├── part-00000-4b2e1f2a-c000.snappy.parquet
  • ├── part-00001-9c3f2e1d-c000.snappy.parquet
  • └── part-00002-7a1b4c8e-c000.snappy.parquet
  • 1. Parquet Payload Data Files

Delta Lake persists raw table rows in standard, immutable Apache Parquet format compressed with Snappy by default (or zstd).

Parquet is a columnar binary format providing efficient column pruning, vectorized decompression, and dictionary encoding.

Parquet files in Delta Lake are append-only and immutable . Updating or deleting a record never modifies an existing Parquet file in place; instead, new Parquet files are written and the transaction log updates pointers to reference the new files while marking the old files as logically removed.

2. The _delta_log/ Transaction Log

The _delta_log/ subdirectory resides at the root of the Delta table and acts as the single source of truth for table state, metadata, and data file references.

Every modification to the table (such as an INSERT , UPDATE , DELETE , MERGE , OPTIMIZE , or schema change) constitutes a discrete, atomic transaction that generates a new JSON commit log file.

Commit files follow a strict zero-padded 20-digit numerical naming convention starting at 00000000000000000000.json .

2. Anatomy of the Transaction Log: Commit Actions

Within each JSON commit file, Delta Lake records an array of atomic metadata actions. If a transaction succeeds, all of its actions are written to the JSON file; if a transaction fails, no commit file is created (guaranteeing atomicity). +-------------------------------------------------------------------------+

  • | DELTA TRANSACTION LOG COMMIT ACTIONS |
  • +-------------------------------------------------------------------------+
  • | 1. add | Registers a newly created Parquet file + stats |
  • | 2. remove | Marks an existing Parquet file as logically deleted|
  • | 3. metaData | Records schema, partition columns, table properties|
  • | 4. protocol | Defines minimum reader/writer protocol versions |
  • | 5. setTransaction | Tracks streaming transaction IDs for idempotency |
  • | 6. commitInfo | Audit info: user, timestamp, operation type, params|
  • +-------------------------------------------------------------------------+
  • Detailed Breakdown of Core Actions
  • 1. add Action

Registers a new Parquet file as part of the active table state. It includes file metadata and embedded column statistics: {

  • "add": {
  • "path": "part-00000-4b2e1f2a-c000.snappy.parquet",
  • "partitionValues": {},
  • "size": 15482910,
  • "modificationTime": 1724673600000,
  • "dataChange": true,

"stats": "{\"numRecords\":100000,\"minValues\":{\"id\":1,\"order_date\":\"2026-08-01\"},\"maxValues\":{\"id\":100000,\"order_date\":\"2026-08-26\"},\"nullCount\":{\"id\":0,\"customer_id\":12}}"

  • }
  • }
  • 2. remove Action
  • Logically deletes a file from the active state without physically removing it from ADLS Gen2 storage: {
  • "remove": {
  • "path": "part-00000-old-file-guid.snappy.parquet",
  • "deletionTimestamp": 1724673600000,
  • "dataChange": true,
  • "extendedFileMetadata": true,
  • "partitionValues": {},
  • "size": 14205800
  • }
  • }
  • 3. metaData Action
  • Defines the table schema, partition columns, table name, and configuration properties: {
  • "metaData": {
  • "id": "c8b321a4-9e32-4d2a-89a1-5d93b91a7420",
  • "format": {"provider": "parquet", "options": {}},

"schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":false},{\"name\":\"amount\",\"type\":\"double\",\"nullable\":true}]}",

  • "partitionColumns": [],
  • "configuration": {"delta.enableChangeDataFeed": "true"},
  • "createdTime": 1724673500000
  • }
  • }
  • 4. commitInfo Action
  • Provides provenance and auditability for table history (viewable via DESCRIBE HISTORY table_name ): {
  • "commitInfo": {
  • "timestamp": 1724673600000,
  • "operation": "MERGE",
  • "operationParameters": {"predicate": "(target.id = source.id)"},
  • "job": {"jobId": "89421", "jobName": "Daily_Ingestion_Job"},
  • "engineInfo": "Databricks-Runtime/15.4.x-photon-scala2.12",
  • "operationMetrics": {"numTargetRowsInserted": "500", "numTargetRowsUpdated": "1200"}
  • }
  • }
  • 3. State Reconstruction & Checkpoint Compaction

When a Spark cluster queries a Delta table, the Spark driver must reconstruct the latest snapshot of the table. A naive log replay would require reading every single JSON commit file from version 0 to version N N N , which would introduce crippling latency for tables with thousands of commits.

The 10-Commit Checkpoint Mechanism

To guarantee fast snapshot reconstruction, Delta Lake creates a checkpoint file every 10 commits :

Trigger : At commit 10 (and every subsequent multiple of 10: 20, 30, 40...), the writer engine reads commits 00000000000000000000.json through 00000000000000000010.json and computes the exact set of active Parquet files by canceling out matched add and remove actions.

Parquet Checkpoint : The driver writes this resolved state into a single compacted Parquet file: 00000000000000000010.checkpoint.parquet .

_last_checkpoint Pointer : Delta Lake updates a lightweight pointer file named _last_checkpoint in _delta_log/ containing JSON metadata pointing to the latest checkpoint version: {"version":10,"size":245,"sizeInBytes":14280,"numOfAddFiles":182} Snapshots at Version 14 Reconstruction:

1. Read `_last_checkpoint` -> Identifies Version 10 Checkpoint.

2. Read `00000000000000000010.checkpoint.parquet` (Loads baseline state of 182 active files).

3. Replay only `00000000000000000011.json` through `00000000000000000014.json`.

4. Result: Table snapshot constructed in milliseconds by scanning only 5 files instead of 15!

Exam Tip : The DP-750 exam tests your knowledge of how checkpoint files improve query planning. Checkpoints prevent driver out-of-memory errors and eliminate slow cloud directory listing bottlenecks by consolidating historical commit logs every 10 transactions.

4. Metadata-Based Data Skipping

When writing new Parquet files, Delta Lake automatically collects and embeds column statistics into the stats field of the add action:

numRecords : Total row count in the file.

minValues : Minimum value for each column in the file.

maxValues : Maximum value for each column in the file.

nullCount : Count of NULL entries per column.

How File Skipping Operates During Query Planning

When an analytical query executes with a filter predicate (e.g., WHERE order_date >= '2026-08-20' AND customer_id = 4501 ), the Spark optimizer evaluates the query predicate against the minValues and maxValues stored in the transaction log. Query Predicate: WHERE order_date >= '2026-08-20'

  • File A Stats: min = '2026-08-01', max = '2026-08-15' --> SKIPPED (Max < 2026-08-20)
  • File B Stats: min = '2026-08-10', max = '2026-08-25' --> READ (Overlap exists)
  • File C Stats: min = '2026-08-21', max = '2026-08-26' --> READ (Min >= 2026-08-20)

Because statistics are stored directly in the _delta_log metadata, Spark can discard 90%+ of Parquet files before reading a single byte of data from storage , dramatically slashing I/O cost and latency.

Configuration of Indexed Columns

By default, Delta Lake collects statistics for the first 32 columns defined in the table schema. For wide tables (e.g., 100+ columns), if frequently filtered columns reside beyond column 32, you can adjust this limit via table properties: ALTER TABLE sales_delta_table

  • SET TBLPROPERTIES ('delta.dataSkippingNumIndexedCols' = '64');
  • 5. Schema Enforcement vs. Schema Evolution

Delta Lake provides robust governance mechanisms to balance data quality integrity against changing business schemas. +-------------------------------------------------------------------------+

  • | SCHEMA ENFORCEMENT VS. SCHEMA EVOLUTION |
  • +-------------------------------------------------------------------------+
  • | SCHEMA ENFORCEMENT (DEFAULT) | SCHEMA EVOLUTION (EXPLICIT) |
  • | - Rejects incoming bad data | - Dynamically adds new columns |
  • | - Throws AnalysisException | - Enabled via mergeSchema option |
  • | - Prevents schema pollution | - Safely handles evolving sources |
  • +-------------------------------------------------------------------------+
  • 1. Schema Enforcement (Schema Validation)

Default Behavior : Delta Lake validates every write operation against the existing table schema registered in metaData .

If an incoming DataFrame contains columns not present in the target schema, or if column data types are incompatible (e.g., attempting to write a StringType into a LongType column), Delta Lake aborts the transaction immediately and throws an AnalysisException .

Benefit : Guarantees downstream Gold tables and BI dashboards never encounter unexpected schema corruptions.

2. Schema Evolution ( mergeSchema )

When schema changes are intentional (such as upstream API additions), Delta Lake allows seamless schema evolution:

  • In PySpark DataFrame Writes: # Explicitly allow schema evolution during append or overwrite
  • df_incoming.write \
  • .format("delta") \
  • .mode("append") \
  • .option("mergeSchema", "true") \
  • .saveAsTable("bronze_telemetry")
  • In Spark SQL Session Configuration: -- Enable automatic schema merging for all subsequent SQL operations
  • SET spark.databricks.delta.schema.autoMerge.enabled = true;
  • In MERGE INTO Upsert Statements: MERGE WITH SCHEMA EVOLUTION INTO target_table AS t
  • USING source_updates AS s
  • ON t.user_id = s.user_id
  • WHEN MATCHED THEN UPDATE SET *
  • WHEN NOT MATCHED THEN INSERT *;
  • Rules of Delta Schema Evolution

Allowed Operations : Adding new nullable columns, adding nested struct fields, and widening types (e.g., ByteType → \rightarrow → ShortType → \rightarrow → IntegerType → \rightarrow → LongType under Type Widening).

Disallowed Operations : Changing incompatible data types (e.g., StringType to DoubleType ), dropping existing columns, or renaming columns (unless Column Mapping is explicitly enabled via delta.columnMapping.mode = 'name' ).

Loading diagram... Delta Lake Architecture and Snapshot State Reconstruction