ExamFinch.com My Study

Study guide

2.1 Advanced Delta Lake Features

Practice this exam

Databricks Certified Data Engineer Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Developing Code for Data Processing using Python and SQL - Core Spark SQL & PySpark

1.1 Complex PySpark Transformations & DataFrame API 1.2 Advanced Spark SQL Patterns 1.3 Streaming Workloads with Structured Streaming 1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

2 Chapter 2: Developing Code for Data Processing using Python and SQL - Streaming & Advanced Patterns

2.1 Advanced Delta Lake Features 2.2 Custom & Stateful Streaming Operations 2.3 Python & PySpark Code Modularization 2.4 User-Defined Functions (UDFs) & Performance Impact

3 Chapter 3: Data Ingestion & Acquisition

3.1 Auto Loader (cloudFiles) Production Ingestion 3.2 Change Data Capture (CDC) Ingestion Patterns 3.3 Batch & Streaming Connectors

4 Chapter 4: Data Transformation, Cleansing, and Quality

4.1 Data Validation & Quality Frameworks 4.2 Data Cleansing, Deduplication & Standardization 4.3 Silver and Gold Medallion Layer Refinement 4.4 Handling Schema Drift and Evolution

5 Chapter 5: Data Sharing and Federation

5.1 Delta Sharing Architecture & Governance 5.2 Lakehouse Federation

6 Chapter 6: Monitoring and Alerting & Observability

6.1 System Tables & Observability 6.2 Databricks Workflows & Job Monitoring 6.3 Spark UI & Query Execution Diagnostics 6.4 Data Quality & Pipeline Alerting

7 Chapter 7: Cost & Performance Optimisation - Data Layout & Compute

7.1 Data Storage & Layout Optimization 7.2 Compute Choice & Cluster Sizing 7.3 Photon Engine & Query Acceleration

8 Chapter 8: Cost & Performance Optimisation - Skew & Cost Governance

8.1 Partitioning & Data Skew Remediation 8.2 Cost Governance & DBU Optimization

9 Chapter 9: Ensuring Data Security and Compliance

9.1 Unity Catalog Access Control & Least Privilege 9.2 Dynamic Data Masking & Row/Column-Level Security 9.3 Secrets Management & Network Security 9.4 Compliance, Audit & Data Privacy

10 Chapter 10: Data Governance in Unity Catalog

10.1 Unity Catalog Governance Architecture 10.2 Data Lineage & Tagging in Unity Catalog 10.3 Managed vs External Tables & Storage Governance

11 Chapter 11: Debugging and Deploying - Asset Bundles & CI/CD

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code 11.2 CI/CD Automation & Databricks CLI / REST API 11.3 Troubleshooting Pipeline & Job Failures 11.4 Code Versioning & Databricks Git Integration

12 Chapter 12: Data Modelling & Lakehouse Analytics

12.1 Medallion Architecture Design Patterns 12.2 Dimensional Modeling in the Lakehouse 12.3 Analytics & BI Optimization

2.1 Advanced Delta Lake Features

Key Takeaways Explain This Topic AI

Change Data Feed (CDF) tracks row-level changes using _change_type values: update_preimage, update_postimage, insert, and delete.

Delta Time Travel relies on the transaction log to query older versions using VERSION AS OF or TIMESTAMP AS OF syntax, provided the data hasn't been vacuumed.

Liquid Clustering replaces Z-Ordering for newer Delta tables, automatically adjusting data layout on write without requiring OPTIMIZE commands in many cases.

The Delta transaction log (_delta_log) contains JSON commit files and Parquet checkpoint files, which provide ACID guarantees via optimistic concurrency control.

Optimistic Concurrency Control (OCC) enables multiple simultaneous writers by attempting a commit and resolving non-conflicting file changes automatically.

Last updated: July 2026

Delta Lake is the foundation of the Databricks Lakehouse, providing reliability, performance, and advanced features over standard Parquet files. Understanding its advanced features is critical for the Databricks Certified Data Engineer Professional exam, particularly when dealing with complex data pipelines, strict performance requirements, and auditing.

Change Data Feed (CDF)

Change Data Feed (CDF) allows you to track row-level changes between versions of a Delta table. This is extremely useful for propagating incremental updates to downstream tables, performing auditing, or triggering event-based architectures. To use CDF, you must first enable it on the table: ALTER TABLE my_table SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

When CDF is enabled, Delta Lake records metadata about data mutations. You can query these changes using the table_changes function in SQL or readChangeFeed in PySpark. You specify a starting version or timestamp. # PySpark CDF Read

  • cdf_df = spark.read.format("delta") \
  • .option("readChangeFeed", "true") \
  • .option("startingVersion", 5) \
  • .table("my_table")

The resulting DataFrame includes the original table columns plus several metadata columns. The most important is _change_type , which can have four possible values:

insert : Represents a newly inserted row.

delete : Represents a deleted row.

update_preimage : The state of a row before an update.

update_postimage : The state of a row after an update.

By leveraging _change_type , you can write efficient MERGE statements downstream to apply exactly what changed, rather than reprocessing entire datasets. This reduces compute costs and latency significantly in large-scale pipelines. When multiple updates occur on the same row, CDF captures each transition, allowing a complete audit trail of the row's lifecycle.

Time Travel

Delta Time Travel allows you to access historical versions of a table. Every operation on a Delta table creates a new version, and as long as the underlying Parquet files have not been permanently deleted (via the VACUUM command), you can query them.

Time travel is accessed via two primary mechanisms:

  • VERSION AS OF
  • TIMESTAMP AS OF -- Querying by version
  • SELECT * FROM my_table VERSION AS OF 10;
  • - Querying by timestamp
  • SELECT * FROM my_table TIMESTAMP AS OF '2023-10-01 12:00:00';

In PySpark, you use .option("versionAsOf", 10) or .option("timestampAsOf", "2023-10-01 12:00:00") .

Time travel is incredibly useful for:

Auditing : Reconstructing the state of the data at a specific point in time to satisfy compliance requirements.

Rollbacks : Using the RESTORE TABLE command to revert the active state of a table to a previous version if bad data is accidentally ingested.

Machine Learning : Reproducing models by training on the exact same snapshot of data used previously.

Note that VACUUM removes data files no longer referenced by the current version that are older than the retention period (default 7 days). Once vacuumed, you cannot time travel to versions relying on those deleted files. You can configure the retention duration using delta.deletedFileRetentionDuration .

Liquid Clustering vs. Z-Ordering

Data layout significantly impacts query performance. Historically, Databricks relied on Z-Ordering to co-locate related information in the same set of files. Z-Ordering is applied during the OPTIMIZE command and helps reduce the amount of data read during queries (data skipping). OPTIMIZE my_table ZORDER BY (category_id, date);

However, Z-Ordering has limitations: it is compute-intensive, requires manual maintenance, and suffers from diminishing returns if you Z-Order by too many columns.

Liquid Clustering is the modern alternative introduced in newer Databricks Runtime versions. It dynamically adapts data layout without the rigid restructuring of Z-Ordering. It is particularly effective for tables that grow rapidly or have changing query patterns. It automatically adjusts data layout based on clustering keys, making it resilient to data skew and evolving access patterns.

To enable Liquid Clustering, you use CLUSTER BY during table creation or alteration: CREATE TABLE my_table (id INT, date DATE, data STRING)

  • USING DELTA
  • CLUSTER BY (date);

Key differences:

Flexibility : Liquid Clustering allows you to easily add or remove clustering keys using ALTER TABLE ... CLUSTER BY without rewriting the entire table history.

Automation : In many setups, Liquid Clustering can incrementally cluster data on write or via background processes, reducing the need for heavy, periodic OPTIMIZE FULL commands.

Performance : It provides better read performance for high-cardinality columns and handles skew more gracefully than Z-Ordering.

Delta Transaction Log Internals

The magic of Delta Lake lies in its transaction log, stored in the _delta_log directory at the root of the table. This log serves as the single source of truth for the table's state and provides ACID (Atomicity, Consistency, Isolation, Durability) guarantees.

Every transaction (e.g., INSERT, UPDATE, MERGE) results in a new JSON file in the _delta_log directory (e.g., 00000000000000000000.json , 00000000000000000001.json ). These JSON files contain an array of actions, such as:

add : Adding a Parquet file to the table state, including statistical metadata (min, max, null counts) used for data skipping.

remove : Removing a Parquet file from the table state (logically deleting it before vacuuming).

commitInfo : Metadata about who and what caused the transaction, timestamp, and operation metrics.

metaData : Table schema, partition columns, and configuration properties.

To prevent the engine from having to read thousands of JSON files to construct the current state, Delta Lake periodically creates checkpoint files (typically every 10 commits). A checkpoint is a Parquet file that aggregates the state of all previous JSON logs. When reading a table, the engine finds the latest checkpoint, applies any subsequent JSON logs, and determines exactly which Parquet files are currently valid. The checkpointing mechanism ensures that reconstructing table state remains fast regardless of how many transactions the table has processed over its lifetime.

Optimistic Concurrency Control (OCC) Delta Lake uses OCC to handle multiple writers. When a cluster wants to write data:

It reads the current state (e.g., version 10).

It writes new data files to the storage layer.

It attempts to commit version 11.

If another cluster committed version 11 in the meantime, the first cluster checks if the changes conflict. If they don't conflict (e.g., appending to different partitions), Delta automatically resolves the conflict and commits as version 12. If they do conflict, it throws a ConcurrentAppendException or similar error, preserving data integrity. Conflicts are typically resolved at the file level; if two concurrent operations modify the same file, a conflict will occur. By understanding the inner workings of the transaction log, data engineers can design highly concurrent data pipelines that safely write to the same Delta table from multiple jobs simultaneously without corrupting data or violating ACID properties.