Study guide
8.4 Data Retention, Snapshot Isolation, & VACUUM Operations
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
8.4 Data Retention, Snapshot Isolation, & VACUUM Operations
Key Takeaways Explain This Topic AI
Delta Lake Time Travel allows querying historical table snapshots using VERSION AS OF <version> or TIMESTAMP AS OF <timestamp> syntax.
VACUUM permanently deletes underlying Parquet data files that have been logically removed and are older than the table's retention threshold ( delta.deletedFileRetentionDuration , default 7 days).
The default 7-day safety retention threshold prevents the accidental deletion of files currently being accessed by active, long-running queries or concurrent writing transactions.
Executing VACUUM invalidates Time Travel queries that reference versions older than the retention threshold, returning a FileNotFoundException if the underlying Parquet files are physically purged.
Transaction log retention ( delta.logRetentionDuration , default 30 days) is managed independently from data file retention; log files older than 30 days are automatically pruned during checkpointing.
Last updated: August 2026
8.4 Data Retention, Snapshot Isolation, & VACUUM Operations
Because Delta Lake operates on an immutable, multi-version concurrency control (MVCC) model, updates, deletes, merges, and compaction operations do not overwrite or delete existing Parquet files in place. Instead, they write new Parquet files and record remove actions in the _delta_log/ transaction log.
While this architectural model enables powerful features like Time Travel , point-in-time auditing, and zero-downtime rollback, it also causes unreferenced historical data files to accumulate in cloud storage over time. Data engineers must manage this storage lifecycle using the VACUUM command while understanding its operational constraints and safety guardrails.
1. Delta Lake Time Travel Mechanics
Time Travel enables data engineers and analysts to query historical snapshots of a Delta table exactly as they existed at a specific transaction version or point in time. +-------------------------------------------------------------------------+
- | TIME TRAVEL USE CASES |
- +-------------------------------------------------------------------------+
- | 1. Pipeline Debugging: Compare current output with prior version |
- | 2. ML Reproducibility: Train models against identical historical data |
- | 3. Instant Rollback: Restore table after accidental corrupted write |
- | 4. Regulatory Auditing: Reconstruct regulatory reports at period-end |
- +-------------------------------------------------------------------------+
- Time Travel Syntax
- 1. SQL Version and Timestamp Syntax: -- Query specific version number
- SELECT * FROM sales_silver VERSION AS OF 14;
- - Query exact point in time (ISO 8601 or standard timestamp string)
- SELECT * FROM sales_silver TIMESTAMP AS OF '2026-08-20 14:30:00';
- - Syntax using the @ symbol
- SELECT * FROM sales_silver@v14;
- SELECT * FROM sales_silver@20260820143000000;
- 2. PySpark DataFrame Syntax: # Load specific version
- df_v14 = spark.read.format("delta").option("versionAsOf", 14).table("sales_silver")
- # Load specific timestamp
- df_ts = spark.read.format("delta").option("timestampAsOf", "2026-08-20 14:30:00").table("sales_silver")
- 3. Table Rollback with RESTORE : -- Rollback the table to a known good historical snapshot
- RESTORE TABLE sales_silver TO VERSION AS OF 12;
- RESTORE TABLE sales_silver TO TIMESTAMP AS OF '2026-08-19 00:00:00';
- 2. Physical File Cleanup with the VACUUM Command
When rows are deleted or updated, the old Parquet files remain on ADLS Gen2 storage as "tombstoned" files. The VACUUM command scans the transaction log, identifies all files with remove actions that are older than a retention threshold, and physically deletes them from cloud object storage. -- Remove tombstoned files older than the default retention period (7 days / 168 hours)
VACUUM sales_silver;
- - Explicitly specify retention duration (e.g., retain 14 days / 336 hours)
- VACUUM sales_silver RETAIN 336 HOURS;
- - Perform a Dry Run to preview files that would be deleted without deleting them
- VACUUM sales_silver DRY RUN; VACUUM OPERATIONAL WORKFLOW
- ADLS Gen2 Storage Files Transaction Log Status
- +---------------------------+ +-------------------------------------+
- | File A (Active v12) | <----> | Referenced in latest snapshot: KEEP |
- | File B (Removed at v8) | <----> | Removed 9 days ago: DELETE (older) |
- | File C (Removed at v11) | <----> | Removed 2 days ago: KEEP (< 7 days) |
- | File D (Staged uncommitted| <----> | Unreferenced in log > 7d: DELETE |
- +---------------------------+ +-------------------------------------+
- 3. The 7-Day Safety Retention Guardrail
By default, Delta Lake enforces a 7-day (168-hour) minimum retention threshold for VACUUM . Attempting to run VACUUM with a retention threshold lower than 168 hours will fail with an IllegalArgumentException : java.lang.IllegalArgumentException:
requirement failed: The retention threshold is 168 hours. You are trying to vacuum with 0 hours.
Why the 7-Day Threshold Exists:
Long-Running Queries : If a reader started a 2-hour analytical query on snapshot version 10, physically deleting version 10 files after 30 minutes would cause the query to crash with a FileNotFoundException .
Concurrent Writers : Concurrent streaming or batch jobs staging files or validating OCC commits could have their active temporary files deleted mid-transaction.
Catastrophic Data Loss Prevention : Prevents accidental immediate deletion of entire tables.
Overriding the Retention Guardrail (Emergency / Dev Only)
To force VACUUM with zero or low retention (e.g., for compliance with GDPR "Right to Be Forgotten" or cleaning dev environments), you must explicitly disable the retention check in the Spark session: -- 1. Disable safety retention check in Spark configuration
SET spark.databricks.delta.vacuum.retentionDurationCheck.enabled = false;
- - 2. Execute zero-hour VACUUM to purge all historical data immediately
- VACUUM sales_silver RETAIN 0 HOURS;
Exam Trap & Caution : Disabling retentionDurationCheck and running VACUUM RETAIN 0 HOURS in production is dangerous. It will crash any concurrent readers/writers and permanently destroys all Time Travel history prior to the current snapshot.
4. Log Retention vs. Data File Retention
Delta Lake manages metadata lifecycle and physical data lifecycle using two independent retention properties: +-----------------------------------------------------------------------------------+
- | DATA RETENTION VS. TRANSACTION LOG RETENTION |
- +-----------------------------------------------------------------------------------+
- | PROPERTY: delta.deletedFileRetentionDuration |
- | - Default: 'interval 7 days' |
- | - Controls: Physical Parquet data files marked as removed |
- | - Action: Cleaned up ONLY when VACUUM is explicitly executed |
- | |
- | PROPERTY: delta.logRetentionDuration |
- | - Default: 'interval 30 days' |
- | - Controls: Commit JSON files (00000000000000000000.json) in _delta_log |
- | - Action: Automatically pruned during checkpointing every 10 commits |
- +-----------------------------------------------------------------------------------+
- Setting Table Retention Properties: ALTER TABLE sales_silver SET TBLPROPERTIES (
- 'delta.deletedFileRetentionDuration' = 'interval 14 days',
- 'delta.logRetentionDuration' = 'interval 60 days'
- );
What Happens to Time Travel After VACUUM ?
If you run VACUUM sales_silver RETAIN 168 HOURS (7 days), you can still query the table history metadata using DESCRIBE HISTORY sales_silver (because the JSON logs are kept for 30 days).
However, attempting to query data rows via SELECT * FROM sales_silver VERSION AS OF 2 (where version 2 was created 10 days ago) will fail with a FileNotFoundException because the underlying Parquet payload files were physically deleted.
5. Multi-Threaded Parallel Deletion Performance
On large enterprise tables containing millions of tombstoned files, single-threaded driver deletion during VACUUM can take hours due to cloud object storage REST API latency.
Azure Databricks accelerates VACUUM by distributing file listing and deletion across Spark worker executors: -- Enable parallel executor-based deletion for high-scale VACUUM operations
SET spark.databricks.delta.vacuum.parallelDelete.enabled = true;
When enabled, worker tasks issue concurrent HTTP DELETE calls to ADLS Gen2, reducing multi-million file vacuum times from hours to minutes.