Study guide
6.3 Data Retention Policies, Time Travel Security, & Tag Governance
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
6.3 Data Retention Policies, Time Travel Security, & Tag Governance
Key Takeaways Explain This Topic AI
Delta Lake time travel enables querying historical table snapshots by version number or timestamp, powered by immutable JSON commit logs and checkpoint files in the _delta_log directory.
Table retention is governed by two critical table properties: delta.logRetentionDuration (controls transaction log history; default 30 days) and delta.deletedFileRetentionDuration (controls tombstoned Parquet data file retention before VACUUM eligibility; default 7 days).
The VACUUM command permanently removes unreferenced, tombstoned Parquet files older than the retention threshold, reclaiming cloud storage and permanently truncating time travel history prior to the vacuum horizon.
Complying with GDPR/CCPA 'Right to be Forgotten' mandates a two-step hard deletion protocol: executing a DELETE/MERGE operation followed by a VACUUM to physically erase PII from historical Parquet files in ADLS Gen2.
Unity Catalog tags provide a standardized key-value metadata taxonomy for catalogs, schemas, tables, and columns, enabling automated governance classifications, PII discovery, and tag-based policy enforcement.
Last updated: August 2026
6.3 Data Retention Policies, Time Travel Security, & Tag Governance
DP-750 Exam Focus: Understand Delta Lake transaction log retention ( delta.logRetentionDuration ), deleted data file retention ( delta.deletedFileRetentionDuration ), and time travel mechanics. Master the operational and governance consequences of VACUUM , the required protocol for GDPR/CCPA hard data deletion, and how to apply and query Unity Catalog tags for Attribute-Based Access Control (ABAC) and data classification.
1. Delta Lake Time Travel & Transaction Log Mechanics
Delta Lake brings ACID transactional guarantees to cloud object storage (ADLS Gen2) by pairing immutable Parquet data files with an ordered, serialized transaction log located in the _delta_log/ directory.
Every write operation—whether an INSERT , UPDATE , DELETE , MERGE , or OPTIMIZE —creates a new atomic commit JSON file ( 000000.json , 000001.json , etc.). When rows are updated or deleted, Delta Lake does not modify existing Parquet files in place. Instead, it writes new Parquet files containing the modified data and records tombstones (metadata markers) in the transaction log indicating that the older Parquet files are logically obsolete for future transactions. DELTA LAKE SNAPSHOT ISOLATION
- _delta_log/
- +---------------+ +---------------+ +---------------+
- | 000000.json | --> | 000001.json | --> | 000002.json |
- | (Add: file_A) | | (Add: file_B) | | (Remove: file_A| <-- Tombstone Marker
- +---------------+ +---------------+ | Add: file_C) |
- +---------------+
- |
- +--------------------------------------------+--------------------------------------------+
- | Snapshot at Version 1 (file_A, file_B) | Snapshot at Version 2 (file_B, file_C)
- v v
- [ Query Version 1 ] [ Query Version 2 (Current) ]
- Reads: file_A.parquet, file_B.parquet Reads: file_B.parquet, file_C.parquet
- Time Travel Query Syntax
Data engineers can query historical states of any Delta table using version numbers or timestamps: -- 1. Querying table state at a specific commit version
SELECT * FROM prod_sales.curated.orders VERSION AS OF 14;
- - 2. Querying table state at a specific historical timestamp
- SELECT * FROM prod_sales.curated.orders TIMESTAMP AS OF '2026-08-01 12:00:00';
- - 3. PySpark DataFrame Time Travel API
- df_v14 = spark.read.table("prod_sales.curated.orders").option("versionAsOf", 14).load()
- df_ts = spark.read.table("prod_sales.curated.orders").option("timestampAsOf", "2026-08-01T12:00:00Z").load()
- Auditing History with DESCRIBE HISTORY
The DESCRIBE HISTORY command displays the complete chronological log of commits, detailing operations, user identities, cluster IDs, and affected file metrics: DESCRIBE HISTORY prod_sales.curated.orders;
- version timestamp userName operation operationParameters job.jobId
- 15 2026-08-26 10:15:00 etl_sp@corp.com MERGE {"predicate": "[...id = ...id]"} 984120
- 14 2026-08-25 18:30:00 engineer@corp.com OPTIMIZE {"zOrderBy": "["customer_id"]"} NULL
- 13 2026-08-25 04:00:00 etl_sp@corp.com WRITE {"mode": "Append"} 984120
- Restoring Historical Snapshots
If a pipeline erroneously corrupts a table, engineers can roll back the table instantaneously using RESTORE : RESTORE TABLE prod_sales.curated.orders TO VERSION AS OF 13;
Restoring does not delete transaction history; it appends a new commit ( version 16 ) that resets the active snapshot to match version 13.
2. Table Retention Properties: Log vs. File Retention
Delta Lake data retention is governed by two complementary table properties configured in TBLPROPERTIES : delta.logRetentionDuration = Duration transaction log commits are preserved (Default: 30 days) delta.deletedFileRetentionDuration = Duration tombstoned data files are preserved before VACUUM eligibility (Default: 7 days) \begin{aligned}
\text{delta.logRetentionDuration} &= \text{Duration transaction log commits are preserved (Default: 30 days)} \\
\text{delta.deletedFileRetentionDuration} &= \text{Duration tombstoned data files are preserved before VACUUM eligibility (Default: 7 days)}
\end{aligned} delta.logRetentionDuration delta.deletedFileRetentionDuration = Duration transaction log commits are preserved (Default: 30 days) = Duration tombstoned data files are preserved before VACUUM eligibility (Default: 7 days)
- 1. delta.logRetentionDuration
- Default Value: interval 30 days
Function: Controls how long .json commit files and checkpoint files are retained in _delta_log/ before being cleaned up during automatic log compaction.
Impact: Once commit logs older than this threshold are purged, you can no longer query history or time travel past that timestamp, even if physical data files still exist.
- 2. delta.deletedFileRetentionDuration
- Default Value: interval 7 days (168 hours)
Function: Controls how long tombstoned (logically removed or replaced) Parquet files remain in storage before the VACUUM command is permitted to physically delete them.
Safety Margin: The default 7-day threshold ensures that concurrent long-running queries or streaming readers referencing older snapshots do not fail due to missing underlying files. -- Altering retention properties for an enterprise Delta table
- ALTER TABLE prod_sales.curated.orders SET TBLPROPERTIES (
- 'delta.logRetentionDuration' = 'interval 60 days',
- 'delta.deletedFileRetentionDuration' = 'interval 14 days'
- );
- 3. The VACUUM Command & Storage Lifecycle Management
While logical DELETE , UPDATE , and MERGE operations mark older files as unreferenced, they do not delete physical files from Azure Data Lake Storage Gen2. Over time, storage costs increase as obsolete Parquet files accumulate.
The VACUUM command recursively scans the Delta table directory and permanently deletes Parquet data files that are no longer referenced in the latest table snapshot and are older than the specified retention threshold. VACUUM EXECUTION TIMELINE
- Day -10 Day -7 (Retention Threshold) Day 0 (Now)
- ---+------------------------+------------------------------------+---------------->
- | | |
- v v v
- [ File 101.parquet ] [ Threshold Boundary ] [ Active Snapshot ]
- (Tombstoned 10 days ago) (7 Days / 168 Hours) (Currently Referenced Files)
- |
- v
- PERMANENTLY DELETED by: VACUUM prod_sales.curated.orders RETAIN 168 HOURS;
- VACUUM Syntax and Operations -- 1. Dry run: Preview which files will be deleted without removing them
- VACUUM prod_sales.curated.orders RETAIN 168 HOURS DRY RUN;
- - 2. Execute physical vacuum with default retention (168 hours / 7 days)
- VACUUM prod_sales.curated.orders;
- - 3. Execute physical vacuum with explicit 14-day retention
- VACUUM prod_sales.curated.orders RETAIN 336 HOURS;
- Critical Governance & Safety Rules
Irrevocable Time Travel Truncation: Once VACUUM removes tombstoned files, any attempt to time travel back to a version referencing those deleted files fails with a FileNotFoundException .
The Retention Safety Check: Databricks enforces a built-in safety check preventing VACUUM with a retention threshold under 168 hours (7 days). Overriding this requires disabling spark.databricks.delta.vacuum.parallelDelete.enabled or setting spark.databricks.delta.retentionDurationCheck.enabled = false in cluster Spark configuration.
4. GDPR / CCPA Compliance: The "Right to be Forgotten" Protocol
Under data privacy regulations like GDPR (Article 17) and CCPA, organizations must permanently erase personal data (PII) upon consumer request within strict statutory timeframes (e.g., 30 days).
The Problem: Logical Delete vs. Physical Storage
Executing a standard DELETE FROM customers WHERE customer_id = 'CUST_8841' merely marks the row as deleted in the newest snapshot commit. The customer's PII remains completely legible in older Parquet files accessible via time travel, violating compliance mandates. GDPR HARD PURGE COMPLIANCE WORKFLOW
- Step 1: Execute Logical Delete / Anonymization
- +-------------------------------------------------------------------------+
- | DELETE FROM prod_crm.customers.dim_customer WHERE customer_id = '8841'; |
- +-------------------------------------------------------------------------+
- |
- v
- Step 2: Lower File Retention Threshold
- +-------------------------------------------------------------------------+
- | ALTER TABLE prod_crm.customers.dim_customer |
- | SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '0 hours'); |
- +-------------------------------------------------------------------------+
- |
- v
- Step 3: Execute Physical Vacuum (Hard Purge)
- +-------------------------------------------------------------------------+
- | -- Executed on compute with retention check disabled |
- | VACUUM prod_crm.customers.dim_customer RETAIN 0 HOURS; |
- +-------------------------------------------------------------------------+
- |
- v
- Step 4: Restore Enterprise Retention Policy
- +-------------------------------------------------------------------------+
- | ALTER TABLE prod_crm.customers.dim_customer |
- | SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '7 days'); |
- +-------------------------------------------------------------------------+
Exam Scenario: To ensure GDPR compliance for data deletion requests, you must execute the DELETE statement to remove data from the active snapshot AND execute VACUUM to purge tombstoned files from ADLS Gen2 storage.