Study guide

6.1 Automated Data Lineage Tracking in Catalog Explorer

Microsoft Certified: Azure Databricks Data Engineer Associate DP-750

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Azure Databricks Architecture & Compute Selection

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

2 Chapter 2: Compute Configuration, Performance & Cost Management

2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup

3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown

4 Chapter 4: Unity Catalog Privilege Model & Access Control

4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation

5 Chapter 5: Advanced Data Security, Masking & ABAC

5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks

6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing

6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming

7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment

10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables

10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs

11 Chapter 11: Workflow Orchestration with Lakeflow Jobs

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles

12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines

13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning

13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

6.1 Automated Data Lineage Tracking in Catalog Explorer

Key Takeaways Explain This Topic AI

Unity Catalog automatically captures end-to-end data lineage at both table and column granularity across SQL queries, Python/Scala/R notebooks, Lakeflow Jobs, and BI dashboards without manual instrumentation.

Lineage graphs track provenance bidirectionally: upstream root-cause analysis traces data back to source ingest tables, while downstream impact analysis reveals which reports, dashboards, and ML models will be affected by schema changes.

Runtime requirements mandate Databricks Runtime 11.3 LTS or higher running in Unity Catalog-enabled compute modes (Shared or Single User); legacy Hive metastore tables, unmanaged files, and ephemeral temp views are excluded from catalog lineage.

Lineage visualization strictly respects Unity Catalog access controls: users require SELECT on the target asset to view its lineage, and upstream/downstream nodes are masked or redacted if the user lacks BROWSE or SELECT privileges.

Unity Catalog persists lineage metadata for 365 days (1 year), accessible via interactive visual graphs in Catalog Explorer, SQL queries against system.lineage tables, or automated REST API endpoints.

Last updated: August 2026

6.1 Automated Data Lineage Tracking in Catalog Explorer

DP-750 Exam Focus: Master Unity Catalog's automated lineage capture architecture, distinguishing between table-level and column-level lineage. Understand runtime prerequisites (DBR 11.3 LTS+, Shared or Single User access modes), permissions required to view and traverse lineage graphs, metadata redaction for unauthorized upstream/downstream assets, the 365-day retention lifecycle, and how to perform root-cause and impact analysis.

1. Architectural Foundations of Unity Catalog Lineage

In enterprise data lakehouses, establishing data trust, validating regulatory compliance (such as BCBS 239, GDPR, and HIPAA), and diagnosing data quality anomalies require complete visibility into how data originates, transforms, and flows across workloads.

Legacy Apache Spark environments relied on external, third-party governance agents or intrusive code instrumentation (such as custom listeners or manual Spline/Atlas agents) to capture execution telemetry. These approaches suffered from high operational overhead, inconsistent coverage across languages, and significant security vulnerabilities when collecting cluster metadata.

Unity Catalog automated data lineage operates natively at the query planning and execution layer inside Azure Databricks. As Spark parses, optimizes, and executes queries via Catalyst and the Photon engine, Unity Catalog intercepts the logical query plan ( LogicalPlan ), extracts the source and target relations, and maps exact column-level projection and transformation semantics. +-----------------------------------------------------------------------------------+

  • | UNITY CATALOG RUNTIME LINEAGE CAPTURE |
  • +-----------------------------------------------------------------------------------+
  • | |
  • | [ SQL Query / PySpark / Scala Notebook / Lakeflow Job / DLT Pipeline / Dashboard]|
  • | | |
  • | v |
  • | +---------------------------------------+ |
  • | | Catalyst Optimizer & Photon Engine | |
  • | | - Analyzed Logical Plan Extraction | |
  • | | - Column Projection & Expression Map| |
  • | +---------------------------------------+ |
  • | | |
  • | v |
  • | +---------------------------------------+ |
  • | | Unity Catalog Metastore Service | |
  • | | - Aggregates Upstream / Downstream | |
  • | | - Applies RBAC / Permission Masks | |
  • | | - Persists Lineage Graph (365 Days) | |
  • | +---------------------------------------+ |
  • | | |
  • | +----------------------+----------------------+ |
  • | | | |
  • | v v |
  • | +---------------------------+ +---------------------------+ |
  • | | Catalog Explorer UI | | system.lineage Tables | |
  • | | Interactive Visual Graph | | Databricks SQL Analytics | |
  • | +---------------------------+ +---------------------------+ |
  • +-----------------------------------------------------------------------------------+
  • Key Characteristics of Native Lineage Capture

Zero Code Instrumentation: Engineers do not modify existing SQL queries, PySpark scripts, or ETL pipelines. Lineage is captured automatically during standard job and notebook execution.

Polyglot Support: Captures operations written in ANSI SQL, Python (PySpark and Pandas on Spark), Scala, and R.

Real-Time Aggregation: Lineage events are streamed to the Unity Catalog governance plane in near-real-time as tasks complete, instantly reflecting changes in Catalog Explorer.

Centralized Account Scope: Because Unity Catalog spans multiple regional workspaces attached to the same metastore, lineage tracks operations across workspaces (e.g., an ingestion job in a Dev workspace writing to a table queried by a dashboard in a Prod workspace).

2. Table-Level vs. Column-Level Lineage

Unity Catalog captures dependencies at two distinct granularities: Table-Level Lineage and Column-Level Lineage .

Table-Level Lineage

Table-level lineage tracks dependencies between datasets as whole entities. It identifies:

Source Tables & Views: The upstream inputs queried by a SELECT , MERGE , INSERT , COPY INTO , or DataFrame read operation.

Target Tables & Views: The downstream table or view populated or modified by CREATE TABLE AS SELECT (CTAS), INSERT INTO , MERGE INTO , CREATE VIEW , or DataFrame write operations.

External Data Paths: Ingestion from cloud storage locations (ADLS Gen2 URIs via Unity Catalog external locations or volumes). -- Example 1: Table-to-Table CTAS Operation

  • CREATE OR REPLACE TABLE gold_sales.analytics.monthly_revenue AS
  • SELECT
  • date_trunc('month', o.order_date) AS order_month,
  • c.country,
  • SUM(o.net_amount) AS total_revenue
  • FROM silver_sales.orders.cleaned_orders o
  • INNER JOIN silver_crm.customers.dim_customer c
  • ON o.customer_id = c.customer_id
  • GROUP BY 1, 2;

In this query, Unity Catalog records silver_sales.orders.cleaned_orders and silver_crm.customers.dim_customer as upstream parents of gold_sales.analytics.monthly_revenue .

Column-Level Lineage

Column-level lineage provides deep semantic tracking by recording how individual target columns are derived from specific upstream source columns, including transformations and expressions:

Direct Mapping (1-to-1): Direct projections where target column country originates directly from source column c.country .

Composite Expressions (Many-to-1): Transformations combining multiple source columns (e.g., concat(first_name, ' ', last_name) AS full_name ) link full_name to both first_name and last_name .

Aggregations & Functions: Expressions such as SUM(o.net_amount) link total_revenue directly to the net_amount source column in cleaned_orders .

Conditional Logic: CASE WHEN , COALESCE , and scalar UDFs link target columns to all referenced input columns evaluated within the conditional branches.

Source Table Source Column Target Table Target Column Lineage Relationship

silver_sales.orders.cleaned_orders order_date gold_sales.analytics.monthly_revenue order_month Transformed via date_trunc()

silver_sales.orders.cleaned_orders net_amount gold_sales.analytics.monthly_revenue total_revenue Aggregated via SUM()

silver_crm.customers.dim_customer country gold_sales.analytics.monthly_revenue country Direct Projection (Joined)

Exam Tip: Indirect column dependencies—such as columns referenced purely in a WHERE filter clause (e.g., WHERE status = 'COMPLETED' ) without appearing in the SELECT projection—do not create column-level lineage for target output columns, but the underlying table dependency is captured at the table level.

3. End-to-End Asset Provenance and Impact Analysis

Unity Catalog lineage does not stop at tables and columns. It connects the entire data engineering ecosystem, providing end-to-end traceability across compute jobs, orchestration pipelines, user notebooks, and consumption endpoints. END-TO-END PROVENANCE GRAPH

  • [ ADLS Gen2 Ingest ]
  • | (Auto Loader / cloudFiles)
  • v
  • [ Table: bronze.iot.telemetry_raw ]
  • |
  • +---> [ Lakeflow Pipeline Task / Job: 902184 ]
  • v
  • [ Table: silver.iot.telemetry_curated ]
  • |
  • +---> [ Notebook: /Shared/Analytics/DailyMetrics ]
  • v
  • [ Table: gold.iot.device_performance ]
  • |
  • +-----------------------+-----------------------+
  • | |
  • v v
  • [ Databricks SQL Dashboard ] [ MLflow Model: PredictiveMaintenance ]
  • "IoT Fleet Health Executive" Registered in Unity Catalog
  • Upstream Root-Cause Analysis (Backward Lineage)

When an anomaly or data quality defect is detected on a Gold reporting table or Databricks SQL Dashboard, data engineers use backward lineage to trace the issue upstream:

Open the target table in Catalog Explorer and select the Lineage tab.

Expand upstream nodes to trace the exact lineage path back through intermediate Silver tables, Bronze ingestion tables, and the specific Lakeflow Job or notebook that performed each write.

Identify the exact commit or query that introduced corrupted or missing values.

Downstream Impact Analysis (Forward Lineage)

Before modifying a schema (e.g., renaming a column, altering data types, or deprecating a legacy table), data engineers perform forward lineage analysis:

Inspect downstream dependencies connected to the table or column in Catalog Explorer.

Identify all downstream consumers: dependent views, machine learning models, downstream ETL pipelines, Lakeflow Jobs, and Databricks SQL Dashboards.

Proactively notify affected downstream stakeholders or update dependent transformations before releasing schema changes, preventing production job failures.

4. Runtime Prerequisites and Supported Workloads

To ensure automated lineage capture occurs, compute clusters and queries must meet specific environment constraints:

Technical Prerequisites Matrix

  • Feature / Dimension Supported Configuration Unsupported / Non-Capturing Configuration
  • Databricks Runtime (DBR) DBR 11.3 LTS or higher DBR 11.2 or lower
  • Compute Access Modes Shared (Multi-User) or Single User (Assigned) No Isolation Shared (legacy clusters)
  • SQL Warehouses Serverless, Pro, and Classic (all versions) Non-Unity Catalog compute
  • Catalog Types Unity Catalog Managed & External Tables / Views Legacy hive_metastore tables, DBFS root mounts

Languages SQL, Python (PySpark/Pandas API on Spark), Scala, R Pure local OS Python file writes outside Spark/Delta APIs

Ephemeral Objects Materialized Views & Streaming Tables Spark temporary views ( createTempView ) not persisted to UC

Workload Lineage Capture Scenarios

Lakeflow Spark Declarative Pipelines (Delta Live Tables): Lineage is captured automatically between Streaming Tables and Materialized Views defined across pipeline stages.

Delta Sharing: When data is shared via Delta Sharing, Unity Catalog captures the share as an egress endpoint, providing auditability for external data distributions.

Machine Learning Models: Models trained using PySpark/Feature Store tables and registered in Unity Catalog automatically capture input feature lineage.

5. Permissions and Security Governance in Lineage Visualization

Unity Catalog enforces strict security boundaries and access controls when displaying lineage in Catalog Explorer or returning lineage records via APIs.

Permissions Required to View Lineage

To view the lineage tab of any object in Catalog Explorer, a user must possess:

USE CATALOG on the parent catalog.

USE SCHEMA on the parent schema.

SELECT privilege on the specific table or view.

Metadata Masking and Privacy Protection

In enterprise environments, a data engineer might have permission to view a Gold reporting table ( gold.finance.revenue ), but might not possess read permissions on the raw upstream Bronze ingestion table ( bronze.hr.payroll_raw ) that contributed to an aggregated metric.

Unity Catalog handles cross-privilege lineage traversing through Metadata Redaction : USER PERMISSION VIEW

  • [ bronze.hr.payroll_raw ] -----> [ silver.finance.comp ] -----> [ gold.finance.revenue ]
  • (NO SELECT) (WITH SELECT) (WITH SELECT)
  • | | |
  • v v v
  • [ Redacted Node ] [ Visible Lineage Node ] [ Target Lineage Node ]
  • "You do not have "silver.finance.comp" "gold.finance.revenue"
  • permission to view Schema & Columns Visible Full Details Visible
  • this object's metadata"

Redacted Nodes: If a user lacks SELECT or BROWSE permissions on an upstream or downstream object in the graph, Unity Catalog renders the node as Masked / Redacted .

The BROWSE Privilege: The BROWSE privilege allows users to see metadata (such as catalog, schema, and table names) and discover the existence of objects in the lineage graph without granting access to read the underlying row data.

Data Exfiltration Prevention: Column-level lineage graphs hide column names and transformation logic for redacted objects to prevent unauthorized inference of proprietary business logic or sensitive data definitions.

6. Lineage Retention and Programmatic Querying

Lineage metadata is retained in the Unity Catalog metastore for a rolling window of 365 days (1 year) . This retention policy ensures organizations have sufficient historical depth to meet annual governance and auditing mandates.

Accessing Lineage Data

Data engineers can consume lineage through three primary interfaces:

Catalog Explorer UI: High-performance, interactive graphical visualization with node filtering, column expansion, and upstream/downstream depth toggles.

System Tables ( system.lineage ): SQL-queryable Delta tables containing structured lineage records across the entire account (detailed extensively in Section 6.2).

Databricks REST API: The Lineage REST API allows external data catalog tools (such as Microsoft Purview, Collibra, or Alation) to synchronize and export lineage graphs programmatically. # Example REST API Request to fetch table lineage

curl -X GET -H "Authorization: Bearer <token>" \

"https://<databricks-instance>/api/2.0/lineage-tracking/table-lineage?table_name=gold_sales.analytics.monthly_revenue"

Loading diagram... Automated Lineage Graph & Access Control Boundary