ExamFinch.com My Study

Study guide

2.2 Lineage Inspection & Data Traceability

Practice this exam

Databricks Certified Data Analyst Associate Exam

Track your progress as you read

Finished sections get a check mark.

1 Platform Overview & Unity Catalog Foundations

1.1 Databricks Architecture & Compute Concepts 1.2 Databricks Marketplace & Data Sharing Basics 1.3 Unity Catalog Namespace & Object Hierarchy 1.4 Governed Asset Discovery & System Tables

2 Managing & Cleaning Governed Data

2.1 Dataset Discovery & Asset Tagging 2.2 Lineage Inspection & Data Traceability 2.3 SQL Data Cleaning & Transformation Techniques

3 Data Ingestion & Import Patterns

3.1 Workspace Uploads & Cloud File Ingestion 3.2 Incremental File Ingestion with Auto Loader 3.3 Ingesting Data via Delta Sharing & External Connections

4 Databricks SQL Warehouses & Core Querying

4.1 Databricks SQL Interface & SQL Assistant 4.2 SQL Warehouses Sizing & Management 4.3 Table Creation & Data Modification in Databricks SQL

5 Advanced SQL Authoring & Time Travel

5.1 Advanced Aggregations, Window Functions & Analytics 5.2 Complex Joins, CTEs & Set Operations 5.3 Views, Materialized Views & Delta Time Travel

6 Query Performance Analysis & Photon Engine

6.1 Photon Engine Architecture & Acceleration 6.2 Query History & Execution Profile Inspection 6.3 Delta Lake Audit History & Commit Logs

7 Liquid Clustering, Caching & Performance Troubleshooting

7.1 Caching Hierarchy in Databricks SQL 7.2 Data Layout Optimization: Liquid Clustering vs Partitioning & Z-Ordering 7.3 Troubleshooting Slow & Failing Queries

8 AI/BI Dashboards & Visualization Design

8.1 AI/BI Dashboards Overview & Canvas Layout 8.2 Chart Types & Visualization Selection 8.3 Notebook & SQL Editor Visualizations

9 Dashboard Interactivity, Publishing & Alerts

9.1 Dashboard Parameters & Interactive Filtering 9.2 Publishing, Sharing & Access Control 9.3 Scheduled Refresh, Email Subscriptions & Alerts

10 Developing & Maintaining AI/BI Genie Spaces

10.1 AI/BI Genie Architecture & Space Setup 10.2 Curation via Instructions, Descriptions & Sample Questions 10.3 Trusted Assets & Parameterized Functions 10.4 Monitoring, Benchmarks, Feedback & Maintenance

11 Data Modeling & Medallion Architecture

11.1 Dimensional Modeling Fundamentals 11.2 Medallion Architecture for Analytics 11.3 Constraint Enforcement & Relationships in Unity Catalog

12 Securing Data & Unity Catalog Governance

12.1 Unity Catalog Privilege Model & Grants 12.2 Ownership, Inheritance & Access Control Patterns 12.3 Fine-Grained Security: Row Filters & Column Masks

2.2 Lineage Inspection & Data Traceability

Key Takeaways Explain This Topic AI

Unity Catalog automatically captures runtime table-level and column-level lineage for all SQL queries and Delta Live Tables pipelines without manual instrumentation since Runtime 11.3 LTS.

Lineage data is retained in Unity Catalog system tables (system.access.table_lineage and system.access.column_lineage) for 1 year (365 days).

Column-level lineage traces transformations through CTEs, joins, aggregations, and CREATE TABLE AS SELECT (CTAS) statements down to the source tables and input columns.

Accessing Unity Catalog lineage visual graph in Catalog Explorer requires SELECT privileges on the target table, while viewing lineage details across catalogs requires appropriate schema permissions.

Last updated: July 2026

2.2 Lineage Inspection & Data Traceability

Data lineage provides a comprehensive history of how data flows from origin to destination across an organization's analytics ecosystem. In the Databricks Data Intelligence Platform , data traceability is natively integrated into Unity Catalog . Unity Catalog captures table-level and column-level data lineage automatically during query execution without requiring manual pipeline annotations or third-party agent installations. Understanding lineage inspection is crucial for data analysts to trace data origins, perform impact analyses prior to schema refactoring, and ensure compliance with regulatory audit standards.

Unity Catalog Data Lineage Foundations

Unlike legacy data governance frameworks that rely on manual documentation or external parsing, Unity Catalog's lineage engine operates directly within the Databricks query execution runtime.

Automatic Lineage Capture Engine

Unity Catalog captures real-time data lineage for all SQL queries, Python/Scala DataFrame transformations, Delta Live Tables (DLT) pipelines, Auto Loader jobs, and AI/BI Dashboard queries executed within the workspace since Databricks Runtime 11.3 LTS .

Key operational characteristics of the lineage engine include:

Zero Instrumentation Overhead : Lineage is extracted directly from the logical query execution plan generated by the Catalyst Optimizer, introducing zero execution latency on SQL Warehouses or all-purpose clusters.

Cross-Language Traceability : Lineage captures data flows regardless of whether queries are authored in SQL, Python, Scala, or R.

Granular Operation Tracking : Lineage tracks operations including CREATE TABLE AS SELECT (CTAS), INSERT INTO , MERGE INTO , UPDATE , COPY INTO , and view materializations.

Format Support : Lineage is supported across all managed and external Delta Lake tables, registered views, and external data sources accessible via Unity Catalog foreign catalogs.

Table-Level vs. Column-Level Lineage

Unity Catalog tracks data dependencies at two distinct levels of granularity:

Table-Level Lineage : Maps relationships between source tables/views and target tables/views. For example, if gold_monthly_sales is built by joining silver_orders and silver_customers , table-level lineage records both silver tables as direct upstream parents of gold_monthly_sales .

Column-Level Lineage : Traces individual column derivations through intermediate transformations, aggregations, window functions, and Common Table Expressions (CTEs). If gold_monthly_sales.total_revenue is calculated as SUM(silver_orders.item_price * silver_orders.quantity) , column-level lineage links total_revenue directly to item_price and quantity .

Navigating Visual Lineage in Catalog Explorer

Data analysts can inspect data lineage visually through Catalog Explorer to understand dataset origins and evaluate downstream dependencies.

Interactive Lineage Graph UI

When viewing any table or view in Catalog Explorer, clicking the Lineage tab opens an interactive, DAG-based (Directed Acyclic Graph) visualization. Key features of the visual lineage interface include:

Upstream and Downstream Navigation : The graph displays upstream source datasets on the left and downstream targets (including tables, views, AI/BI dashboards, Genie spaces, and ML models) on the right.

Column Lineage Inspection : Clicking on any column highlights its specific lineage paths across the entire graph, dimming unrelated columns and tables.

Depth Expansion : Users can expand graph nodes to trace data lineage up to 9 hops upstream or downstream.

Asset Metadata Overlay : Hovering over graph nodes reveals table owner, data format, last update timestamp, and active tags.

Impact Analysis for Schema and Pipeline Modifications

Before altering a Gold-layer reporting table or dropping a column, data analysts must perform an impact analysis to prevent breaking downstream production assets.

Impact Analysis Task Catalog Explorer Lineage Workflow Risk Mitigation

Dropping a Column Select column -> View downstream nodes -> Check attached AI/BI Dashboards Prevents breaking active executive dashboard widgets

Renaming a Table View table lineage -> Identify downstream views and DLT pipelines Updates SQL view definitions before altering base table

Upstream Data Bug Select erroneous Gold metric -> Navigate upstream graph to Bronze source Pinpoints raw ingestion files introducing corrupted records

Querying Lineage System Tables for Programmatic Audits

While Catalog Explorer offers an intuitive visual interface, enterprise governance teams often require programmatic access to lineage data. Unity Catalog stores historical lineage in system tables under system.access .

Schema Structure of Lineage System Tables

Lineage events are retained in Unity Catalog system tables for 365 days (1 year) . Analysts can query these tables directly using Databricks SQL.

System Table Description Key Query Columns

system.access.table_lineage Records table-to-table dependencies and query metadata source_table_full_name , target_table_full_name , entity_type , created_by , event_time

system.access.column_lineage Records column-to-column transformation mapping source_column_name , target_column_name , source_table_full_name , target_table_full_name

Auditing Downstream Dependencies with SQL

Data analysts can execute recursive or joined queries against system.access.column_lineage to identify all production tables relying on sensitive raw columns. -- Querying column lineage to find all tables consuming SSN from raw landing

  • SELECT DISTINCT
  • cl.target_table_full_name,
  • cl.target_column_name,
  • cl.entity_type,
  • cl.created_by
  • FROM system.access.column_lineage cl
  • WHERE cl.source_table_full_name = 'raw_landing.hr.employee_onboarding'
  • AND cl.source_column_name = 'social_security_number'
  • ORDER BY cl.target_table_full_name;

To view lineage system tables, analysts must hold SELECT privileges on system.access schemas or be assigned the metastore admin / workspace admin role.

Real-World Scenario: Diagnosing Metric Anomalies in Executive Dashboards

In March 2026, an executive dashboard displays a sudden 40% drop in quarterly recurring revenue (QRR). The lead data analyst is assigned to trace the anomaly back to its root cause.

Visual Lineage Tracing : The analyst opens the target dashboard in Catalog Explorer, clicks the Lineage tab, and identifies the Gold table main.finance.gold_executive_metrics .

Column Lineage Inspection : Clicking on qrr_amount highlights upstream paths leading to silver_subscriptions.monthly_fee and silver_subscriptions.discount_rate .

Upstream Investigation : Expanding upstream lineage reveals that a recent ETL pipeline migration on March 15 replaced silver_subscriptions with an unvalidated staging view ( stg_subscriptions_v2 ) where discount_rate was incorrectly divided by 100 twice.

Resolution : Using lineage inspection, the analyst pinpoints the exact query line and intermediate view causing the metric corruption within 15 minutes, avoiding hours of manual code debugging.