3.5 Lakeflow Declarative Pipelines (Delta Live Tables)

Databricks Certified Data Engineer Associate Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction

About the Databricks Certified Data Engineer Associate Exam Study Plan and Strategies

2 Domain 1: Databricks Intelligence Platform (10%)

1.1 The Lakehouse Architecture 1.2 Databricks Workspace and Components 1.3 Delta Lake Fundamentals 1.4 Databricks Repos, Secrets, and Development Tools 1.5 Databricks SQL and the Photon Engine 1.6 Databricks Runtime and Compute Configuration

3 Domain 2: Development and Ingestion (30%)

2.1 Auto Loader (cloudFiles) 2.2 Reading Data with Spark SQL and PySpark 2.3 Writing Data to Delta Tables 2.4 Spark SQL Query Fundamentals 2.5 Complex Data Types and Higher-Order Functions 2.6 User-Defined Functions (UDFs) 2.7 Structured Streaming Fundamentals 2.8 Databricks Notebooks and Code Execution 2.9 Data Extraction from External Sources 2.10 Table Management and Properties

4 Domain 3: Data Processing & Transformations (31%)

3.1 The Medallion Architecture (Bronze, Silver, Gold) 3.2 Delta Lake Optimization: OPTIMIZE, VACUUM, and Liquid Clustering 3.3 Advanced SQL Transformations 3.4 Incremental Data Processing Patterns 3.5 Lakeflow Declarative Pipelines (Delta Live Tables) 3.6 Implementing a Multi-Hop Pipeline 3.7 Slowly Changing Dimensions (SCD) Patterns 3.8 Data Deduplication Strategies

5 Domain 4: Productionizing Data Pipelines (18%)

4.1 Databricks Workflows and Lakeflow Jobs 4.2 Databricks Asset Bundles (Declarative Automation Bundles) 4.3 Pipeline Monitoring, Alerting, and Error Handling 4.4 Testing and Deployment Best Practices 4.5 Pipeline Performance and Cost Optimization 4.6 Designing Idempotent and Recoverable Pipelines

6 Domain 5: Data Governance & Quality (11%)

5.1 Unity Catalog: Architecture and Three-Level Namespace 5.2 Access Control and Permissions 5.3 Data Lineage and Audit Logging 5.4 Delta Sharing 5.5 Lakehouse Federation 5.6 Data Quality Best Practices 5.7 Row-Level and Column-Level Security 5.8 Compliance and PII Data Management

3.5 Lakeflow Declarative Pipelines (Delta Live Tables)

Key Takeaways Explain This Topic AI

Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) is a declarative ETL framework: you declare the desired tables and it manages orchestration, dependencies, retries, and scaling.

Streaming tables ingest append-only/incremental data, while materialized views recompute their full result and are ideal for aggregated gold tables.

Expectations are data-quality constraints that can warn (track), drop invalid rows, or fail the pipeline (ON VIOLATION DROP ROW / FAIL UPDATE).

Pipelines auto-resolve dependencies from the queries and run datasets in the correct order, so you never wire up the DAG manually.

Triggered mode runs once and stops; continuous mode runs nonstop. Development mode reuses a cluster and skips retries; production mode restarts and retries on failure.

Last updated: June 2026

Declarative ETL

Lakeflow Declarative Pipelines — formerly Delta Live Tables (DLT) and sometimes shown as Lakeflow Spark Declarative Pipelines — is a framework for building reliable ETL. The product was renamed when Databricks unified data engineering under Lakeflow (Lakeflow Connect for ingestion, Lakeflow Declarative Pipelines for transformation, and Lakeflow Jobs, formerly Workflows, for orchestration). You will see the new name on the exam but should recognize "DLT" as the same thing.

The core idea is declarative : you write queries describing the desired end state of each table (the WHAT), and the framework figures out the HOW — execution order, cluster management, error handling, retries, and incrementalization. This contrasts with imperative notebooks where you manually orchestrate each step.

Dataset Types

• Dataset Behavior Best for
• Streaming table Append-only, processes new data incrementally Bronze ingestion, silver from streams
• Materialized view Recomputes its full query result on refresh Gold aggregations, dimensions

View Temporary, not persisted to storage Intermediate logic, reuse within a pipeline CREATE OR REFRESH STREAMING TABLE bronze

AS SELECT * FROM read_files('/landing', format => 'json');

• CREATE OR REFRESH MATERIALIZED VIEW daily_sales
• AS SELECT order_date, SUM(amount) AS total FROM silver GROUP BY order_date;

Expectations: Data Quality

Expectations are declarative data-quality constraints attached to a dataset. Each expectation is a boolean condition; the ON VIOLATION action decides what happens to rows that fail it:

• Action Behavior
• (none) — warn Records that fail are kept but counted as violations in metrics
• DROP ROW Invalid records are dropped from the output (but tracked)
• FAIL UPDATE The pipeline update fails immediately on any violation CREATE OR REFRESH STREAMING TABLE silver (
• CONSTRAINT valid_id EXPECT (id IS NOT NULL) ON VIOLATION DROP ROW,
• CONSTRAINT valid_amt EXPECT (amount > 0) ON VIOLATION FAIL UPDATE
• )
• AS SELECT * FROM STREAM(bronze);

Expectation metrics (records passed, dropped, failed) are recorded in the pipeline's event log , giving you observable data-quality tracking without extra code. In Python the equivalent decorators are @dlt.expect , @dlt.expect_or_drop , and @dlt.expect_or_fail .

Automatic Dependency Resolution

You never declare execution order. The framework reads each query, sees that silver selects from bronze and gold selects from silver , and builds the dependency DAG automatically — running bronze first, then silver, then gold, and parallelizing independent branches. Add a new table that reads an existing one and it is slotted into the graph without rewiring anything.

Execution Modes and Environments

Two orthogonal mode choices appear on the exam.

Pipeline execution mode (how often it runs):

Triggered — the pipeline processes available data once and then stops . Use for scheduled batch refreshes.

Continuous — the pipeline runs nonstop , ingesting and updating tables as new data arrives for low-latency streaming.

Development vs Production mode (how it behaves on failure and cluster reuse):

Development — reuses the cluster between runs (faster iteration) and disables automatic retries so errors surface immediately for debugging.

Production — restarts the cluster for each run and retries on transient failures for reliability.

• Triggered Continuous
• Lifecycle Runs once, stops Runs forever
• Latency Batch Near real-time
• Cost Lower (on-demand) Higher (always on)

Key exam distinction: triggered vs continuous controls scheduling/latency , while development vs production controls cluster reuse and retry behavior — they are independent settings, and confusing the two is a classic trap.

Streaming Tables vs Materialized Views in Depth

Choosing between the two persisted dataset types is one of the most tested decisions:

• Streaming table Materialized view
• Processing Incremental, append-only source Recomputes (or incrementally refreshes) full result
• Source reads Uses STREAM() / read_files Standard SELECT over tables
• Reprocessing data Does not reprocess old rows Reflects all current upstream data
• Typical layer Bronze, silver Gold, dimensions
• Handles updates upstream No (append only) Yes (recomputed)

A streaming table is the right pick when the source only appends and you want each row processed once. A materialized view is right when results must reflect updates and deletes in the source, such as a current daily-revenue total — because it recomputes, it naturally incorporates changes a streaming table would miss.

Other framework features

Unity Catalog integration governs pipeline tables with the same permissions and lineage as any UC table.

The event log captures lineage, expectation metrics, and run history for auditing and debugging.

Full refresh can be triggered to rebuild a table from scratch when logic changes, versus the default incremental update.

Pipelines can be authored in SQL or Python , and a single pipeline may mix both languages across its notebooks.

Serverless pipelines let Databricks manage compute autoscaling without you sizing clusters, and they enable features like incremental materialized-view refresh.

Pipeline parameters (configuration values) let one pipeline definition run against different paths or environments, supporting dev/staging/prod promotion without code changes.