Study guide
10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
Key Takeaways Explain This Topic AI
Triggered pipeline mode executes all available data through the DAG once and shuts down compute, optimizing costs for batch and scheduled workloads, whereas Continuous mode keeps compute active for sub-second streaming latency.
Development mode accelerates pipeline authoring by keeping compute clusters active across runs to avoid restart latency and disabling automated task retries so syntax/runtime errors surface immediately.
Production mode enforces enterprise operational rigor by spinning down compute when idle, restarting clusters cleanly per run, and enabling automated retries upon transient infrastructure failures.
Serverless DLT eliminates manual cluster configuration, virtual machine quota management, and driver tuning, delivering instant sub-15-second compute provisioning and intelligent auto-scaling.
The DLT Event Log is a Delta-backed system table queried using the event_log(pipeline_id) table-valued function or Unity Catalog system tables to audit data quality metrics, flow latency, cluster sizing, and lineage.
Last updated: August 2026
10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
DP-750 Exam Focus: Master the operational management, compute provisioning, and telemetry auditing of Delta Live Tables and Lakeflow Declarative Pipelines. Understand the operational trade-offs between Triggered and Continuous execution modes, Development (Dev) and Production (Prod) modes, and the advantages of Serverless DLT compute . Learn how to write SQL queries against the event_log() table-valued function to extract data quality metrics, flow progress, and error diagnostics.
1. Pipeline Execution Modes: Triggered vs. Continuous
When configuring a Delta Live Tables pipeline, data engineers select an Execution Mode that governs how the runtime engine schedules compute and processes incoming data. +---------------------------------------------------------------------------------------------------------+
- | TRIGGERED MODE VS. CONTINUOUS MODE |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | TRIGGERED MODE (Batch / Scheduled) CONTINUOUS MODE (Real-Time Streaming) |
- | - Starts compute on schedule - Compute runs 24/7 continuously |
- | - Ingests all available pending data - Micro-batches execute continuously |
- | - Processes full DAG once - Ingests data with millisecond/second latency |
- | - Shuts down compute automatically - Incurs constant DBU / VM compute cost |
- | - Maximizes cost savings - Maximizes freshness for real-time streaming |
- | |
- +---------------------------------------------------------------------------------------------------------+
- 1. Triggered Execution Mode
Execution Mechanics : When triggered manually or on a cron schedule (e.g., hourly via Databricks Workflows), the pipeline spins up compute, processes all newly arrived data from sources across the entire Directed Acyclic Graph (DAG), commits all table snapshots, and immediately terminates the compute cluster .
Cost Efficiency : Clusters run only for the exact duration of the data processing update. No idle compute costs are incurred between schedules.
Best Use Case : Hourly/daily batch ETL, periodic CDC syncs, dimensional recomputations, and cost-sensitive data warehouse loading.
2. Continuous Execution Mode
Execution Mechanics : The pipeline cluster remains active indefinitely. The runtime continuously polls streaming sources (e.g., Azure Event Hubs, Kafka topics, Auto Loader storage paths) and processes incoming records in ultra-low latency micro-batches.
Latency Profile : End-to-end ingestion latency is minimized (often sub-second to a few seconds).
Cost Profile : Incurs continuous Databricks Unit (DBU) consumption and cloud VM infrastructure charges.
Best Use Case : Mission-critical real-time analytics, continuous IoT sensor monitoring, live fraud detection, and sub-minute operational dashboards.
Execution Mode Comparison
- Operational Attribute Triggered Mode Continuous Mode
- Compute Lifecycle Ephemeral (Spins up, processes, shuts down) Persistent (Runs 24/7 until manually stopped)
- Data Freshness Bounded by schedule interval (e.g., hourly) Continuous / Near real-time (sub-second)
- Cost Profile Pay only for active processing time Continuous compute and DBU billing
- Checkpoints & State Preserved across runs in storage Maintained in memory & persisted to storage
- Primary Workloads Batch ETL, Scheduled Medallion Lakehouses Real-time Streaming, Live Alerts, Real-time BI
- 2. Pipeline Environments: Development vs. Production Mode
In the DLT pipeline settings UI or JSON configuration, engineers can toggle between Development (Dev) and Production (Prod) modes. +---------------------------------------------------------------------------------------------------------+
- | DEVELOPMENT MODE VS. PRODUCTION MODE |
- +---------------------------------------------------------------------------------------------------------+
- | DEVELOPMENT MODE (Dev) PRODUCTION MODE (Prod) |
- | - Cluster Reuse: Kept active across updates - Cluster Restart: Fresh cluster or shutdown |
- | - Retries: Disabled (fails immediately on error) - Retries: Automatic retries on transient errors |
- | - Goal: Rapid iterative code/query testing - Goal: Enterprise SLA, stability, and cost governance|
- +---------------------------------------------------------------------------------------------------------+
- 1. Development Mode ( development: true )
Cluster Reuse : The driver and worker compute cluster is kept running even after a pipeline update completes. When you modify a SQL or Python file and click Start , the pipeline reuses the existing cluster, eliminating the 3–5 minute VM provisioning delay .
Immediate Error Propagation : Automated retries on task failure are disabled. When a syntax error, type mismatch, or schema failure occurs, the engine halts immediately and displays the stack trace in the UI for interactive debugging.
Manual Triggering : Pipelines in Dev mode do not execute on automated schedules.
2. Production Mode ( development: false )
Cluster Lifecycle : Compute clusters are automatically spun down upon completion of Triggered updates, eliminating idle cloud costs.
Automated Fault Tolerance : The engine automatically retries failed tasks and updates upon encountering transient infrastructure failures (e.g., Azure VM spot evictions or transient network timeouts).
Strict Governance : Locks target tables to production schemas in Unity Catalog and disallows ad-hoc manual state overrides.
3. Serverless Compute for Delta Live Tables
Traditionally, configuring DLT pipelines required data engineers to configure Azure VM worker types (e.g., Standard_D4ds_v5 ), driver node sizes, min/max worker auto-scaling thresholds, and Databricks Runtime (DBR) versions.
Serverless DLT replaces manual infrastructure management with fully managed, instant compute hosted within the Azure Databricks serverless compute plane.
Architectural Advantages of Serverless DLT
Instant Cluster Startup : Serverless compute provisions in under 15 seconds (compared to 3–7 minutes for classic Azure VM provisioning), dramatically accelerating development iterations and scheduled batch jobs.
Intelligent Dynamic Auto-Scaling : The serverless engine continuously monitors pipeline CPU utilization, memory pressure, and streaming queue backlog, scaling worker resources up or down dynamically at the task level rather than VM level.
Zero VM Quota & Subscription Overhead : Computes execute within the secure Databricks compute plane, eliminating Azure subscription core quota bottlenecks, VNet peering configurations, and Azure subnet exhaustion issues.
Automated Runtime Upgrades & Security Patching : Runtimes and security patches are applied transparently without requiring manual DBR version migrations. // Example: DLT Pipeline Configuration JSON enabling Serverless
- {
- "id": "c9a221f4-7e82-4b2a-89a1-5d93b91a7420",
- "name": "Production_Financial_Ingest",
- "serverless": true,
- "development": false,
- "continuous": false,
- "catalog": "prod_catalog",
- "target": "finance_silver",
- "libraries": [
- {
- "notebook": {
- "path": "/Workspace/Pipelines/financial_transformations"
- }
- }
- ]
- }
- 4. Querying the Pipeline Event Log
Every Delta Live Tables pipeline automatically records comprehensive operational telemetry, execution statistics, data quality metrics, and cluster lifecycle events into a managed, append-only Delta table known as the Event Log .
Querying the Event Log via SQL Table-Valued Function
Data engineers can query the event log directly in Databricks SQL or notebooks using the built-in event_log() table-valued function: -- Query the full event log for a specific pipeline
- SELECT *
- FROM event_log("c9a221f4-7e82-4b2a-89a1-5d93b91a7420")
- ORDER BY timestamp DESC;
- Anatomy of Event Log Schema
Column Data Type Description
id STRING Unique identifier for the discrete log entry.
timestamp TIMESTAMP Exact UTC timestamp of the recorded event.
sequence STRUCT Monotonically increasing event ordering sequence.
origin STRUCT Metadata identifying pipeline ID, update ID, flow ID, and dataset name.
event_type STRING Type of event (e.g., flow_progress , user_action , planning_information , crane_alert ).
message STRING Human-readable log message or status description.
level STRING Severity level: INFO , WARN , ERROR , or METRICS .
details STRING (JSON) Structured JSON payload containing data quality expectation metrics, flow execution duration, row counts, and error stack traces.
5. Deep SQL Queries for Pipeline Observability
Data quality metrics and execution statistics reside in the structured JSON payload of the details column when event_type = 'flow_progress' .
1. Extracting Data Quality & Expectation Metrics
To monitor how many records passed, failed, or were dropped by each expectation across pipeline datasets: SELECT
- timestamp,
- origin.flow_name AS dataset_name,
- expectation_metric.key AS expectation_name,
- CAST(expectation_metric.value:passed_records AS BIGINT) AS passed_records,
- CAST(expectation_metric.value:failed_records AS BIGINT) AS failed_records,
- ROUND(
- CAST(expectation_metric.value:failed_records AS DOUBLE) /
NULLIF(CAST(expectation_metric.value:passed_records AS DOUBLE) + CAST(expectation_metric.value:failed_records AS DOUBLE), 0) * 100,
- 2
- ) AS failure_percentage
- FROM event_log("c9a221f4-7e82-4b2a-89a1-5d93b91a7420"),
LATERAL EXPLODE(FROM_JSON(details:flow_progress:data_quality:expectations, 'MAP<STRING, STRUCT<passed_records: BIGINT, failed_records: BIGINT>>')) AS expectation_metric
- WHERE event_type = 'flow_progress'
- ORDER BY timestamp DESC;
- 2. Auditing Pipeline Execution Latency and Row Throughput
To track processing duration, input row counts, and output throughput across each individual flow in the DAG: SELECT
- timestamp,
- origin.update_id,
- origin.flow_name AS table_name,
- details:flow_progress:metrics:num_output_rows AS output_rows,
- details:flow_progress:data_quality:dropped_records AS dropped_rows,
- details:flow_progress:status AS execution_status
- FROM event_log("c9a221f4-7e82-4b2a-89a1-5d93b91a7420")
- WHERE event_type = 'flow_progress'
- AND origin.flow_name IS NOT NULL
- ORDER BY timestamp DESC;
- 3. Diagnosing Pipeline Errors and Root-Cause Failures
To isolate fatal exceptions, constraint failures ( ON VIOLATION FAIL UPDATE ), or cluster provisioning failures: SELECT
- timestamp,
- origin.flow_name AS failed_dataset,
- message AS error_summary,
- details:error:exception:message AS exception_details,
- details:error:exception:stack_trace AS full_stack_trace
- FROM event_log("c9a221f4-7e82-4b2a-89a1-5d93b91a7420")
- WHERE level = 'ERROR'
- ORDER BY timestamp DESC;