Study guide
11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts
Key Takeaways Explain This Topic AI
Lakeflow Jobs supports three primary trigger mechanisms: Scheduled (Quartz Cron expressions with explicit timezones), File Arrival (event-driven execution on ADLS Gen2 external locations or Unity Catalog Volumes), and Continuous (daemon execution for streaming).
File arrival triggers monitor cloud storage paths (external locations or UC volumes), launching job runs with configurable debounce quiet periods ( wait_after_last_change_seconds ) to prevent triggering runs on partial multi-part file writes.
Continuous execution mode maintains persistent job execution, immediately restarting the pipeline upon completion or termination to support 24/7 Structured Streaming pipelines.
max_concurrent_runs controls job concurrency; setting max_concurrent_runs: 1 enforces serialized execution to prevent race conditions and concurrent write conflicts ( ConcurrentAppendException ) on target Delta tables.
Robust failure handling combines task-level retry policies (retry count, retry on timeout, retry interval, and exponential backoff) with task and job-level timeout limits to prevent hung runs from incurring runaway cloud costs.
Last updated: August 2026
11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts
DP-750 Exam Focus: Master the configuration of automated trigger mechanisms in Lakeflow Jobs: Scheduled (Cron with timezones), File Arrival (event-driven triggers on ADLS Gen2 External Locations and Unity Catalog Volumes), and Continuous execution. Understand concurrency control with max_concurrent_runs , task retry policies with exponential backoff, and hard/soft timeout boundaries for FinOps and SLA protection.
1. Trigger Types in Lakeflow Jobs
Automating pipeline execution requires matching the trigger mechanism to the incoming data frequency, latency requirements, and system architecture. LAKEFLOW JOBS TRIGGER ARCHITECTURE
- +------------------------+ +------------------------+ +------------------------+
- | 1. SCHEDULED TRIGGER | | 2. FILE ARRIVAL TRIGGER| | 3. CONTINUOUS TRIGGER |
- | (Time-Driven) | | (Event-Driven) | | (Stream-Driven) |
- | - Quartz Cron syntax | | - Monitors ADLS / UC Vol| | - 24/7 Perpetual Daemon|
- | - Explicit Timezones | | - Debounce quiet window| | - Auto-restart on exit |
- | - Periodic batch ETL | | - Ingest on file landing| | - Structured Streaming |
- +------------------------+ +------------------------+ +------------------------+
- 1. Scheduled Triggers (Cron)
How It Works : Executes the job periodically based on a standard Quartz Cron expression .
Timezone Awareness : Always specify an explicit timezone_id (e.g., "America/New_York" , "UTC" , "Europe/London" ). Omitting timezones or relying on server defaults can cause pipelines to shift during Daylight Saving Time (DST) transitions.
Pause State : Schedules can be paused ( "pause_status": "PAUSED" ) during maintenance windows or environment freezes without deleting the schedule configuration. // Example: Scheduled Trigger Configuration (Weekdays at 06:00 UTC)
- {
- "schedule": {
- "quartz_cron_expression": "0 0 6 ? * MON-FRI",
- "timezone_id": "UTC",
- "pause_status": "UNPAUSED"
- }
- }
- 2. File Arrival Triggers (Event-Driven)
How It Works : Monitors a specific cloud storage location—such as a Unity Catalog Volume path ( /Volumes/catalog/schema/volume/landing/ ) or an ADLS Gen2 External Location ( abfss://container@account.dfs.core.windows.net/raw/ ).
Automatic Execution : When a new file arrives or an existing file is updated, Lakeflow Jobs automatically initiates a new job run.
Debounce Window ( wait_after_last_change_seconds ) : When upstream systems upload large multi-part files (e.g., a 10 GB parquet batch written across several minutes), triggering immediately on the first byte would corrupt downstream ingestion. The quiet period specifies how long the storage path must experience zero write activity before Databricks considers the arrival complete and launches the job (default: 50 seconds).
Minimum Time Between Triggers ( min_time_between_triggers_seconds ) : Throttles execution frequency to prevent high-velocity file arrivals from triggering thousands of overlapping runs (e.g., minimum 300 seconds between runs).
Privileges Required : The Job execution identity (Service Principal) must have READ FILES on the Unity Catalog Volume or External Location. // Example: File Arrival Trigger on a Unity Catalog Volume
- {
- "file_arrival": {
- "url": "/Volumes/raw_catalog/landing_zone/partner_feeds/",
- "min_time_between_triggers_seconds": 300,
- "wait_after_last_change_seconds": 60
- }
- }
- 3. Continuous Execution Mode
How It Works : The job runs perpetually as a 24/7 daemon. When a run completes, terminates, or encounters a recoverable error, Lakeflow Jobs immediately spins up a new run instance.
Primary Use Case : Long-running Structured Streaming jobs and continuous real-time feature transformation pipelines.
2. Concurrency Control & max_concurrent_runs
When jobs are triggered frequently (via schedules, webhooks, or file arrivals), a new run may trigger while a previous run is still processing. +---------------------------------------------------------------------------------------------------+
- | CONCURRENCY CONTROL: MAX_CONCURRENT_RUNS = 1 |
- +---------------------------------------------------------------------------------------------------+
- | |
- | Time: 08:00 08:30 09:00 09:30 |
- | | | | | |
- | Run 1: [==============================================] (Completes at 08:45) |
- | | |
- | Run 2: * Trigger Fires! |
- | [ QUEUED / WAITING ] -> [==============================] |
- | (Starts at 08:45 once Run 1 finishes) |
- +---------------------------------------------------------------------------------------------------+
- Operational Rules for max_concurrent_runs
max_concurrent_runs: 1 (Default & Gold Standard for Delta Writes) : Ensures that only a single instance of the job executes at any given moment. Subsequent trigger events are placed into a FIFO Queue (up to a workspace queue limit) and execute sequentially.
Preventing Write Conflicts : Multiple concurrent runs appending or merging into the same target Delta table can cause optimistic concurrency control conflicts ( ConcurrentAppendException or ConcurrentModificationException ). Setting max_concurrent_runs: 1 serializes execution and completely eliminates race conditions.
Scaling Concurrency ( max_concurrent_runs > 1 ) : Appropriate for stateless ETL tasks, parameter-driven backfills, or read-only analytical workloads where runs operate on completely disjoint partition slices.
3. Task-Level Retry Policies & Exponential Backoff
Distributed systems inevitably experience transient network blips, API rate-limiting (HTTP 429), or temporary deadlocks on external databases. Lakeflow Jobs allows configuring granular Retry Policies at the task level. EXPONENTIAL BACKOFF RETRY MECHANICS
- [ Task Fails ] ===> Attempt 1 Failed (HTTP 429 Rate Limit)
- |
- +---> Wait: 10,000 ms (10s) ===========> [ Attempt 2 Fails ]
- |
- +---> Wait: 20,000 ms (20s) =======> [ Attempt 3 Fails ]
- |
- +---> Wait: 40,000 ms (40s) ===> [ Attempt 4: SUCCESS! ]
- Retry Policy Parameters
max_retries : The maximum number of retry attempts after the initial execution fails (e.g., 3 ). A value of 0 disables retries.
min_retry_interval_millis : The initial delay before the first retry attempt (e.g., 10000 for 10 seconds).
retry_on_timeout : A boolean flag ( true / false ). If true , the task will retry even if the failure was caused by reaching the task's timeout limit.
Exponential Backoff Formula
When retries are triggered, Azure Databricks applies an exponential backoff formula with jitter to prevent retrying tasks from overwhelming downstream services ( thundering herd problem ): Delay ( n ) = min_retry_interval_millis × 2 ( n − 1 ) \text{Delay}(n) = \text{min\_retry\_interval\_millis} \times 2^{(n-1)} Delay ( n ) = min_retry_interval_millis × 2 ( n − 1 )
- Where n n n represents the retry attempt index (1, 2, 3...). // Example: Robust Task Retry Configuration
- {
- "task_key": "fetch_rest_api_data",
- "max_retries": 3,
- "min_retry_interval_millis": 15000,
- "retry_on_timeout": false,
- "timeout_seconds": 600
- }
- 4. Timeout Governance: Task-Level vs. Job-Level
Unchecked, hanging Spark jobs (e.g., infinite loops in custom Python code, unresolved network sockets, or Cartesian product joins) can run indefinitely on Azure VMs, consuming thousands of dollars in compute costs without making progress. +---------------------------------------------------------------------------------------------------+
- | TIMEOUT GOVERNANCE HIERARCHY |
- +---------------------------------------------------------------------------------------------------+
- | |
- | JOB-LEVEL TIMEOUT (`timeout_seconds = 7200` / 2 Hours) |
- | +---------------------------------------------------------------------------------------------+ |
- | | | |
- | | Task 1: Ingest (Timeout: 1200s) ===> Completed in 300s | |
- | | | |
- | | Task 2: Transform (Timeout: 1800s) ===> HANGS! Terminated at 1800s by Task Timeout! | |
- | | | |
- | | Overall Job Hard Kill: If total DAG wall-clock time reaches 7200s, ENTIRE JOB TERMINATES. | |
- | +---------------------------------------------------------------------------------------------+ |
- +---------------------------------------------------------------------------------------------------+
- 1. Task-Level Timeout ( timeout_seconds )
Scope : Applied to an individual task within the DAG.
Behavior : If the task execution duration exceeds this threshold, Databricks automatically issues a cancel command to the cluster executor and fails the task.
Best Practice : Set task-level timeouts based on historical P99 runtimes plus a reasonable safety buffer (e.g., 2x average duration).
2. Job-Level Timeout ( timeout_seconds )
Scope : Applied to the entire Job Run from start to finish.
Behavior : Provides an absolute safety ceiling for the total wall-clock duration of all tasks, retries, and provisioning steps combined. If reached, all running tasks are immediately aborted and the job state transitions to TIMED_OUT .