Study guide
11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables
Key Takeaways Explain This Topic AI
Task dependencies ( depends_on ) define the execution graph, while Run If conditional trigger rules govern whether a task executes based on upstream task status ( ALL_SUCCESS , AT_LEAST_ONE_SUCCESS , NONE_FAILED , ALL_FAILED , AT_LEAST_ONE_FAILED , ALL_DONE ).
Task Values provide a native, lightweight inter-task key-value sharing mechanism in Python/Scala via dbutils.jobs.taskValues.set(key, value) and dbutils.jobs.taskValues.get(taskKey, key, default) without writing temporary files to cloud storage.
Dynamic value references using template syntax {{tasks.<task_name>.values.<key>}} and built-in system variables ( {{job.id}} , {{job.run_id}} , {{start_time.iso_date}} ) enable dynamic parameter passing across heterogeneous task types including SQL queries and JARs.
Parameter precedence follows a strict hierarchy: Runtime Trigger Overrides > Task-Level Parameters > Job-Level Base Parameters.
Combining If/Else Condition tasks with Task Values enables intelligent data-driven routing, such as directing pipeline execution to quarantine workflows or alerting systems when row-count anomalies occur.
Last updated: August 2026
11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables
DP-750 Exam Focus: Master dynamic control flow, state sharing, and parameter management in Lakeflow Jobs. Understand the exact operational mechanics of all six Run If conditional trigger rules, programmatic state sharing with dbutils.jobs.taskValues , dynamic variable templating ( {{tasks.<task_name>.values.<key>}} , {{job.id}} , {{start_time.iso_date}} ), and parameter precedence hierarchies.
1. Task Dependencies & The Run If Execution Engine
In a multi-task DAG, tasks are linked via the depends_on property, which specifies one or more upstream task keys. By default, a downstream task executes only when all direct upstream tasks succeed . However, real-world enterprise pipelines require sophisticated failure recovery, branch convergence, and cleanup routines.
Lakeflow Jobs provides Run If conditional triggers that dictate the exact operational condition under which a downstream task is triggered. +---------------------------------------------------------------------------------------------------------+
- | LAKEFLOW JOBS "RUN IF" TRIGGER ENGINE |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | +-----------------------+ +-----------------------+ |
- | | Upstream Task A | | Upstream Task B | |
- | | (e.g., Ingest POS) | | (e.g., Ingest Web) | |
- | +-----------------------+ +-----------------------+ |
- | \ / |
- | \ / |
- | v v |
- | +-------------------------------------+ |
- | | DOWNSTREAM TASK C | |
- | | | |
- | | Evaluates Selected `Run If` Trigger | |
- | +-------------------------------------+ |
- | | |
- | +-----------------------+-----------------------+-----------------------+ |
- | | | | | |
- | v v v v |
- | [ ALL_SUCCESS ] [ NONE_FAILED ] [ AT_LEAST_ONE_FAILED ] [ ALL_DONE ] |
- | (Standard Flow) (Tolerates Skips) (Alerting & Incident) (Resource Cleanup) |
- +---------------------------------------------------------------------------------------------------------+
- Comprehensive Run If Trigger Reference
| Run If Trigger Condition | Execution Rule & Upstream Dependency Evaluation | Typical Architectural Use Case |
|:---|:---|:---||
| ALL_SUCCESS (Default) | Executes only when every direct upstream task completes with status SUCCESS . If any upstream task fails or is skipped, this task is skipped. | Standard linear or fan-in ETL where all data sources are required. |
| AT_LEAST_ONE_SUCCESS | Executes if at least one direct upstream task completes with SUCCESS . Does not execute if all upstream tasks fail or are skipped. | High-availability fan-in where secondary data feeds provide redundancy. |
| NONE_FAILED | Executes if no direct upstream task failed. Upstream tasks can be SUCCESS , SKIPPED , or excluded by upstream conditions. | Downstream tasks following conditional If/Else branches where one branch was intentionally skipped. |
| ALL_FAILED | Executes only if every direct upstream task terminates with status FAILED . | Specialized disaster recovery, fallback data restoration, or primary/secondary pipeline failure triage. |
| AT_LEAST_ONE_FAILED | Executes if one or more direct upstream tasks terminate with status FAILED . | Automated incident ticket creation (ServiceNow, Jira), Slack error alerting, or emergency resource teardown. |
| ALL_DONE | Executes when all upstream tasks finish execution, regardless of whether they succeeded, failed, or were skipped. | Mandatory final cleanup tasks, audit logging, telemetry reporting, or releasing external distributed locks. |
Exam Tip: If a task follows an If/Else condition where one branch is skipped, setting the downstream convergence task to ALL_SUCCESS will cause it to be skipped! You must use NONE_FAILED to allow the convergence task to execute after conditional branching.
2. Programmatic State Sharing with Task Values ( dbutils.jobs.taskValues )
In traditional architectures, passing small runtime variables (such as row counts, dynamic file paths, anomaly scores, or max timestamps) between tasks required writing temporary JSON or Delta files to ADLS Gen2 storage. This introduced storage I/O latency, required storage cleanup, and created concurrency conflicts.
Task Values provide a high-performance, in-memory, key-value state store managed directly by the Lakeflow Jobs Control Plane. +---------------------------------------------------------------------------------------------------+
- | TASK VALUES INTER-TASK COMMUNICATION |
- +---------------------------------------------------------------------------------------------------+
- | |
- | TASK 1: "validate_orders" (PySpark) |
- | +--------------------------------------------------------------------------------------------+ |
- | | bad_rows = df.filter(col("is_valid") == False).count() | |
- | | dbutils.jobs.taskValues.set(key="quarantine_count", value=bad_rows) | |
- | +--------------------------------------------------------------------------------------------+ |
- | | |
- | (Persisted in Job Control Plane - Max 48 KB) |
- | v |
- | TASK 2: "process_silver" (PySpark) TASK 3: "notify_sql" (Databricks SQL) |
- | +---------------------------------------------+ +----------------------------------------+ |
- | | count = dbutils.jobs.taskValues.get( | | SELECT * FROM silver.orders | |
- | | taskKey="validate_orders", | | WHERE quarantine_records > | |
- | | key="quarantine_count", | | '{{tasks.validate_orders.values. | |
- | | default=0 | | quarantine_count}}'; | |
- | | ) | +----------------------------------------+ |
- | +---------------------------------------------+ |
- +---------------------------------------------------------------------------------------------------+
- Task Values API Syntax & Rules
- Setting a Task Value (Upstream Task): # PySpark Notebook or Python Task
- row_count = df.count()
- max_watermark = "2026-08-26T12:00:00Z"
- # Set task values (Key-Value pairs)
- dbutils.jobs.taskValues.set(key="ingested_rows", value=row_count)
- dbutils.jobs.taskValues.set(key="watermark_timestamp", value=max_watermark)
Retrieving a Task Value Programmatically (Downstream Python/Scala Task): # Downstream PySpark Notebook or Python Task
- ingested_count = dbutils.jobs.taskValues.get(
- taskKey="ingest_task",
- key="ingested_rows",
- default=0,
- debugValue=100 # Used when running notebook interactively outside of a Job
- )
- print(f"Processing {ingested_count} records from upstream task.")
- Task Values Constraints & DP-750 Gotchas
Payload Size Limit : The total size of all task values set by a single task cannot exceed 48 KB (JSON-serialized). Task values are designed for metadata, metrics, and scalar values—never for passing DataFrames or binary payloads.
Data Types : Must be JSON-serializable primitives (integers, floats, strings, booleans, lists, dicts).
Scope : Scoped strictly to the specific job run instance . A task cannot access task values from a prior or concurrent run of the job.
Interactive Execution : When testing a notebook interactively in a workspace, taskValues.get() will fail unless a debugValue parameter is provided.
3. Dynamic Value References & Built-in System Variables
Lakeflow Jobs supports Dynamic Value References using double curly-brace syntax ( {{ ... }} ). This allows injecting upstream task values, job metadata, and temporal parameters directly into SQL queries, Python CLI arguments, notebook base parameters, and webhook notification payloads.
1. Referencing Upstream Task Values in Configuration & SQL
Any task type—including SQL tasks, JAR tasks, and dbt tasks—can consume task values set by upstream Python/Scala tasks without writing custom retrieval code: -- Databricks SQL Query Task consuming an upstream Task Value
- INSERT INTO silver.metrics.pipeline_audit
- VALUES (
- '{{job.id}}',
- '{{job.run_id}}',
- '{{tasks.ingest_raw_events.values.ingested_rows}}',
- CURRENT_TIMESTAMP()
- );
- 2. Built-in Dynamic System Variables Reference
- | Dynamic Variable Syntax | Description & Output Format | Example Value |
- |:---|:---|:---||
- | {{job.id}} | Unique numerical identifier of the Job definition | 984210542389 |
- | {{job.run_id}} | Unique numerical identifier of the specific Job Run | 451289632 |
- | {{job.name}} | Name of the Job | "Enterprise_Medallion_ETL" |
- | {{task.name}} | Name of the currently executing task key | "curate_silver_customers" |
- | {{task.run_id}} | Unique run identifier of the specific task attempt | 874512963 |
| {{task.execution_count}} | Number of times this task has executed in this run (1-based) | 1 (or 2 on retry) |
- | {{start_time.iso_date}} | Job start date formatted in ISO 8601 UTC date | 2026-08-26 |
- | {{start_time.timestamp}} | Job start timestamp formatted in seconds since epoch | 1787745600 |
| {{parent_run_id}} | Run ID of the parent job (if triggered via a Run Job task) | 451289000 | // Example: Passing dynamic variables to a Python Script Task in Job Definition
- {
- "task_key": "process_daily_partition",
- "spark_python_task": {
- "python_file": "/Volumes/main/etl/scripts/daily_transform.py",
- "parameters": [
- "--date", "{{start_time.iso_date}}",
- "--run_id", "{{job.run_id}}",
- "--threshold", "{{tasks.calculate_threshold.values.dynamic_cutoff}}"
- ]
- }
- }
- 4. Parameter Hierarchy & Precedence Rules
Enterprise pipelines often require setting default parameters while allowing ad-hoc backfills or environments to override values at runtime. PARAMETER PRECEDENCE HIERARCHY
- HIGHEST PRECEDENCE ===> [ 1. RUNTIME TRIGGER PARAMETER OVERRIDES ]
- (Passed via API `run-now` or UI "Run with parameters")
- |
- v
- [ 2. TASK-LEVEL PARAMETERS ]
- (Configured inside individual task definition)
- |
- v
- LOWEST PRECEDENCE ===> [ 3. JOB-LEVEL BASE PARAMETERS ]
- (Configured at top-level job definition)
- Parameter Resolution Rules
Job-Level Base Parameters : Defined at the top-level job configuration. These serve as global fallback variables inherited by all tasks in the DAG.
Task-Level Parameters : Defined inside specific task definitions. If a task parameter shares the same key as a job base parameter, the task-level parameter overrides the job-level value for that task only.
Runtime Trigger Overrides : Passed dynamically when triggering a job via the Databricks UI ( "Run now with different parameters" ) or via the REST API ( POST /api/2.1/jobs/run-now with job_parameters ). Runtime overrides take highest precedence , overwriting both job-level and task-level parameter defaults across all tasks. # In Databricks Notebook Task: Accessing Parameters via dbutils.widgets
- dbutils.widgets.text("target_environment", "dev", "Target Deployment Env")
- dbutils.widgets.text("batch_date", "2026-01-01", "Batch Processing Date")
- env = dbutils.widgets.get("target_environment")
- batch_date = dbutils.widgets.get("batch_date")
- print(f"Executing ETL for Environment: {env}, Date: {batch_date}")