Study guide
12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure
Key Takeaways Explain This Topic AI
Databricks Asset Bundles (DAB) provide an Infrastructure-as-Code (IaC) framework to define, package, test, deploy, and manage complete lakehouse projects (Jobs, DLT Pipelines, ML models) as declarative code.
The root manifest file is databricks.yml , which organizes project definitions into bundle , include , variables , resources , and targets blocks.
Resource declarations ( resources: jobs: ... , resources: pipelines: ... ) specify task dependencies, cluster configurations, schedules, permissions, and parameters in version-controlled YAML.
Variables declared under variables enable clean parameterization across environments using the substitution syntax ${var.variable_name} and built-in namespace properties.
The bundle lifecycle runs through databricks bundle validate , deploy , run , and destroy ; a bundle can also be deployed straight through the workspace REST APIs as a service principal, but the CLI is preferred because it tracks which resources the bundle already created in that target.
Last updated: August 2026
12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure
As enterprise lakehouses grow in complexity, managing pipelines, scheduled workflows, cluster configurations, and permissions through the web UI becomes error-prone and unscalable. Infrastructure drift between development, staging, and production environments frequently causes deployment failures.
Databricks Asset Bundles (DAB) is Databricks' official Infrastructure-as-Code (IaC) standard. DAB allows data engineers to define complete data engineering projects—including multi-task Lakeflow Jobs, Delta Live Tables (DLT) declarative pipelines, MLflow experiments, custom Python wheels, and notebooks—as declarative, version-controlled YAML and source files.
1. DAB Architecture & Directory Anatomy
A Databricks Asset Bundle project structures source code, resource configuration manifests, and test suites into a modular directory hierarchy: my_lakehouse_project/
- ├── databricks.yml # Root bundle configuration manifest
- ├── resources/ # Modular resource YAML definitions
- │ ├── sales_ingest_job.yml # Multi-task Lakeflow Job definition
- │ └── telemetry_pipeline.yml # Delta Live Tables pipeline definition
- ├── src/ # Source code (Python, SQL, Wheels)
- │ ├── __init__.py
- │ ├── transforms.py
- │ └── dlt_bronze_silver.py
- ├── tests/ # Unit and integration test suites
- │ ├── conftest.py
- │ └── test_transforms.py
- ├── fixtures/ # Sample datasets for unit testing
- │ └── sample_events.json
└── pyproject.toml # Build configuration for Python wheels +-------------------------------------------------------------------------+
- | DAB COMPILATION & DEPLOYMENT FLOW |
- +-------------------------------------------------------------------------+
- | |
- | 1. LOCAL DEV 2. BUNDLE ENGINE 3. WORKSPACE |
- | +--------------------+ +-------------------+ +----------------+ |
- | | databricks.yml | | CLI parses YAML | | Workspace File | |
- | | resources/*.yml | --> | Interpolates vars | -> | /Users/.. | |
- | | src/*.py | | Builds wheels/pkg | | Jobs / DLT API | |
- | +--------------------+ +-------------------+ +----------------+ |
- | |
- +-------------------------------------------------------------------------+
- 2. The databricks.yml Root Manifest Schema
The databricks.yml file sits at the root of the project. It defines the bundle metadata, file inclusions, variable definitions, and target environments: # Root bundle identifier
bundle:
name: retail_sales_analytics
# Include modular YAML definitions from the resources/ folder
include:
- resources/*.yml
# Global variable declarations with defaults
variables:
catalog_name:
- description: "Target Unity Catalog catalog for Delta tables"
- default: "dev_catalog"
schema_name:
- description: "Target Unity Catalog schema"
- default: "retail"
worker_node_type:
- description: "Azure VM node type for compute clusters"
- default: "Standard_D4ds_v5"
notification_email:
- description: "Email recipient for failure alerts"
- default: "dataops-alerts@corp.com"
# Target environment definitions
targets:
dev:
- mode: development
- default: true
workspace:
host: https://adb-dev.7.azuredatabricks.net
prod:
mode: production
workspace:
- host: https://adb-prod.7.azuredatabricks.net
- root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
variables:
- catalog_name: prod_catalog
- worker_node_type: Standard_D8ds_v5
- Variable Interpolation Syntax
DAB supports dynamic string interpolation within YAML manifests using the ${...} syntax:
- Custom Variables : ${var.catalog_name} , ${var.schema_name}
- Bundle Context : ${bundle.name} , ${bundle.target} , ${bundle.environment}
Workspace Context : ${workspace.current_user.userName} , ${workspace.current_user.short_name} , ${workspace.host}
3. Resource Definitions: Jobs & Delta Live Tables Pipelines
Resource configurations define the cloud assets provisioned in Azure Databricks.
Defining a Multi-Task Job ( resources/sales_ingest_job.yml ) resources:
jobs:
sales_daily_etl_job:
name: "[${bundle.target}] Daily Sales Processing DAG"
schedule:
- quartz_cron_expression: "0 0 4 * * ?"
- timezone_id: "UTC"
- pause_status: UNPAUSED
email_notifications:
on_failure:
- "${var.notification_email}"
job_clusters:
- job_cluster_key: etl_cluster
new_cluster:
- spark_version: "15.4.x-photon-scala2.12"
- node_type_id: "${var.worker_node_type}"
autoscale:
- min_workers: 2
- max_workers: 8
spark_conf:
spark.databricks.delta.preview.enabled: "true"
tasks:
- # Task 1: Bronze Ingestion via Python Script
- task_key: ingest_bronze
- job_cluster_key: etl_cluster
spark_python_task:
python_file: ../src/ingest_bronze.py
parameters:
- "--catalog"
- "${var.catalog_name}"
- "--schema"
- "${var.schema_name}"
- # Task 2: Silver Cleansing (Depends on Bronze Ingestion)
- task_key: transform_silver
depends_on:
- task_key: ingest_bronze
- job_cluster_key: etl_cluster
notebook_task:
notebook_path: ../src/notebooks/cleanse_silver.py
base_parameters:
- catalog: "${var.catalog_name}"
- schema: "${var.schema_name}"
Defining a Delta Live Tables (DLT) Pipeline ( resources/telemetry_pipeline.yml ) resources:
pipelines:
telemetry_dlt_pipeline:
- name: "[${bundle.target}] Clickstream Telemetry Pipeline"
- catalog: "${var.catalog_name}"
- target: "${var.schema_name}"
- continuous: false
- serverless: true
libraries:
- notebook:
path: ../src/dlt_bronze_silver.py
configuration:
- pipelines.autoOptimize.zOrderCols: "customer_id,event_timestamp"
- 4. DAB CLI Commands & Lifecycle Execution
The Databricks CLI provides a suite of bundle commands to manage every stage of the lifecycle: +-------------------------------------------------------------------------+
- | DAB CLI COMMAND EXECUTION LIFECYCLE |
- +-------------------------------------------------------------------------+
- | |
- | 1. VALIDATE 2. DEPLOY 3. RUN & MONITOR |
- | $ databricks bundle $ databricks bundle $ databricks bundle run |
- | validate deploy -t dev -t dev sales_daily_job |
- | | | | |
- | v v v |
- | Checks syntax, Uploads files, Triggers remote run, |
- | schemas & types creates Jobs & DLT streams live output logs |
- +-------------------------------------------------------------------------+
- CLI Command Matrix
Command Purpose Key Flags & Arguments
databricks bundle init Generates a new bundle project from built-in or custom templates (default Python, SQL, or MLOps templates). --template-dir <path>
databricks bundle validate Compiles YAML manifests, checks schema validation against Databricks OpenAPI specs, and verifies variable references without deploying . -t, --target <target>
databricks bundle deploy Builds artifacts (e.g., Python wheels), uploads source files to the target workspace path, and creates/updates Jobs and Pipelines. -t, --target <target> , --force
databricks bundle run Executes a specific Job, Pipeline, or Task defined in the bundle in the target environment. -t, --target <target> <resource_key> , --refresh-all
databricks bundle destroy Tears down and deletes all deployed resources (Jobs, Pipelines, uploaded files) in the target environment. -t, --target <target> , --auto-approve
databricks bundle schema Prints the complete JSON schema for databricks.yml to aid IDE autocompletion and validation. --output <file.json>
- Validating and Deploying via CLI # Step 1: Validate bundle syntax for the development target
- databricks bundle validate -t dev
# Output confirms valid structure:
- # Name: retail_sales_analytics
- # Target: dev
- # Workspace: https://adb-dev.7.azuredatabricks.net
- # Valid: true
- # Step 2: Deploy bundle to development workspace
- databricks bundle deploy -t dev
- # Step 3: Trigger a test run of the daily sales job
- databricks bundle run -t dev sales_daily_etl_job
- 5. Deploying Bundle Resources Through the REST API
The DP-750 blueprint lists two bundle deployment bullets: "Deploy a bundle by using the Azure Databricks command-line interface (CLI)" and "Deploy a bundle by using REST APIs." The CLI above covers the first. The second matters whenever the deploying agent is not a shell - an Azure Function, a Logic App, an external orchestrator, or a governance service that must create resources through an authenticated HTTP call.
How the Two Paths Relate
databricks bundle deploy is itself an API client. It resolves the target, interpolates variables, uploads artifacts to the workspace, and then calls the workspace REST APIs to create or update each resource - Jobs, Pipelines, Model Serving endpoints, and so on. Anything the CLI does, an HTTP client can do directly.
Path Best for Trade-off
CLI ( databricks bundle deploy -t prod ) GitHub Actions, Azure DevOps, any agent with a shell Requires the CLI on the runner
REST API Serverless deployers, custom control planes, platforms with no shell You reimplement target resolution and state management yourself
Authenticating the Call
Production deployments authenticate as a service principal using an OAuth machine-to-machine token, never a personal access token belonging to an engineer. Section 12.3 covers the run_as identity pattern that pairs with this. # 1. Exchange the service principal client credentials for an OAuth token
- TOKEN=$(curl -s -X POST \
- https://adb-prod.7.azuredatabricks.net/oidc/v1/token \
- u "$DATABRICKS_CLIENT_ID:$DATABRICKS_CLIENT_SECRET" \
- d "grant_type=client_credentials&scope=all-apis" | jq -r .access_token)
- # 2. Create or update the job the bundle defines
- curl -s -X POST \
- https://adb-prod.7.azuredatabricks.net/api/2.2/jobs/create \
- H "Authorization: Bearer $TOKEN" \
- H "Content-Type: application/json" \
- -data @resources/daily_sales_job.json
- # 3. Create the declarative pipeline the bundle defines
- curl -s -X POST \
- https://adb-prod.7.azuredatabricks.net/api/2.0/pipelines \
- H "Authorization: Bearer $TOKEN" \
- H "Content-Type: application/json" \
- -data @resources/telemetry_pipeline.json
- Why the CLI Is Still the Default Answer
Deploying resource-by-resource over HTTP gives up what the bundle abstraction provides: the CLI knows which resources the bundle previously created in that target, so a redeploy updates them and a databricks bundle destroy removes exactly that set. A hand-rolled REST deployment must track those identifiers itself, or it will create a duplicate job on every run.
Use the REST path when the environment genuinely cannot run the CLI. Otherwise, invoke the CLI and let it call the APIs for you.
6. Artifact Building & Custom Python Wheels
DAB natively manages Python package compilation. If your project contains custom libraries, DAB can automatically build a .whl package and attach it to job tasks: # Declare artifact build in databricks.yml
artifacts:
core_transforms:
type: whl
path: .
build: "python -m build --wheel"
resources:
jobs:
wheel_processing_job:
name: "[${bundle.target}] Wheel Pipeline"
tasks:
- task_key: execute_wheel
- job_cluster_key: etl_cluster
python_wheel_task:
- package_name: "retail_core_lib"
- entry_point: "process_daily_sales"
parameters:
- "--batch-date=2026-08-26"
libraries: