ExamFinch.com My Study

Study guide

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code

Practice this exam

Databricks Certified Data Engineer Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Developing Code for Data Processing using Python and SQL - Core Spark SQL & PySpark

1.1 Complex PySpark Transformations & DataFrame API 1.2 Advanced Spark SQL Patterns 1.3 Streaming Workloads with Structured Streaming 1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

2 Chapter 2: Developing Code for Data Processing using Python and SQL - Streaming & Advanced Patterns

2.1 Advanced Delta Lake Features 2.2 Custom & Stateful Streaming Operations 2.3 Python & PySpark Code Modularization 2.4 User-Defined Functions (UDFs) & Performance Impact

3 Chapter 3: Data Ingestion & Acquisition

3.1 Auto Loader (cloudFiles) Production Ingestion 3.2 Change Data Capture (CDC) Ingestion Patterns 3.3 Batch & Streaming Connectors

4 Chapter 4: Data Transformation, Cleansing, and Quality

4.1 Data Validation & Quality Frameworks 4.2 Data Cleansing, Deduplication & Standardization 4.3 Silver and Gold Medallion Layer Refinement 4.4 Handling Schema Drift and Evolution

5 Chapter 5: Data Sharing and Federation

5.1 Delta Sharing Architecture & Governance 5.2 Lakehouse Federation

6 Chapter 6: Monitoring and Alerting & Observability

6.1 System Tables & Observability 6.2 Databricks Workflows & Job Monitoring 6.3 Spark UI & Query Execution Diagnostics 6.4 Data Quality & Pipeline Alerting

7 Chapter 7: Cost & Performance Optimisation - Data Layout & Compute

7.1 Data Storage & Layout Optimization 7.2 Compute Choice & Cluster Sizing 7.3 Photon Engine & Query Acceleration

8 Chapter 8: Cost & Performance Optimisation - Skew & Cost Governance

8.1 Partitioning & Data Skew Remediation 8.2 Cost Governance & DBU Optimization

9 Chapter 9: Ensuring Data Security and Compliance

9.1 Unity Catalog Access Control & Least Privilege 9.2 Dynamic Data Masking & Row/Column-Level Security 9.3 Secrets Management & Network Security 9.4 Compliance, Audit & Data Privacy

10 Chapter 10: Data Governance in Unity Catalog

10.1 Unity Catalog Governance Architecture 10.2 Data Lineage & Tagging in Unity Catalog 10.3 Managed vs External Tables & Storage Governance

11 Chapter 11: Debugging and Deploying - Asset Bundles & CI/CD

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code 11.2 CI/CD Automation & Databricks CLI / REST API 11.3 Troubleshooting Pipeline & Job Failures 11.4 Code Versioning & Databricks Git Integration

12 Chapter 12: Data Modelling & Lakehouse Analytics

12.1 Medallion Architecture Design Patterns 12.2 Dimensional Modeling in the Lakehouse 12.3 Analytics & BI Optimization

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code

Key Takeaways Explain This Topic AI

Databricks Asset Bundles (DABs) use a databricks.yml file to define infrastructure as code, supporting structured deployment of jobs, pipelines, and artifacts.

Target environments (e.g., dev, staging, prod) are declared within bundle.yml or databricks.yml to ensure strict separation of configurations across workspaces.

The databricks bundle validate command parses the bundle configuration for syntax errors and structural validity without performing a deployment.

Running databricks bundle deploy synchronizes local files to the target Databricks workspace and updates resource definitions like jobs or DLT pipelines.

Databricks bundles can substitute variables and environment-specific parameters dynamically, allowing DRY (Don't Repeat Yourself) infrastructure configurations.

Last updated: July 2026

Databricks Asset Bundles (DABs) fundamentally shift how data engineering teams manage their Databricks infrastructure, moving from manual UI-based configurations to a robust Infrastructure-as-Code (IaC) methodology. This section explores the structural, operational, and lifecycle aspects of DABs, which are a major focus area for the Databricks Certified Data Engineer Professional exam.

The Anatomy of Databricks Asset Bundles

At its core, a Databricks Asset Bundle is a collection of source files and a centralized configuration file, typically named databricks.yml (or occasionally bundle.yml ), that describes how these files should be deployed and executed in a Databricks workspace. By treating Databricks resources (like Jobs, Delta Live Tables pipelines, and Model Serving endpoints) as code, DABs enable version control, automated testing, and CI/CD integration. This paradigm shift from manual click-ops to programmatic deployment ensures that environments can be easily recreated, audited, and rolled back if necessary. The bundle directory structure usually contains the databricks.yml at the root, a src directory for code (like Python scripts, notebooks, or SQL files), a tests directory for unit and integration testing, and a resources directory for modularized YAML definitions if the project is large.

The databricks.yml Structure

The databricks.yml file is the heart of a DAB. It defines the bundle's name, its target environments, the resources it manages, and any workspace variables. Here is a comprehensive example of a databricks.yml file: bundle:

name: my_etl_project

variables:

catalog_name:

  • description: "The catalog to use for deployment"
  • default: "dev_catalog"

targets:

dev:

default: true

workspace:

host: https://adb-123456789.1.azuredatabricks.net

variables:

catalog_name: "dev_catalog"

prod:

workspace:

host: https://adb-987654321.2.azuredatabricks.net

variables:

catalog_name: "prod_catalog"

resources:

jobs:

daily_etl:

name: "[${bundle.target}] Daily ETL Job"

tasks:

- task_key: process_data

notebook_task:

  • notebook_path: ./src/process.py
  • job_cluster_key: default_cluster

job_clusters:

- job_cluster_key: default_cluster

new_cluster:

  • spark_version: 14.3.x-scala2.12
  • node_type_id: i3.xlarge
  • num_workers: 2

In this example, the bundle dynamically adapts based on the target environment. The ${bundle.target} syntax is a bundle variable substitution that dynamically injects the target name (e.g., dev or prod ) into the job name. Furthermore, Databricks bundles can integrate deeply with Unity Catalog by parameterizing the catalog and schema names, ensuring that development jobs read and write to development catalogs, completely isolated from production data.

Target Environment Definitions

A critical feature of DABs is the ability to define multiple targets . A target maps a logical environment (like dev , staging , or prod ) to a specific Databricks workspace and configuration set. In larger enterprise architectures, each target typically corresponds to an entirely separate Databricks workspace to ensure the blast radius of any misconfiguration is kept strictly to non-production environments.

Environment Isolation

Target Typical Use Case Configuration Characteristics

dev Local development and testing default: true , often uses personal user credentials, points to a dev workspace or specific user folder. Permissions are permissive.

staging Integration testing Pre-production validation, uses service principals, runs against isolated staging catalogs. Performance testing is often done here to validate cluster sizing.

prod Production workloads Strictly uses service principals, locked-down permissions, deployed exclusively via CI/CD, writes to production catalogs. Read-only access for developers.

By switching targets, data engineers can deploy the exact same codebase to different workspaces without modifying the underlying source files, eliminating the "it works on my machine" problem. When the bundle is deployed to the prod target, the Databricks CLI natively uses the prod configuration, including specific service principal credentials, avoiding accidental cross-contamination.

Declaring Resources

DABs support the declaration of various Databricks resources. The most common resources declared in a databricks.yml file are:

Jobs: Scheduled or triggered workflows consisting of multiple tasks (notebooks, Python wheels, SQL scripts). These definitions include cluster sizing, retry policies, and timeout constraints.

Pipelines: Delta Live Tables (DLT) pipelines, specifying the edition (core, pro, advanced) and the path to the pipeline source code, as well as target schemas.

Artifacts: Compiled code (like Python .whl files) that need to be built and uploaded to the workspace before execution.

Model Serving Endpoints: Definitions for serving MLflow models as REST APIs.

Example: Declaring a DLT Pipeline resources:

pipelines:

my_dlt_pipeline:

  • name: "${bundle.target}_dlt_pipeline"
  • development: true # Set to false in prod target

clusters:

  • label: "default"
  • num_workers: 1

libraries:

- notebook:

path: ./src/dlt_pipeline.sql

When declaring DLT pipelines, the development flag is crucial. In a dev target, setting this to true ensures that clusters are reused across pipeline updates, significantly accelerating the development cycle. In a prod target, this must be overridden to false so the pipeline runs in production mode, spinning up fresh, isolated clusters for execution and terminating them upon completion.

The Bundle Lifecycle Commands

The Databricks CLI provides a suite of commands to manage the bundle lifecycle. Understanding these commands and their order of operations is heavily tested on the professional exam.

1. databricks bundle validate

Before deploying, you must ensure your configuration is syntactically correct. The databricks bundle validate command parses the databricks.yml file, resolves variable substitutions, and checks for schema violations. Crucially, this command does not communicate with the Databricks workspace to create or modify resources. It is purely a local validation step, making it ideal for the early stages of a CI/CD pipeline. By running validation on every commit, teams can catch syntax errors long before they attempt to deploy broken configurations.

2. databricks bundle deploy

The databricks bundle deploy command synchronizes your local project state with the target Databricks workspace. Under the hood, this command performs several critical actions:

It builds any declared artifacts (e.g., compiling a Python wheel using standard setuptools or poetry ).

It uploads the local files (notebooks, SQL scripts, built artifacts) to the Databricks workspace (typically into the workspace files or DBFS depending on the configuration).

It calls the Databricks REST API to create or update the declared resources (Jobs, DLT pipelines) to match the state defined in the bundle.

It applies any tag or permission changes specified in the YAML configuration.

By default, if no target is specified via the -t flag, it deploys to the target marked with default: true .

3. databricks bundle run

Once a bundle is deployed, you can trigger its resources using databricks bundle run . This command requires you to specify the resource key you wish to execute. # Runs the daily_etl job in the default (dev) target

databricks bundle run daily_etl

  • # Runs the daily_etl job in the prod target
  • databricks bundle run daily_etl -t prod

This command submits a run request to the Databricks Jobs or DLT API and streams the output/logs back to your local terminal, allowing you to monitor execution without switching to the Databricks UI. It is highly beneficial for executing integration tests as part of a CD pipeline.

Practical Scenario: DRY Configurations and Overrides

Imagine you have a single data processing pipeline that needs to run in development with a small cluster and in production with a massive cluster. Instead of writing two separate JSON definitions, DABs allow you to handle this gracefully using variables and target overrides. variables:

worker_count:

default: 2

targets:

prod:

variables:

worker_count: 10

resources:

jobs:

scalable_job:

job_clusters:

- job_cluster_key: main

new_cluster:

num_workers: ${var.worker_count}

When deployed to dev , the job is provisioned with 2 workers. When deployed to prod , the override takes effect, and the job is provisioned with 10 workers. This variable injection is a powerful feature that ensures parity between environments while accommodating necessary scaling differences. Another common pattern is using bundle variables to swap out instance profiles or IAM roles depending on the environment, allowing a dev job to read from a sandbox S3 bucket while the prod job reads from the secured, production S3 bucket.