Study guide
12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides
Key Takeaways Explain This Topic AI
DAB targets block enables clean isolation between development, staging, and production environments within a single, unified codebase.
Setting mode: development automatically prefixes resource names with [dev <username>] , sets isolated workspace paths under /Users/<username>/.bundle/ , and automatically pauses job schedules and continuous triggers.
Setting mode: production enforces strict deployment rules: it requires clean Git working trees, deploys to shared workspace paths ( /Shared/.bundle/ ), activates schedules, and requires deployment as a Service Principal.
Target overrides allow fine-grained customization of compute node types, auto-scale ranges, notification recipients, and Unity Catalog targets per environment.
Deploying production bundles via Microsoft Entra Service Principals with run_as identity ensures jobs execute persistently without relying on individual employee user accounts.
Last updated: August 2026
12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides
Enterprise lakehouse architectures require rigorous isolation between Development, Staging (User Acceptance Testing / Pre-Production), and Production environments. Code executing in development must never write to production Delta tables, consume production compute budgets, or trigger active alerting webhooks.
Databricks Asset Bundles (DAB) solves this through its Target and Environment Model . By defining target blocks and applying target overrides in databricks.yml , data engineers maintain a single source of truth while ensuring environment-specific configurations (such as catalogs, cluster sizes, schedules, and permissions) are applied deterministically during deployment.
1. Development vs. Production Deployment Modes
The mode directive inside a DAB target fundamentally alters how Databricks builds, names, isolates, and executes deployed resources: targets:
dev:
- mode: development # Developer isolation mode
- default: true
prod:
mode: production # Strict enterprise production mode +-------------------------------------------------------------------------+
- | DEVELOPMENT MODE VS. PRODUCTION MODE |
- +-------------------------------------------------------------------------+
- | BEHAVIOR / SETTING | mode: development | mode: production |
- |---------------------------|-----------------------|---------------------|
- | Resource Name Prefix | [dev ${user.short}] | None (Clean name) |
- | Deployment Path | /Users/${user}/... | /Shared/.bundle/... |
- | Job Schedules & Triggers | PAUSED (Automatic) | UNPAUSED (Active) |
- | DLT Continuous Pipelines | PAUSED (Run on-demand)| CONTINUOUS (Active) |
- | Git Working Tree Check | Allows uncommitted | FAILS if dirty |
- | Resource Ownership | Deploying Developer | Service Principal |
- +-------------------------------------------------------------------------+
- 1. mode: development Deep-Dive
When an engineer runs databricks bundle deploy -t dev :
Resource Name Mangling : If a job is defined as name: Daily Ingestion , DAB renames it to [dev alice] Daily Ingestion . This prevents collision when multiple developers deploy the same bundle in a shared development workspace.
Automatic Schedule Pausing : Scheduled cron triggers and continuous DLT pipelines are automatically set to PAUSED to avoid unexpected compute costs and rogue background executions during development.
Path Isolation : Files are deployed to /Users/alice@corp.com/.bundle/retail_sales/dev/files/ .
Dirty Git Tolerance : Developers can deploy local edits and uncommitted branch code to iterate quickly.
2. mode: production Deep-Dive
When a release pipeline runs databricks bundle deploy -t prod :
Clean Git State Enforcement : DAB verifies that the local Git working tree is clean and matches the remote commit. If uncommitted local changes exist, deployment aborts immediately to prevent unversioned artifacts in production.
Standard Naming & Shared Paths : Resource names are preserved without user prefixes, and files deploy to /Shared/.bundle/${bundle.name}/prod/files/ .
Active Scheduling : Job schedules and continuous pipelines are deployed in their active, unpaused operational states.
2. Multi-Target Configuration & Override Mechanics
Target overrides allow teams to customize any property of a bundle (workspace URL, variables, cluster specifications, notification emails, and permissions) based on the target environment.
Comprehensive databricks.yml Multi-Target Manifest bundle:
name: lakehouse_sales_pipeline
include:
- resources/*.yml
# Global variable definitions
variables:
catalog:
- description: "Unity Catalog name"
- default: "dev_catalog"
schema:
- description: "Target schema name"
- default: "sales_raw"
cluster_node_type:
- description: "Worker VM size"
- default: "Standard_D4ds_v5"
min_workers:
- description: "Minimum auto-scaling workers"
- default: 1
max_workers:
- description: "Maximum auto-scaling workers"
- default: 2
targets:
# 1. Local Developer Target (Default)
dev:
- mode: development
- default: true
workspace:
host: https://adb-dev-workspace.7.azuredatabricks.net
variables:
- catalog: dev_catalog
- min_workers: 1
- max_workers: 2
# 2. Staging / UAT Environment
staging:
mode: production
workspace:
- host: https://adb-staging-workspace.7.azuredatabricks.net
- root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
variables:
- catalog: staging_catalog
- cluster_node_type: Standard_D8ds_v5
- min_workers: 2
- max_workers: 4
# 3. Production Environment
prod:
mode: production
workspace:
- host: https://adb-prod-workspace.7.azuredatabricks.net
- root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
run_as:
service_principal_name: 11111111-2222-3333-4444-555555555555
variables:
- catalog: prod_catalog
- schema: sales_core
- cluster_node_type: Standard_E8ds_v5 # Memory-optimized for production Photon
- min_workers: 4
- max_workers: 16
permissions:
- level: CAN_MANAGE
- group_name: data_engineering_leads
- level: CAN_VIEW
- group_name: bi_analysts
- 3. Service Principal Deployment & the run_as Identity Pattern
In production, data pipelines must never run under the identity of an individual developer . If an employee leaves the company or their Entra ID account is deactivated, all jobs owned by that user fail immediately.
The run_as Architecture
DAB supports the run_as block to specify the execution identity of deployed workflows: targets:
prod:
mode: production
run_as:
service_principal_name: 11111111-2222-3333-4444-555555555555 +-------------------------------------------------------------------------+
- | SERVICE PRINCIPAL RUN_AS EXECUTION PATTERN |
- +-------------------------------------------------------------------------+
- | |
- | CI/CD Runner (Azure DevOps / GitHub Actions) |
- | ├── Authenticates via Entra Service Principal Secret / OIDC |
- | └── Executes: databricks bundle deploy -t prod |
- | |
- | Azure Databricks Production Workspace |
- | ├── Deploys multi-task Lakeflow Job with Owner: Service Principal |
- | ├── Executes Job under Service Principal Unity Catalog Privileges |
- | │ (SELECT / MODIFY on prod_catalog.sales_core.*) |
- | └── Result: Zero dependency on individual developer credentials |
- +-------------------------------------------------------------------------+
Requirements for run_as Service Principal Deployment:
The Service Principal must exist in Microsoft Entra ID and be synchronized into the Azure Databricks Account Console via SCIM.
The Service Principal must be granted workspace access and assigned the Workspace Access entitlement.
The CI/CD agent or deploying identity must possess the Service Principal User role to deploy assets on behalf of that Service Principal.
The Service Principal must hold appropriate Unity Catalog permissions ( USE CATALOG , USE SCHEMA , CREATE TABLE , MODIFY , SELECT ) on the target production catalog and schemas.
4. Fine-Grained Resource Property Overrides
In addition to global variables, DAB allows overriding specific attributes of individual resources per target directly inside the target block or using target-specific resource schemas: # File: databricks.yml
resources:
jobs:
sales_job:
name: "Sales Processing Pipeline"
tasks:
- task_key: process_data
- job_cluster_key: main_cluster
notebook_task:
notebook_path: ../src/process.py
targets:
dev:
mode: development
resources:
jobs:
sales_job:
schedule:
pause_status: PAUSED
prod:
mode: production
resources:
jobs:
sales_job:
schedule:
- pause_status: UNPAUSED
- quartz_cron_expression: "0 0 2 * * ?"
email_notifications:
on_failure:
- "pagerduty-prod@corp.com"
- 5. Multi-Workspace Isolation vs. Single-Workspace Namespace Isolation
Organizations adopt two primary architectural patterns for environment separation in Azure Databricks:
Architectural Pattern Infrastructure Setup DAB Configuration Strategy
Multi-Workspace Isolation (Recommended) Separate Databricks workspaces for Dev, Staging, and Prod (e.g., adb-dev , adb-staging , adb-prod ) linked to a single Unity Catalog metastore. Configure unique workspace.host URLs under each target block. Physical compute and networking isolation between tiers.
Single-Workspace Namespace Isolation A single shared Databricks workspace where Dev, Staging, and Prod exist as separate Unity Catalog catalogs ( dev_cat , stage_cat , prod_cat ). Targets point to the same workspace.host , but override variables.catalog to isolate storage layers via Unity Catalog ACLs.