Study guide
3.2 Azure Managed Identity, Storage Credentials, & External Locations
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
3.2 Azure Managed Identity, Storage Credentials, & External Locations
Key Takeaways Explain This Topic AI
Azure Access Connector (Microsoft.Databricks/accessConnectors) provides credential-free, Azure Entra ID-managed identity authentication between Databricks and ADLS Gen2.
The Azure RBAC role 'Storage Blob Data Contributor' must be assigned to the Access Connector's managed identity on the target ADLS Gen2 storage account or container.
A Unity Catalog Storage Credential is a securable object in the metastore that encapsulates the Azure Access Connector resource ID and managed identity.
A Unity Catalog External Location pairs a Storage Credential with a specific ADLS Gen2 URL (abfss://container@account.dfs.core.windows.net/path), performing automated URL validation on creation.
External Locations prevent overlapping URL registrations and govern privileges including CREATE EXTERNAL TABLE, CREATE EXTERNAL VOLUME, READ FILES, and WRITE FILES.
Last updated: August 2026
3.2 Azure Managed Identity, Storage Credentials, & External Locations
DP-750 Exam Focus: Master the two-tier cloud storage security model in Unity Catalog: Cloud-level IAM (Azure Access Connector, User-Assigned Managed Identities, and Azure RBAC) combined with Catalog-level Securables (Storage Credentials and External Locations). Understand the exact DDL commands, Azure RBAC roles ( Storage Blob Data Contributor ), and URL validation rules required to establish secure external storage paths.
The Need for Credential-less Cloud Storage Architecture
In legacy Azure Databricks configurations, connecting to Azure Data Lake Storage Gen2 (ADLS Gen2) typically relied on one of three patterns:
Storage Account Access Keys: Broad account-level keys stored in Databricks Secrets or notebooks, presenting a severe credential leak risk.
Shared Access Signature (SAS) Tokens: Time-bounded tokens requiring ongoing rotation and management overhead.
DBFS Mounts ( dbfs:/mnt/... ): Global filesystem mounts that exposed underlying storage to all users on shared compute clusters, bypassing table-level security.
Unity Catalog eliminates all hardcoded secrets, keys, and global mount points by introducing a native, credential-less identity model built on Azure Managed Identities and Azure Resource Manager (ARM) .
Azure Access Connector Architecture
An Azure Access Connector ( Microsoft.Databricks/accessConnectors ) is a native first-party Azure resource that acts as the identity bridge between the Azure Databricks control/data planes and Azure Storage. +-----------------------------------------------------------------------------------------+
- | Azure Cloud |
- | |
- | +--------------------------+ +----------------------------+ |
- | | Azure Databricks | | ADLS Gen2 Storage | |
- | | Unity Catalog Plane | | (container@account.dfs...) | |
- | +--------------------------+ +----------------------------+ |
- | | ^ |
- | | Assumes Identity via ARM | |
- | v | |
- | +---------------------------------------+ | |
- | | Azure Access Connector | | |
- | | (Microsoft.Databricks/ | | |
- | | accessConnectors) | | |
- | | | | |
- | | Managed Identity: | | |
- | | - System-Assigned (SAMI), or | | |
- | | - User-Assigned (UAMI) | | |
- | +---------------------------------------+ | |
- | | | |
- | +--- Assigned RBAC: Storage Blob Data --------+ |
- | Contributor on Target Container |
- +-----------------------------------------------------------------------------------------+
- Managed Identity Types
When deploying an Access Connector in Azure, administrators choose between two managed identity strategies:
Identity Strategy Architecture & Resource Binding Enterprise Use Case & Scalability
System-Assigned Managed Identity (SAMI) Created automatically with the Access Connector. Its identity lifecycle is strictly bound to the Access Connector resource. Simplest setup for single-subscription architectures with straightforward storage topology.
User-Assigned Managed Identity (UAMI) Created independently in Microsoft Entra ID (Azure AD) and attached to the Access Connector resource. Enterprise Best Practice. Allows a single managed identity to be assigned across multiple storage accounts and shared across multiple Access Connectors across subscriptions.
Azure RBAC Configuration on ADLS Gen2
For Unity Catalog to read, write, and manage files in an ADLS Gen2 container or path, the Access Connector's managed identity must be granted appropriate Azure Role-Based Access Control (RBAC) permissions in the Azure Portal or via Azure CLI/Terraform.
Required Azure RBAC Roles
Storage Blob Data Contributor (Mandatory for Read/Write/Manage): Grants full read, write, and delete permissions to blob data within the target container. This role is required for metastore root storage, managed table storage, and writable external locations.
Storage Blob Data Reader (Read-Only Locations): Grants read-only access. Used when an external location is strictly configured for source ingestion and must be protected against accidental writes or deletes.
Storage Blob Data Owner : Grants full control including POSIX ACL modifications. Generally unnecessary unless advanced ACL delegation is required.
Exam Trap: Assigning traditional management roles such as Contributor or Owner at the Resource Group or Storage Account level does NOT grant data-plane access to ADLS Gen2 blob storage. Unity Catalog will fail to access data unless the specific Storage Blob Data Contributor (or Reader) data-plane role is assigned.
Unity Catalog Storage Credentials
A Storage Credential is a top-level securable object in the Unity Catalog metastore that encapsulates authentication to Azure storage. It represents the Databricks-side abstraction of the Azure Access Connector. +-------------------------------------------------------------------+
- | Unity Catalog Storage Credential Object |
- | |
- | Name: `azure_prod_credential` |
- | Cloud Provider: Azure Managed Identity |
- | Azure Connector ID: |
- | `/subscriptions/<sub>/resourceGroups/<rg>/ |
- | providers/Microsoft.Databricks/accessConnectors/prod-connector`|
- | User-Assigned Identity ID (Optional): |
- | `/subscriptions/<sub>/.../userAssignedIdentities/uami-prod-data`|
- +-------------------------------------------------------------------+
- DDL Syntax for Creating Storage Credentials
- 1. Using System-Assigned Managed Identity (SAMI): CREATE STORAGE CREDENTIAL azure_prod_credential
- AZURE_MANAGED_IDENTITY (
CONNECTOR_ID = '/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/rg-databricks-prod/providers/Microsoft.Databricks/accessConnectors/prod-access-connector'
- )
- COMMENT 'Primary storage credential for production ADLS Gen2 containers';
- 2. Using User-Assigned Managed Identity (UAMI): CREATE STORAGE CREDENTIAL azure_enterprise_uami_cred
- AZURE_MANAGED_IDENTITY (
CONNECTOR_ID = '/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/rg-databricks-prod/providers/Microsoft.Databricks/accessConnectors/prod-access-connector',
MANAGED_IDENTITY_ID = '/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/rg-identities/providers/Microsoft.ManagedIdentity/userAssignedIdentities/uami-databricks-storage'
- )
- COMMENT 'Credential using dedicated enterprise User-Assigned Managed Identity';
- Ownership and Permissions on Storage Credentials
By default, only Metastore Admins can create storage credentials.
Metastore Admins can delegate credential creation using: GRANT CREATE STORAGE CREDENTIAL ON METASTORE TO group_principal;
Users who possess the CREATE EXTERNAL LOCATION privilege on a Storage Credential can use it to register External Locations.
Unity Catalog External Locations
An External Location is a securable object in Unity Catalog that combines a cloud storage URI path with a Storage Credential . It establishes a governed boundary over a specific container or subdirectory in ADLS Gen2. +-------------------------------------------------------------------+
- | Unity Catalog External Location Object |
- | |
- | Name: `raw_landing_location` |
- | Storage URL: `abfss://raw-data@adlsprod.dfs.core.windows.net/` |
- | Storage Credential: `azure_prod_credential` |
- | Encryption / Validation: Validated on creation |
- +-------------------------------------------------------------------+
- DDL Syntax for Creating External Locations CREATE EXTERNAL LOCATION raw_landing_location
- URL 'abfss://raw-data@adlsprod.dfs.core.windows.net/landing/'
- WITH (STORAGE CREDENTIAL azure_prod_credential)
- COMMENT 'External location for raw landing ingestion files';
- Automated URL Validation and Overlap Rules
When a CREATE EXTERNAL LOCATION statement is executed, Unity Catalog performs automated checks:
Active URL Validation: Unity Catalog immediately contacts Azure Storage using the specified Storage Credential to verify that it can list, read, and write a test file at the designated URL path. If the Azure RBAC assignment is missing or the storage account firewall blocks traffic, creation aborts with an authorization error.
Bypassing Validation (Optional): In scenarios where storage will become available later, validation can be skipped with WITHOUT VALIDATION : CREATE EXTERNAL LOCATION staged_location
- URL 'abfss://staged@adlsprod.dfs.core.windows.net/'
- WITH (STORAGE CREDENTIAL azure_prod_credential)
- WITHOUT VALIDATION;
Overlap Prevention: Unity Catalog prevents overlapping external location URLs unless strict parent-child containment hierarchy rules are respected. You cannot define two external locations pointing to the exact same URI with different credentials.
Privilege Hierarchy on Storage Securables
Unity Catalog uses standard ANSI SQL GRANT and REVOKE statements to control access to Storage Credentials and External Locations:
Securable Object Privilege Description & Intended Grantee
Storage Credential CREATE EXTERNAL LOCATION Allows a data engineer or storage admin to create external locations referencing this credential.
Storage Credential CREATE EXTERNAL TABLE (Legacy/Direct) Allows creating external tables directly against the credential.
External Location CREATE EXTERNAL TABLE Allows data engineers to register external Delta or Parquet tables at paths inside the location.
External Location CREATE EXTERNAL VOLUME Allows creating external volumes for non-tabular files inside the location.
External Location READ FILES Allows users to query raw files directly using SQL ( SELECT * FROM text. abfss://...``) or PySpark.
External Location WRITE FILES Allows users or pipelines to write raw files or copy data into the external path. -- Granting team privileges on an External Location
- GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION raw_landing_location TO `data_engineers`;
- GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION raw_landing_location TO `data_engineers`;
- GRANT CREATE EXTERNAL VOLUME ON EXTERNAL LOCATION raw_landing_location TO `data_engineers`;
- End-to-End Authentication and Access Flow
Understanding the runtime authentication flow is essential for diagnosing access issues on the DP-750 exam: 1. Data Engineer issues query:
SELECT * FROM prod_catalog.raw.iot_telemetry;
2. Databricks Runtime queries Unity Catalog Metastore:
- Metastore verifies user has SELECT privilege on table.
- Metastore identifies table is External, residing at abfss://raw@adls.../iot
- Metastore checks External Location & Storage Credential.
3. Databricks Control Plane requests short-lived token:
- Calls Azure Resource Manager (ARM) / Entra ID using Azure Access Connector.
- Entra ID validates Access Connector's Managed Identity.
- Issues temporary down-scoped storage token.
4. Databricks Compute Cluster accesses ADLS Gen2 directly:
- Uses temporary storage token to read raw Parquet/Delta files.
- Cluster never receives or handles persistent credentials or keys.