ExamFinch.com My Study

Study guide

4.2 Account-Level Identities, SCIM Sync, & Group Management

Practice this exam

Microsoft Certified: Azure Databricks Data Engineer Associate DP-750

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Azure Databricks Architecture & Compute Selection

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

2 Chapter 2: Compute Configuration, Performance & Cost Management

2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup

3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown

4 Chapter 4: Unity Catalog Privilege Model & Access Control

4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation

5 Chapter 5: Advanced Data Security, Masking & ABAC

5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks

6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing

6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming

7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment

10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables

10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs

11 Chapter 11: Workflow Orchestration with Lakeflow Jobs

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles

12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines

13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning

13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

4.2 Account-Level Identities, SCIM Sync, & Group Management

Key Takeaways Explain This Topic AI

Unity Catalog centralizes identity management at the Databricks Account level (accounts.azuredatabricks.net), decoupling users, groups, and service principals from individual workspaces.

Automated SCIM (System for Cross-domain Identity Management) synchronization from Microsoft Entra ID provisions users and security groups directly to the Databricks Account.

Legacy workspace-local groups cannot be referenced in Unity Catalog access control grants; all governance policies require centrally synchronized Account Groups.

Service principals authenticated via OAuth 2.0 machine-to-machine (M2M) credentials replace personal access tokens (PATs) for production CI/CD pipelines, Lakeflow Jobs, and automated ETL.

The core RBAC best practice in Unity Catalog is to grant privileges exclusively to functional Account Groups rather than individual user accounts.

Last updated: August 2026

4.2 Account-Level Identities, SCIM Sync, & Group Management

DP-750 Exam Focus: Master identity architecture in Azure Databricks. Understand the differences between Account-level identities and legacy workspace-local identities, how System for Cross-domain Identity Management (SCIM) automates provisioning from Microsoft Entra ID, how to configure and utilize Service Principals with OAuth M2M authentication for automated pipelines, and why granting privileges to Account Groups is the required enterprise best practice.

The Account-Level Identity Plane

In legacy Azure Databricks deployments, identity management was decentralized. Each workspace maintained its own isolated directory of users and groups. If an enterprise operated five workspaces across development, staging, and production, user onboarding, offboarding, and group permission updates had to be manually repeated in all five workspaces.

Unity Catalog introduces a centralized Account-level identity plane managed through the Databricks Account Console ( accounts.azuredatabricks.net ). +-----------------------------------------------------------------------------------------+

  • | Microsoft Entra ID |
  • | (Enterprise Identity Provider: Users, Groups, App Registrations) |
  • +-----------------------------------------------------------------------------------------+
  • |
  • | Automated SCIM 2.0 Provisioning
  • v
  • +-----------------------------------------------------------------------------------------+
  • | Azure Databricks Account Console (Central Plane) |
  • | accounts.azuredatabricks.net |
  • | |
  • | +-----------------------+ +-----------------------+ +-----------------------------+ |
  • | | Account Users | | Account Groups | | Account Service Principals | |
  • | | (jane.doe@corp.com) | | (`data-engineers`) | | (sp-lakeflow-prod-etl) | |
  • | +-----------------------+ +-----------------------+ +-----------------------------+ |
  • +-----------------------------------------------------------------------------------------+
  • | |
  • Workspace | | Workspace
  • Assignment v v Assignment
  • +---------------------------------------+ +---------------------------------------+
  • | Workspace: Dev-EastUS | | Workspace: Prod-EastUS |
  • | - Workspace Access Entitlement | | - Workspace Access Entitlement |
  • | - Databricks SQL Access | | - Serverless Compute Access |
  • +---------------------------------------+ +---------------------------------------+
  • The Three Identity Types in Unity Catalog

Users: Individual human identities authenticated via Microsoft Entra ID Single Sign-On (SSO). Represented by their corporate email address (e.g., alex.smith@company.com ).

Service Principals: Non-human automated identities representing applications, CI/CD tools (GitHub Actions, Azure DevOps), orchestration systems (Azure Data Factory), or automated Lakeflow Jobs. Service principals do not consume interactive user licenses.

Groups: Logical collections of users and service principals. Groups simplify access control by allowing administrators to grant privileges to a single collective entity rather than managing hundreds of individual user grants.

Microsoft Entra ID SCIM Synchronization

SCIM (System for Cross-domain Identity Management 2.0) is an open standard protocol that automates the exchange of user and group identity data between identity providers and cloud applications. In Azure Databricks, SCIM synchronizes identities directly from Microsoft Entra ID (formerly Azure Active Directory) into the Databricks Account Console.

Account-Level SCIM vs. Legacy Workspace-Level SCIM

Architectural Attribute Account-Level SCIM (Recommended) Workspace-Level SCIM (Legacy)

Sync Target Databricks Account Console ( accounts.azuredatabricks.net ) Individual workspace instance URL ( adb-xxx.azuredatabricks.net )

Entra ID App Type Databricks Account SCIM Provisioning Connector Workspace-specific Enterprise Application

Identity Scope Creates Account Groups and Account Users available across all workspaces and metastores Creates workspace-local users and groups visible only in that single workspace

Unity Catalog Compatibility Fully Compatible: Account groups can be granted privileges on any UC securable Incompatible: Workspace-local groups cannot be used in Unity Catalog GRANT statements

Offboarding Speed Immediate global deactivation across all workspaces and metastores upon Entra ID disablement Requires manual or per-workspace synchronization runs to revoke access

How SCIM Synchronization Operates

An Azure Cloud Administrator registers a Databricks SCIM Provisioning Application in Microsoft Entra ID.

The administrator generates a SCIM token in the Databricks Account Console and configures the Tenant URL: https://accounts.azuredatabricks.net/api/2.0/accounts/<account-id>/scim/v2

Specific Entra ID Security Groups (e.g., AAD-DataEngineers , AAD-DataAnalysts , AAD-FinanceTeam ) are assigned to the provisioning application.

Entra ID automatically provisions users, group memberships, and identity attributes into the Databricks Account every 40 minutes (or on-demand via Provision on demand).

When an employee departs and is disabled in Entra ID, the SCIM engine deactivates their Databricks account identity, instantly terminating all active sessions, token authentications, and Unity Catalog data access.

Account Groups vs. Legacy Workspace-Local Groups

A frequent source of confusion on the DP-750 exam is the distinction between Account Groups and Workspace-Local Groups . ACCOUNT GROUPS (Unity Catalog Standard) WORKSPACE-LOCAL GROUPS (Legacy)

  • +-------------------------------------+ +-------------------------------------+
  • | - Created in Account Console or SCIM| | - Created inside Workspace Admin UI |
  • | - Synchronized across all workspaces| | - Isolated to one single workspace |
  • | - Usable in Unity Catalog GRANTs | | - CANNOT be used in UC GRANTs |
  • | - Example: `data_engineers_prod` | | - Example: `local_analysts_ws1` |
  • +-------------------------------------+ +-------------------------------------+
  • | |
  • v v

GRANT SELECT ON TABLE ... GRANT SELECT ON TABLE ...

  • [SUCCESS] [ERROR: Invalid Principal]
  • Why Workspace-Local Groups Cannot Govern Unity Catalog

Unity Catalog metastores span multiple workspaces. Because a workspace-local group exists only within the internal database of a single workspace, other workspaces bound to the same metastore cannot resolve its membership. Consequently, Unity Catalog strictly enforces that only Account Groups can be granted privileges on Unity Catalog securables .

Exam Trap: If you attempt to execute GRANT SELECT ON TABLE prod.silver.orders TO local_team_group; , Unity Catalog will return an error stating that the principal local_team_group does not exist or is not an account-level identity. Workspace groups must be migrated to Account Groups via Account SCIM or the Databricks Account Console.

Loading diagram... Microsoft Entra ID SCIM Identity Synchronization & Entitlement Distribution

Service Principals & Automated Machine-to-Machine (M2M) Workloads

A Service Principal is a headless, non-interactive identity used by automated systems, scheduled workflows, and external applications. In production Azure Databricks lakehouses, all scheduled ingestion, Delta Live Tables / Lakeflow pipelines, and CI/CD operations must run under a service principal rather than a human user account.

Why Production Pipelines Must NOT Run Under Personal User Identities

Employee Turnover & Offboarding: When a data engineer leaves the organization, their Entra ID account is deactivated. If production Lakeflow Jobs or Delta Live Tables were scheduled under their personal identity, all associated production pipelines immediately fail with authentication errors.

Security & Auditability: Service principals can be restricted to least-privilege data access without granting interactive workspace login rights.

Token Rotation & Expiration: Personal Access Tokens (PATs) expire or can be accidentally exposed in code. Service principals use managed enterprise OAuth 2.0 tokens.

Service Principal Types in Azure Databricks

Microsoft Entra ID Service Principals: An Application Registration in Microsoft Entra ID ( Application (client) ID ). Best practice for enterprise multi-resource Azure integration (e.g., accessing Azure Key Vault, Azure Event Hubs, and Databricks).

Databricks Managed Service Principals: Created directly in the Databricks Account Console. Useful for Databricks-native automation when direct Entra ID tenant registration is restricted.

OAuth 2.0 Machine-to-Machine (M2M) Authentication

Databricks supports native OAuth 2.0 M2M authentication for service principals, eliminating the need for long-lived Personal Access Tokens (PATs): +--------------------------+ +-------------------------------+ +----------------------------+

  • | Automated CI/CD / Job | | Databricks OAuth Endpoint | | Databricks REST API / UC |
  • | (e.g., GitHub Actions) | | (accounts.azuredatabricks...)| | (Cluster / Query Engine) |
  • +--------------------------+ +-------------------------------+ +----------------------------+
  • | | |
  • | 1. POST /oidc/v1/token | |
  • | (client_id + client_secret) | |
  • +-------------------------------------->| |
  • | | |
  • | 2. Returns short-lived OAuth token | |
  • | (Valid for 60 minutes) | |
  • |<--------------------------------------+ |
  • | |
  • | 3. API Request with Bearer Token (Authorization: Bearer <token>) |
  • +------------------------------------------------------------------------------>|
  • | |
  • | 4. Executes pipeline / queries UC securable under Service Principal identity |

|<------------------------------------------------------------------------------+ -- Granting Unity Catalog Data Privileges to a Service Principal

  • GRANT USE CATALOG ON CATALOG prod_lakehouse TO `00000000-1111-2222-3333-444444444444`;
  • GRANT USE SCHEMA, CREATE TABLE, MODIFY ON SCHEMA prod_lakehouse.raw TO `00000000-1111-2222-3333-444444444444`;

-- Or better: Add the Service Principal to an Account Group and grant to the group!

  • GRANT USE CATALOG ON CATALOG prod_lakehouse TO `etl_pipeline_runners`;
  • GRANT USE SCHEMA, MODIFY ON SCHEMA prod_lakehouse.silver TO `etl_pipeline_runners`;
  • Enterprise Best Practice: Grant to Groups, Never to Users

The fundamental governance rule for the DP-750 exam is: Never assign direct Unity Catalog privileges to individual user accounts. All privileges must be assigned to Account Groups. ANTI-PATTERN (Direct User Grants) RECOMMENDED PATTERN (Group-Based RBAC)

User: alice@corp.com Microsoft Entra ID Security Group:

  • -> GRANT SELECT ON TABLE ... `AAD-Finance-Analysts`
  • User: bob@corp.com |
  • -> GRANT SELECT ON TABLE ... | (Automated SCIM Sync)
  • User: charlie@corp.com v
  • -> GRANT SELECT ON TABLE ... Databricks Account Group: `finance_analysts`
  • |
  • * When 50 analysts join/leave, v
  • 50 individual SQL statements must be run. GRANT SELECT ON SCHEMA finance.gold TO `finance_analysts`;

* Auditing requires parsing thousands of ACLs. * Adding Alice to Entra ID group instantly grants access.

Recommended Group Taxonomy Pattern

Account Group Name Typical Membership Assigned Unity Catalog Privileges

data_engineers_core Data engineers, pipeline developers CREATE CATALOG (Dev), USE CATALOG , USE SCHEMA , CREATE TABLE , MODIFY (Staging/Prod)

data_analysts_bi BI developers, reporting analysts USE CATALOG , USE SCHEMA , SELECT on gold schemas and reporting views

ml_engineers_nlp Data scientists, ML engineers USE CATALOG , USE SCHEMA , SELECT , READ VOLUME , CREATE MODEL

lakeflow_pipeline_runners Production Service Principals USE CATALOG , USE SCHEMA , MODIFY , READ VOLUME , WRITE VOLUME on bronze / silver

governance_auditors Compliance officers, internal auditors USE CATALOG , USE SCHEMA , SELECT on system.access , system.billing , system.lineage