Study guide
1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
Key Takeaways Explain This Topic AI
Single User (Dedicated) access mode restricts cluster execution to a single designated user or service principal, supporting all Spark languages (Python, Scala, SQL, R) and custom JVM libraries.
Shared (Standard) access mode enables multi-tenant workload execution with strong user isolation and fine-grained Unity Catalog data governance, restricting arbitrary Scala and root OS execution.
Unity Catalog enforces securable permissions and credential downscoping at the cluster level, preventing users from accessing underlying cloud storage keys or bypassing table access controls.
Serverless compute architecture runs containerized workloads in a secure, Databricks-managed Azure environment, featuring automated security patching, rapid scaling, and strict network isolation.
Choosing between Single User, Shared, and Serverless compute hinges on language requirements (Scala/R vs. Python/SQL), user concurrency, isolation requirements, and governance boundaries.
Last updated: August 2026
1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
In Azure Databricks, Cluster Access Modes define the security boundary, user isolation model, language runtime capabilities, and Unity Catalog governance compatibility for compute resources.
Selecting the appropriate access mode ensures that enterprise security policies—such as row-level filters, column masking, and storage credential isolation—are strictly enforced while granting developers the language and library capabilities necessary for their workloads.
1. Evolution of Compute Security in Azure Databricks
Before Unity Catalog, Azure Databricks relied on legacy security mechanisms such as High Concurrency Table ACLs and Credential Passthrough . These legacy modes suffered from severe limitations:
Language Restrictions : Table ACLs blocked Python and Scala execution unless restricted through custom wrappers, frequently breaking third-party libraries.
Credential Leaks : Credential passthrough lacked fine-grained row/column masking and did not support automated service principal jobs cleanly.
No Uniform Governance : Compute-level permissions were disconnected from data storage metadata.
Unity Catalog fundamentally replaced these legacy models with standardized, hardware- and process-isolated Cluster Access Modes that enforce centralized identity and access control across all compute types.
2. Single User Access Mode (Dedicated Compute)
Single User Access Mode (formerly called Dedicated mode) allocates the compute cluster exclusively to one designated user identity or one Service Principal . SINGLE USER (DEDICATED) ACCESS MODE
- +-------------------------------------------------------------------------+
- | Assigned Identity: data_engineer@company.com (or Service Principal) |
- | |
- | [ Spark Driver JVM ] <---------------------------------------------+ |
- | - Full Unrestricted Access: Python, Scala, SQL, R | |
- | - Custom Init Scripts, C++ Extensions, JVM Bytecode Allowed | |
- | - Direct OS / DBFS Root Access Allowed | |
- | |
- | Unity Catalog Governance: Evaluated ONLY for the Assigned Identity. |
- | Other users CANNOT attach notebooks or execute commands here. |
- +-------------------------------------------------------------------------+
- Key Characteristics & Language Support
Full Polyglot Capabilities : Supports Python, Scala, SQL, and R without sandboxing or restriction.
Custom Extensions & JVM Manipulation : Developers can install custom C/C++ libraries, use low-level Java Virtual Machine (JVM) reflection, execute shell commands ( %sh ), and run custom cluster-scoped Init Scripts .
Unity Catalog Authorization : All queries against Unity Catalog tables, views, and volumes are authorized strictly using the assigned user's Entra ID identity and Unity Catalog privileges.
Primary Use Cases
Data Science & Advanced Machine Learning : Workloads requiring specialized deep learning frameworks (PyTorch, TensorFlow), GPU libraries (CUDA), or custom R packages.
Legacy Scala & Java Spark Pipelines : Automated production pipelines containing legacy Scala/Java JARs or low-level Spark RDD manipulation.
Dedicated Single-User Development : Engineers requiring root filesystem access or custom native binary debugging.
Security Warning : Because Single User mode grants unrestricted access to the underlying VM and Spark JVM, no other user can be allowed to execute code on this cluster. Doing so would allow session snooping and privilege escalation.
3. Shared Access Mode (Standard / Multi-Tenant Compute)
Shared Access Mode (formerly called Standard mode) is the enterprise multi-tenant compute model designed for concurrent team collaboration under strict Unity Catalog governance. SHARED (STANDARD) ACCESS MODE
- +-------------------------------------------------------------------------+
- | Multi-Tenant Users: Alice, Bob, Carol (Concurrent Sessions) |
- | |
- | [ Process Isolation Sandbox ] |
- | - Supported Languages: Python & ANSI SQL ONLY |
- | - Blocked: Arbitrary Scala, R, Shell (%sh), Unsafe JVM Calls |
- | - Blocked: Direct Local Root Disk Access & Raw Cloud Storage Keys |
- | |
- | Unity Catalog Governance: Evaluated DYNAMICALLY per active user. |
- | - Alice sees masked data; Bob sees unmasked data (ABAC Enforced). |
- +-------------------------------------------------------------------------+
- User Isolation & Sandboxing Mechanics
Process & Language Sandboxing : Multiple users attach their individual notebooks to the shared cluster simultaneously. Each user session runs in an isolated execution sandbox, preventing cross-user variable contamination or memory snooping.
Supported Languages : Supports Python and ANSI SQL . Arbitrary Scala and R execution is blocked/restricted by default because the JVM does not provide native multi-tenant process isolation, and unrestricted Scala could allow a user to bypass Unity Catalog security.
Fine-Grained Dynamic Governance : When a query executes, Unity Catalog dynamically evaluates permissions against the active querying user's Entra ID token:
Row Filters : Applied on-the-fly ( WHERE region = 'EMEA' ).
Column Masking : Applied dynamically ( hash(ssn) for unauthorized analysts).
Tag-Based Policies : Attribute-Based Access Control (ABAC) rules are strictly enforced.
Blocked Operations in Shared Mode
To guarantee multi-tenant security, Shared access mode intentionally blocks:
Execution of arbitrary shell scripts ( %sh ) and direct root filesystem writes.
Access to legacy DBFS root mount points containing plain-text credentials.
Modification of internal Spark JVM configuration parameters at runtime.
Direct acquisition of cloud storage master keys or IAM instance profile tokens.
4. Access Modes & Features Compatibility Comparison
The following matrix outlines the functional differences across all access modes:
- Capability / Feature Single User (Dedicated) Shared (Standard) No Isolation Shared (Legacy) Serverless Compute
- Supported Languages Python, Scala, SQL, R Python, SQL Python, Scala, SQL, R Python, SQL
Unity Catalog Governance Yes (Assigned User) Yes (Multi-User Dynamic) No (Legacy Table ACLs only) Yes (Multi-User Dynamic)
Row Filtering & Column Masking Enforced for assigned user Enforced dynamically per user Not Supported Enforced dynamically per user
Multi-User Concurrency No (Single identity only) Yes (Concurrent users) Yes (Unsecured) Yes (Instant elastic sharing)
- Custom Init Scripts Supported Restricted / Admin only Supported Not Supported (Managed Env)
- JVM & Scala Access Full Unrestricted Blocked / Sandboxed Unrestricted (No security) Blocked / Sandboxed
- Shell Execution ( %sh ) Supported Blocked Supported Blocked
Ideal Target Workload ML/DS, Scala ETL, GPU Team Analytics, BI, Shared ETL Migration only (Deprecated) Serverless Notebooks & Jobs
5. Serverless Compute Architecture
Serverless Compute represents the next generation of compute execution in Azure Databricks, available for Serverless SQL Warehouses , Serverless Notebooks , Serverless Jobs , and Serverless Delta Live Tables . +-------------------------------------------------------------------------+
- | CUSTOMER AZURE SUBSCRIPTION |
- | |
- | [ Customer ADLS Gen2 Storage Account ] <-------------------+ |
- | - Bronze, Silver, Gold Delta Tables | |
- +--------------------------------------------------------------|----------+
- | (Private Link / NCC)
- +--------------------------------------------------------------|----------+
- | DATABRICKS-MANAGED SERVERLESS PLANE | |
- | | |
- | +----------------------------------------------------------v-------+ |
- | | Hardware-Isolated Container Sandboxes | |
- | | - Instant Startup Fleet (<10 seconds) | |
- | | - Automated OS, DBR, and Security Patching | |
- | | - Dynamic Elastic Scaling (Instant scale-to-zero) | |
- | | - Single-Tenant Network Virtualization per Workspace | |
- | +------------------------------------------------------------------+ |
- +-------------------------------------------------------------------------+
- Core Architectural Principles
Shift of Compute Plane : In classic compute, virtual machines reside in the customer's Azure VNet. In Serverless compute, containerized compute fleets run in a secure, Databricks-managed cloud environment located in the same Azure region.
Hardware-Level Sandboxing : Workloads are isolated using enterprise micro-VM and container virtualization boundaries. No container is ever shared across different customer workspaces or corporate tenants.
Network Connectivity Configurations (NCC) : Serverless compute communicates securely with customer storage accounts (ADLS Gen2) over private network backbones using Azure Private Endpoints or managed Network Connectivity Configurations.
Zero Management Overhead : Cloud infrastructure provisioning, OS image hardening, Spark configuration tuning, and security vulnerability patching are managed automatically by Databricks without user intervention.
6. Credential Isolation & Token Downscoping in Unity Catalog
One of the most critical security functions of Unity Catalog across Shared and Serverless compute modes is Credential Isolation through Token Downscoping .
In legacy architectures, compute nodes required storage account keys or broad IAM Managed Identity access. If an analyst ran malicious Python code, they could extract those keys and gain unfettered access to all cloud storage data.
- Step-by-Step Downscoping Execution Workflow [ Data Analyst ]
- |
- | 1. Submits Query: SELECT * FROM sales.orders WHERE region = 'US'
- v
- [ Cluster Driver / Executor ]
- |
- | 2. Requests Data Access Token for target table path
- v
- [ Unity Catalog Metastore ]
- |
- | 3. Validates Entra ID identity, Table Permissions, and Row Filters
- | 4. Requests Downscoped Temporary SAS / Bearer Token from Azure Storage
- v
- [ Azure Storage Security Broker ]
- |
- | 5. Issues Short-Lived Downscoped Token (Strictly path- & time-limited)
- v
- [ Spark Executor ]
- |
- | 6. Directly reads underlying Parquet files via ABFSS using Downscoped Token
- v
- [ ADLS Gen2 Storage ]
Query Submission : An analyst submits a SQL or PySpark query targeting a Unity Catalog table (e.g., main.finance.payments ).
Metastore Authorization : The cluster driver intercepts the storage request and communicates with the Unity Catalog Metastore service.
Policy Evaluation : The metastore authenticates the user's Entra ID token and verifies that the user holds SELECT privileges. It dynamically injects any active row filters or column masks into the query execution plan.
Downscoped Credential Generation : The metastore generates a short-lived, downscoped temporary credential (such as an Azure Storage SAS token or Entra ID bearer token) restricted exclusively to the specific ADLS Gen2 folder path housing that table for the duration of the query.
Secure Execution : The downscoped credential is passed internally to the Spark executor storage subsystem over secure channels. The user's notebook code has no access to inspect, print, or capture this token.
Strict Expiration : Once the query finishes, the temporary token expires automatically, ensuring zero persistent credential exposure.