Study guide
2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
Key Takeaways Explain This Topic AI
The Databricks Runtime (DBR) version you pick simultaneously pins the Apache Spark build, the Delta Lake table features available, the Python version, and every preinstalled library, so runtime selection is a compatibility decision and not just a performance one.
Long-Term Support (LTS) runtimes are the correct choice for scheduled production jobs; pin job compute to a named LTS version so a runtime deprecation cannot silently change Spark behavior between runs.
Databricks Runtime for Machine Learning is a separate image that adds MLflow, PyTorch, TensorFlow, scikit-learn, and XGBoost (with CUDA and cuDNN on GPU variants); using it for plain SQL or ETL work only lengthens cluster start time.
Compute has exactly four permission levels - NO PERMISSIONS, CAN ATTACH TO, CAN RESTART, and CAN MANAGE - and only CAN MANAGE can resize a cluster, edit its configuration, or grant permissions to other principals.
Compute policies control what compute a user is allowed to create; compute ACLs control which existing compute a user is allowed to touch. Production workspaces need both, because neither substitutes for the other.
Last updated: August 2026
2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
DP-750 Exam Focus: Two skill-area 1 bullets land squarely on this section - "Configure compute feature settings, including Photon acceleration, Azure Databricks runtime/Spark version, and machine learning" and "Configure access permissions to a compute resource." Section 2.2 already covered Photon and compute policies. This section closes the loop on runtime versioning and on who is allowed to attach to, restart, or reconfigure a cluster.
1. What a Databricks Runtime Version Actually Pins
A Databricks Runtime (DBR) version is not a Spark version label. It is a complete, immutable machine image containing:
A hardened Ubuntu Linux base and the Databricks container runtime
A Databricks-optimized build of Apache Spark (the Spark version is a property of the DBR, not something you choose independently)
- Delta Lake , including the set of Delta table features the image can read and write
- Language runtimes: a specific Python version, Scala, Java (JDK), and R
- Preinstalled libraries (pandas, NumPy, PyArrow, the ABFSS/Hadoop Azure connectors, JDBC drivers)
- The Photon native execution binaries on Photon-capable images
Because all of that is baked into one image, upgrading a runtime can change results, not just speed: a newer DBR may write a Delta protocol version an older reader cannot open, or ship a pandas major release that changes type coercion in a UDF.
Runtime Variants You Can Select
Variant What it adds Choose it for
Databricks Runtime (standard) Spark, Delta Lake, Python/Scala/R, core connectors Batch ETL, Structured Streaming, SQL, Lakeflow pipelines
Databricks Runtime LTS The same image published on a Long-Term Support channel with an extended support window Anything scheduled: production jobs, pipelines, pinned CI environments
Databricks Runtime for Machine Learning Everything in standard plus MLflow, PyTorch, TensorFlow, scikit-learn, XGBoost, and distributed training libraries; GPU images add CUDA and cuDNN Model training, tuning, batch inference, feature engineering with ML libraries
Photon-enabled runtimes The vectorized C++ engine described in Section 2.2, exposed as a toggle on runtime versions that ship it SQL and DataFrame aggregation, join, and scan-heavy workloads
Serverless compute No DBR picker at all - Databricks manages the version through environment/client versions Notebooks, Lakeflow Spark Declarative Pipelines, SQL warehouses, ingestion pipelines
Read the dropdown carefully. Photon and the ML runtime are both chosen from the Databricks Runtime version selector, but they are distinct images. You select one runtime; the Photon toggle only appears for versions that ship the Photon engine. Serverless compute has no runtime selector at all, which is why exam scenarios that say "serverless" also remove every runtime-version distractor from the answer set.
2. LTS Versus Non-LTS: the Production Stability Decision
Databricks publishes most runtimes on a short support window and designates specific releases as LTS (Long-Term Support) with a substantially longer one. The operational rule that follows from this is simple and heavily testable:
Scheduled jobs and pipelines pin a named LTS version. A job that runs nightly for two years must not silently roll onto a new Spark build.
Interactive/development compute may track the newest runtime so the team can validate new table features before production adopts them.
Never let a production job float onto "latest." When a runtime reaches end of support, jobs pinned to it must be migrated deliberately, with a test run, rather than discovering the change in a 3 a.m. failure.
Runtime Version Gates Table Features
The exam likes scenarios where a feature "does not work" purely because the runtime is too old. Known minimums worth memorizing:
- Capability Minimum Databricks Runtime
- Read Delta tables that use deletion vectors DBR 12.2 LTS and above
- Write deletion vectors with all optimizations ( MERGE , UPDATE , DELETE ) DBR 14.3 LTS and above
- Row-level concurrency on tables with deletion vectors DBR 14.2 and above
- Managed and foreign Apache Iceberg tables in Unity Catalog DBR 16.4 LTS and above
Enabling deletion vectors upgrades the Delta table protocol. After the upgrade, clients on older runtimes can no longer read the table - a classic "we upgraded one table and the downstream Synapse reader broke" scenario.
3. Databricks Runtime for Machine Learning
The ML runtime is the standard runtime plus a curated machine-learning stack, preinstalled and version-matched so the libraries actually work together:
- MLflow with autologging wired into the workspace tracking server
- Deep learning : PyTorch, TensorFlow/Keras, and their distributed training helpers
- Classical ML : scikit-learn, XGBoost, LightGBM
- GPU images : NVIDIA drivers, CUDA, and cuDNN preinstalled and matched to the GPU VM SKU
When not to use it. The ML image is much larger than the standard image, so clusters take noticeably longer to start and cost more per run. A nightly medallion ETL job that runs MERGE statements and writes Delta tables gains nothing from it. Choose the ML runtime only when the workload actually imports ML libraries or needs GPU acceleration. {
- "cluster_name": "feature-engineering-gpu",
- "spark_version": "<ml-gpu-runtime-version>",
- "node_type_id": "Standard_NC6s_v3",
- "num_workers": 2,
- "autotermination_minutes": 30
- }
The spark_version field in the Clusters API is where the runtime choice lives; ML and GPU images are distinct spark_version identifiers, not flags added to a standard runtime.
4. Compute Access Permissions: the Four Levels
Every compute resource has exactly four permission levels. They are strictly cumulative - each level includes everything below it.
Permission level What the principal can do
NO PERMISSIONS The default. The principal cannot attach notebooks to the compute or use it for queries.
CAN ATTACH TO Attach notebooks, run commands on the compute, and view the Spark UI, compute metrics, and event log.
CAN RESTART Everything above, plus start, terminate, and restart the compute.
CAN MANAGE Everything above, plus edit the configuration, resize the cluster, and edit permissions for other principals.
Operational Rules the Exam Tests
You must hold CAN MANAGE on a compute to configure its permissions. Workspace admins hold it implicitly.
Permissions are the exception to the restart rule. Editing any attribute of a running compute requires a restart - except the compute size and permissions , which apply live. You can also update permissions on compute that is neither running nor terminated.
Cloning does not copy permissions. Cloning a compute pre-populates the configuration but deliberately drops compute permissions and attached notebooks . The clone starts with an empty ACL.
Driver logs are locked to CAN MANAGE by default. Secrets are not redacted from the Spark driver stdout and stderr streams, so on job and all-purpose compute in dedicated or standard access mode only CAN MANAGE users can read driver logs. To open them to CAN ATTACH TO and CAN RESTART users, set spark.databricks.acl.needAdminPermissionToViewLogs false in that compute's Spark config. It is a per-compute property - there is no workspace-wide switch.
Autostart checks two ACLs. When a scheduled job or a JDBC/ODBC connection triggers a terminated compute to auto-restart, Databricks evaluates both the compute access control list and the job access control list before starting it.
Terminated compute is deleted after 30 days unless an administrator pins it (up to 100 pinned compute resources per workspace).
How to Set Permissions
Three supported paths, all equivalent:
Workspace UI - Compute -> kebab menu on the row -> Edit permissions -> select a user, group, or service principal and pick a level.
Permissions API - for scripted or CI-driven grants.
Databricks Terraform provider - for infrastructure-as-code environments.
Grant to groups , not individuals, for the same reason Unity Catalog privileges go to account-level groups: onboarding and offboarding become a group membership change instead of an ACL sweep across every cluster.
- 5. Compute ACLs Versus Compute Policies Versus Pool ACLs
- These three controls are constantly confused in exam distractors because they all "restrict compute."
Control Question it answers Failure if missing
Compute policy (Section 2.2) What kind of compute may this user create? Users spin up 64-node Standard_E64 clusters with no auto-termination
Compute ACL (this section) Which existing compute may this user attach to, restart, or reconfigure? Any user can attach to - or terminate - the production streaming cluster
Pool ACL Which users may draw warm instances from this pool? Cost-controlled pools get consumed by ad hoc analytics work
A user granted CAN USE on a policy can create clusters shaped by that policy, but that grant says nothing about the clusters other people already created. Conversely, CAN MANAGE on one cluster does not let a user create new ones outside policy limits.
6. Common Exam Traps
CAN RESTART is not CAN MANAGE. A user who needs to restart a stuck streaming cluster needs CAN RESTART. A user who needs to add two workers needs CAN MANAGE. Granting CAN MANAGE "so they can restart it" over-grants.
CAN ATTACH TO cannot restart. If the cluster is terminated, a CAN ATTACH TO user is stuck until someone with CAN RESTART starts it - or until an autostart-eligible job does.
A larger runtime is not a faster runtime. Scenarios that add the ML runtime to an ETL cluster to "improve performance" are wrong; Photon, right-sized nodes, and better clustering are the performance levers.
"Latest runtime" is rarely the right answer for a production job. Look for the LTS option.