1.2 Advantages of Databricks ML Runtimes and Compute Selection

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

1.2 Advantages of Databricks ML Runtimes and Compute Selection

Key Takeaways Explain This Topic AI

Databricks Runtime for Machine Learning (DBR ML) ships scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, MLflow, Hyperopt, and the feature-engineering client pre-installed and version-compatible.

Choosing DBR ML over standard DBR removes the environment-assembly step entirely: no pip resolution at cluster start, no CUDA/cuDNN driver installation on GPU nodes, no library version conflicts.

Single-node clusters (driver only, zero workers) are the correct topology for pandas, scikit-learn, and single-node XGBoost because they eliminate Spark shuffle and scheduling overhead.

Multi-node clusters are required for Spark MLlib, distributed deep learning, and Hyperopt with SparkTrials , where work is genuinely parallel across executors.

Library scope is a deliberate choice: cluster-scoped for shared infrastructure, notebook-scoped %pip for isolated experimentation, and repo requirements.txt for reproducible production jobs.

Last updated: August 2026

1.2 Advantages of Databricks ML Runtimes and Compute Selection

Every machine learning workload on Databricks starts with a compute decision, and the exam tests two halves of it: which runtime image to attach, and which cluster topology to allocate. Both are graded on the same criterion — does the configuration match the parallelism the training library actually has?

Databricks Runtime for Machine Learning (DBR ML)

The Databricks Runtime for Machine Learning (DBR ML) is a specialized operating environment built on top of the standard Databricks Runtime. It eliminates manual environment setup by pre-packaging and optimizing the industry's standard machine learning, deep learning, and distributed computing libraries.

Key Components Built into DBR ML

Category Pre-Installed Frameworks & Tools Architectural Purpose

Core ML Libraries scikit-learn , xgboost , lightgbm , statsmodels Classical supervised/unsupervised machine learning and gradient boosted decision trees.

Deep Learning PyTorch , TensorFlow , Keras , torchvision , torchaudio Neural network modeling with automatic CUDA/cuDNN GPU runtime acceleration.

Distributed Training Horovod , HorovodRunner , PySpark MLlib , DeepSpeed Distributed data-parallel model training across multi-node worker clusters.

Hyperparameter Tuning Hyperopt , Optuna Bayesian optimization distributed across Spark workers using SparkTrials .

MLOps & Tracking MLflow , databricks-feature-engineering , automl Turnkey tracking of parameters, metrics, artifacts, model registration, and feature lookups.

Hardware Acceleration NVIDIA CUDA Drivers, cuDNN, NCCL, TensorRT Hardware-level optimization for GPU-accelerated computing nodes.

Exam Tip : When creating a cluster for machine learning workloads, always select a Databricks Runtime ML version (e.g., 14.3 LTS ML ) rather than standard DBR. Standard DBR contains Apache Spark and core dependencies, but lacks scikit-learn , torch , xgboost , and MLflow out of the box.

Compute Architecture: CPU vs. GPU & Single-Node vs. Multi-Node

Selecting the correct cluster topology and hardware architecture directly impacts training velocity, memory utilization, and cloud compute cost. +-----------------------------------------------------------------------------------------+

• | COMPUTE TOPOLOGY SELECTION |
• +-----------------------------------------------------------------------------------------+
• |

Is the training framework distributed?

• |
• +----------------------+----------------------+
• | |
• NO YES
• | |
• v v
• [ Single-Node Cluster ] [ Multi-Node Cluster ]
• Driver node only (no workers) - 1 Driver + N Worker Nodes
• Zero Spark network overhead - Distributed data partitions
• Best for: pandas, scikit-learn, - Best for: Spark MLlib, PyTorch FSDP,
• single-GPU deep learning Hyperopt with `SparkTrials`
• Single-Node vs. Multi-Node Clusters

Single-Node Clusters :

Architecture : Allocates only a driver node with 0 worker nodes ( spark.databricks.cluster.profile: singleNode ). Spark operates in local mode ( spark.master: local[*] ).

Target Workloads : Traditional single-threaded or multi-core Python libraries such as scikit-learn , pandas , statsmodels , or standalone XGBoost .

Advantage : Eliminates inter-node network latency, Spark serialization overhead, and the cost of idle worker nodes. All memory and CPU cores on the driver instance are dedicated directly to the local Python process.

Multi-Node Clusters :

Architecture : Allocates 1 driver node and 1 or more worker nodes connected via high-speed cluster networking.

Target Workloads : Distributed Spark MLlib pipelines processing terabyte-scale Delta tables, distributed deep learning using Horovod or PyTorch Distributed Data Parallel (DDP), and hyperparameter sweeps using Hyperopt with SparkTrials .

CPU vs. GPU Acceleration

CPU Clusters : Optimal for tabular data, classical machine learning algorithms ( scikit-learn , LightGBM , XGBoost ), exploratory data analysis with PySpark, and small-scale hyperparameter tuning. Provides the lowest cost-per-compute hour.

GPU Clusters : Required for training deep convolutional neural networks (CNNs), Transformers, Large Language Models (LLMs), high-dimensional embedding generation, and CUDA-accelerated gradient boosting. DBR ML GPU images automatically configure NVIDIA drivers and container runtimes.

Library Management Hierarchy & Scoping

Managing package dependencies across development and production environments requires selecting the appropriate installation scope. Databricks provides three distinct mechanisms: +---------------------------------------------------------------------------------------+

• | LIBRARY SCOPING HIERARCHY |
• +---------------------------------------------------------------------------------------+
• | 1. Cluster-Scoped Libraries |
• | - Installed via Cluster UI, Compute API, or Init Scripts |
• | - Available to all users and all notebooks running on that compute cluster |
• +---------------------------------------------------------------------------------------+
• | 2. Notebook-Scoped Libraries (%pip / %conda) |
• | - Installed inside the notebook execution context via `%pip install <package>` |
• | - Isolated to the specific notebook session; does not mutate cluster environment |
• +---------------------------------------------------------------------------------------+
• | 3. Repo / Workspace-Scoped Libraries |
• | - Managed in Git Folders via `requirements.txt` or wheel packaging |
• | - Loaded dynamically via `%pip install -r requirements.txt` |
• +---------------------------------------------------------------------------------------+
• Cluster-Scoped Libraries

Configuration : Installed via the Databricks Compute UI (under the Libraries tab), cluster initialization scripts ( Init Scripts ), or cluster creation JSON via the REST API.

Scope & Persistence : Installed across both driver and all worker nodes. Persists across cluster restarts.

Best Use Case : Core organization-wide base packages, custom enterprise drivers, or standardized internal SDKs.

Notebook-Scoped Libraries ( %pip Magic)

Configuration : Executed directly inside a notebook cell using the %pip magic command: # Notebook cell: Installing specific dependency versions

%pip install shap==0.44.1 category_encoders==2.6.3

• # Restart Python process automatically to load new packages into kernel
• dbutils.library.restartPython()

Scope & Isolation : The %pip magic command installs packages specifically into the active notebook's Python environment on the driver and automatically replicates them to Spark worker nodes for Python UDF execution. Other notebooks running concurrently on the same shared cluster remain unaffected.

Best Use Case : Rapid experimentation, testing conflicting package versions, and ensuring notebook self-containment.

Repo-Scoped Requirements

Configuration : Developers place a standard requirements.txt or pyproject.toml file at the root of their Databricks Git Folder.

Execution : At the top of a production or training notebook, execute: # Install exact pinned dependencies from repository root

%pip install -r ../requirements.txt

Best Use Case : Production scheduled jobs where deterministic dependency resolution across Git release tags is required.

Summary of Library Scoping Options

Library Type Installation Mechanism Scope of Impact Lifecycle & Persistence Exam Best Practice

Cluster-Scoped Cluster UI / API / Init Script All notebooks and users on the cluster Persists across cluster restarts until modified Use for global shared enterprise drivers and core utilities.

Notebook-Scoped %pip install <pkg> magic command Current notebook session only Ephemeral; re-executed when notebook runs Recommended for ad-hoc experimentation and isolated package testing.

Repo-Scoped %pip install -r requirements.txt Current notebook running within Git Folder Pinned to Git commit / branch Required for reproducible CI/CD production pipelines.