Study guide
3.7 Parallelizing Single-Node Models for Hyperparameter Tuning
Databricks Certified Machine Learning Associate Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Databricks Machine Learning (38% of the Exam)
1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion
2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)
2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate
3 Chapter 3: Model Development (31% of the Exam)
3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff
4 Chapter 4: Model Deployment (12% of the Exam)
4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints
3.7 Parallelizing Single-Node Models for Hyperparameter Tuning
Key Takeaways Explain This Topic AI
SparkTrials replaces Hyperopt's default Trials() and dispatches each trial as a Spark task, training many single-node models concurrently across executors.
SparkTrials is for single-node libraries only — scikit-learn, single-node XGBoost, LightGBM. Using it with pyspark.ml creates nested distributed jobs and fails.
Distributed Spark ML models are tuned with plain Trials() or with pyspark.ml.tuning.CrossValidator , which parallelises with its own parallelism argument.
Higher parallelism shortens wall-clock time but starves TPE of completed trials, degrading Bayesian search toward random search.
A practical heuristic is parallelism between 4 and 8 (bounded by available cores) with max_evals at least four times parallelism so several sequential waves occur.
Last updated: August 2026
3.7 Parallelizing Single-Node Models for Hyperparameter Tuning
Hyperparameter optimization and model validation on enterprise datasets require substantial computational resources. On Databricks, machine learning engineers have access to two powerful distributed tuning paradigms: SparkTrials (which distributes independent single-node model evaluations across Spark cluster executors) and pyspark.ml.tuning.CrossValidator (which distributes K-fold data partitions for distributed Spark ML pipelines). Selecting the correct distributed tuning abstraction and validation strategy is critical for avoiding resource contention, preventing data leakage, and optimizing compute expenditure.
Scaling Hyperopt with SparkTrials
Standard Hyperopt Trials() executes every trial sequentially on a single machine (the Databricks driver node). SparkTrials overrides this execution engine by broadcasting single-node training code and data to Spark worker nodes, executing multiple hyperparameter trials in parallel as separate Spark tasks. +-----------------------------------------------------------------------------+
- | SPARKTRILAS DISTRIBUTED ARCHITECTURE |
- | |
- | DATABRICKS DRIVER NODE |
- | +---------------------------------------------------------------------+ |
- | | Hyperopt fmin() + TPE Bayesian Engine | |
- | | - Proposes candidate hyperparameter configurations |
- | | - SparkContext broadcasts dataset X_train, y_train to cluster |
- | | - Instantiates SparkTrials(parallelism=4) |
- | +---------------------------------------------------------------------+ |
- | |
- | +---------------------------+---------------------------+ |
- | | | | |
- | v v v |
- | SPARK WORKER 1 SPARK WORKER 2 SPARK WORKER 3 |
- | +-------------------+ +-------------------+ +-------------+ |
- | | Task Slot 1 | | Task Slot 2 | | Task Slot 3 | |
- | | Trial 001: Fit | | Trial 002: Fit | | Trial 003: | |
- | | Single-Node Model | | Single-Node Model | | Fit Model | |
- | | (Scikit / XGBoost)| | (Scikit / XGBoost)| | | |
- | +-------------------+ +-------------------+ +-------------+ |
- | | | | |
- | +---------------------------+---------------------------+ |
- | v |
- | All Worker Trial Results (Loss, Metrics) Aggregated back to Driver |
- +-----------------------------------------------------------------------------+
- SparkTrials Parameters from hyperopt import SparkTrials
- spark_trials = SparkTrials(
- parallelism=8, # Number of concurrent trials
- timeout=3600, # Maximum total runtime in seconds (1 hour)
- max_trials_to_fail=5 # Max consecutive failures before aborting fmin
- )
parallelism : Sets the maximum number of trials evaluated concurrently across the cluster. If set to 8, Spark launches 8 independent Spark tasks across executor core slots.
timeout : Hard execution time ceiling. If the timeout expires, fmin() terminates gracefully and returns the best trial identified up to that timestamp.
max_trials_to_fail : Safeguard parameter (default 3, or configurable percentage). If N N N consecutive trials throw unhandled runtime exceptions (e.g., OOM on specific hyperparameter depths), fmin() halts to prevent burning cluster budget on broken code.
The Parallelism vs. Bayesian Optimization Tradeoff
A critical theoretical concept tested on the Databricks ML Associate exam is the tradeoff between parallelism and TPE optimization quality: +-----------------------------------------------------------------------------+
- | PARALLELISM VS. BAYESIAN LEARNING TRADEOFF |
- | |
- | LOW PARALLELISM (e.g., parallelism = 1 to 2) |
- | [+] Max Sequential Learning: Every trial learns from all previous results.|
- | [-] High Wall-Clock Duration: Cluster worker cores sit idle. |
- | |
- | EXCESSIVE PARALLELISM (e.g., parallelism = 50 with max_evals = 50) |
- | [-] Zero Sequential Learning: All 50 trials launch simultaneously. |
- | [-] Degrades to Pure Random Search: TPE has no completed trials to model. |
- | [+] Fast Wall-Clock Duration: Entire cluster saturated immediately. |
- | |
- | RECOMMENDED BALANCED HEURISTIC: |
- | Set parallelism = min(Total Worker Core Slots, 4 to 8) |
- | Set max_evals >= 4 * parallelism (ensures multiple sequential waves) |
- +-----------------------------------------------------------------------------+
- Architectural Rule: Single-Node vs. Distributed Models with Hyperopt
A fundamental architectural rule on Databricks dictates which trial runner to use based on the underlying algorithm library:
Algorithm Framework Target Library Correct Hyperopt Trial Runner Architectural Reason
Single-Node ML Scikit-learn, Single-Node XGBoost, LightGBM, PyTorch SparkTrials(parallelism=N) Distributes multiple single-node training tasks across Spark executor task slots.
Distributed Spark ML pyspark.ml ( LogisticRegression , RandomForestClassifier , GBTClassifier ) Trials() (Standard Sequential) Spark ML models already distribute their internal data partitions and tree training across the entire cluster. Running Spark ML inside SparkTrials causes illegal nested Spark jobs and catastrophic cluster thread contention.
- Worked Implementation: SparkTrials with Single-Node XGBoost
- Single-Node XGBoost with SparkTrials import numpy as np
- import xgboost as xgb
- from sklearn.model_selection import train_test_split
- from sklearn.metrics import roc_auc_score
- from hyperopt import fmin, tpe, hp, STATUS_OK, SparkTrials
- import mlflow
- pdf = spark.table("lakehouse_gold.churn_features").toPandas()
- X = pdf.drop(columns=["customer_id", "churned"])
- y = pdf["churned"]
- X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
- search_space = {
- "max_depth": hp.quniform("max_depth", 3, 8, 1),
- "learning_rate": hp.loguniform("learning_rate", np.log(0.01), np.log(0.2)),
- "n_estimators": hp.quniform("n_estimators", 50, 200, 25)
- }
def objective(params):
- params["max_depth"] = int(params["max_depth"])
- params["n_estimators"] = int(params["n_estimators"])
- clf = xgb.XGBClassifier(**params, random_state=42, eval_metric="logloss")
- clf.fit(X_train, y_train)
- preds = clf.predict_proba(X_val)[:, 1]
- auc = roc_auc_score(y_val, preds)
- return {"loss": -auc, "status": STATUS_OK}
- # Configure SparkTrials to run 4 concurrent trials across workers
- spark_trials = SparkTrials(parallelism=4, timeout=1800, max_trials_to_fail=3)
with mlflow.start_run(run_name="spark_trials_distributed_tuning"):
- best_params = fmin(
- fn=objective,
- space=search_space,
- algo=tpe.suggest,
- max_evals=20,
- trials=spark_trials
- )
- print("Best Distributed Tuning Parameters:", best_params)
- Two Kinds of Parallelism, One Rule
The rule to memorise is that exactly one layer may be distributed .
Model library Where the parallelism lives Correct tuner
scikit-learn, single-node XGBoost/LightGBM Across trials — each trial is one whole model on one executor SparkTrials(parallelism=N)
pyspark.ml estimators Inside each fit — the algorithm already spans the cluster Trials() , or CrossValidator(parallelism=N)
- Running a pyspark.ml model inside a SparkTrials task asks a Spark task to launch
- Spark jobs, which is not a supported execution pattern and results in contention or
- outright failure. Conversely, running scikit-learn under plain Trials() leaves every
executor idle while the driver trains one model at a time.
- Data movement
- SparkTrials broadcasts the training data referenced by the objective function to the
- executors. That works well when the training set fits comfortably in executor memory —
- which is the same precondition that made a single-node library appropriate in the first
- place. If the data is too large to broadcast, the answer is not more parallelism; it is
a distributed pyspark.ml model.