3.6 Grid, Random, and Bayesian Search Strategies

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

3.6 Grid, Random, and Bayesian Search Strategies

Key Takeaways Explain This Topic AI

Grid search evaluates the full Cartesian product of candidate values — exhaustive, reproducible, and exponential in the number of hyperparameters.

Random search samples from distributions for a fixed budget and finds good configurations faster when only a few hyperparameters actually matter.

Bayesian optimisation (Hyperopt's TPE) models the relationship between hyperparameters and loss from completed trials and concentrates later trials in promising regions.

In Spark ML, grid search is expressed with ParamGridBuilder().addGrid(...).build() and executed by CrossValidator or TrainValidationSplit .

Grid search is preferable for a handful of discrete values you must cover exhaustively; Bayesian search wins when evaluations are expensive and the space is large.

Last updated: August 2026

3.6 Grid, Random, and Bayesian Search Strategies

Hyperparameter optimization is the systematic process of identifying the optimal combination of model hyperparameters that minimizes validation loss. While model parameters (such as tree split thresholds or linear coefficients) are learned directly during training, hyperparameters (such as learning rate, tree depth, and regularization penalties) govern the learning algorithm itself. Databricks natively bundles Hyperopt , an open-source Python library for Sequential Model-Based Optimization (SMBO) using Bayesian reasoning.

Hyperparameter Optimization Paradigms Compared +-----------------------------------------------------------------------------+

• | HYPERPARAMETER TUNING STRATEGY COMPARISON |
• | |
• | GRID SEARCH RANDOM SEARCH BAYESIAN OPTIMIZATION |
• | (ParamGridBuilder) (Random Sampling) (Hyperopt / TPE) |
• | +-----------------+ +-----------------+ +-----------------+ |
• | | . . . . . . . . | | . . . | | . . . . | |
• | | . . . . . . . . | | . . . | | . . . . . . | |
• | | . . . . . . . . | | . . . | | . . . | |
• | | . . . . . . . . | | . . . | | | |
• | +-----------------+ +-----------------+ +-----------------+ |
• | - Exhaustive grid - Random distributions - Learns from past trials |
• | - Combinatorial blowup - Better coverage - Focuses on optimal space|
• | - Wastes compute - Uninformed trials - Max Expected Improvement|
• +-----------------------------------------------------------------------------+
• Grid Search

Mechanics: Exhaustively evaluates every point in the Cartesian product of predefined parameter candidate lists.

Limitations: Suffers from the curse of dimensionality . If 5 hyperparameters each have 4 candidate values, Grid Search evaluates 4 5 = 1 , 024 4^5 = 1,024 4 5 = 1 , 024 full training runs. Furthermore, if only 2 of the 5 hyperparameters strongly influence performance, Grid Search wastes 95% of its computational budget re-evaluating identical values of the important parameters across uninformative variations of unimportant ones.

Random Search

Mechanics: Randomly draws candidate parameter combinations from specified continuous or discrete distributions over a fixed budget of N N N evaluations.

Advantages: Discovers superior parameter configurations faster than Grid Search because it tests unique values across continuous dimensions on every single trial.

Limitations: Independent and memoryless; it fails to leverage results from previous trials to guide future exploration.

Bayesian Optimization & Tree of Parzen Estimators (TPE)

Mechanics: Hyperopt implements the Tree of Parzen Estimators (TPE) algorithm. Rather than modeling the objective function response directly P ( y ∣ x ) P(y|x) P ( y ∣ x ) (as in Gaussian Processes), TPE uses Bayes' theorem to model two conditional parameter distributions: P ( x ∣ y ) = { ℓ ( x ) if y < y ∗ g ( x ) if y ≥ y ∗ P(x|y) = \begin{cases} \ell(x) & \text{if } y < y^* \\ g(x) & \text{if } y \ge y^* \end{cases} P ( x ∣ y ) = { ℓ ( x ) g ( x ) ​ if y < y ∗ if y ≥ y ∗ ​ where y ∗ y^* y ∗ is a quantile cutoff (e.g., top 15% best performing trials), ℓ ( x ) \ell(x) ℓ ( x ) is the density of hyperparameters associated with low validation loss, and g ( x ) g(x) g ( x ) is the density associated with the remaining evaluations.

Acquisition Function: TPE chooses candidate hyperparameter vectors x x x that maximize the Expected Improvement (EI) : EI ( x ) = γ y ∗ ℓ ( x ) − ℓ ( x ) ∫ − ∞ y ∗ P ( y ) d y γ ℓ ( x ) + ( 1 − γ ) g ( x ) ∝ ℓ ( x ) g ( x ) \text{EI}(x) = \frac{\gamma y^* \ell(x) - \ell(x) \int_{-\infty}^{y^*} P(y) dy}{\gamma \ell(x) + (1-\gamma) g(x)} \propto \frac{\ell(x)}{g(x)} EI ( x ) = γ ℓ ( x ) + ( 1 − γ ) g ( x ) γ y ∗ ℓ ( x ) − ℓ ( x ) ∫ − ∞ y ∗ ​ P ( y ) d y ​ ∝ g ( x ) ℓ ( x ) ​ Maximizing ℓ ( x ) g ( x ) \frac{\ell(x)}{g(x)} g ( x ) ℓ ( x ) ​ naturally samples parameters that have high probability under good models ℓ ( x ) \ell(x) ℓ ( x ) and low probability under poor models g ( x ) g(x) g ( x ) .

Expressing Each Strategy on Databricks

Grid search — ParamGridBuilder with a Spark ML tuner from pyspark.ml.tuning import ParamGridBuilder, CrossValidator

• param_grid = (ParamGridBuilder()
• .addGrid(lr.regParam, [0.01, 0.1, 1.0])
• .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])
• .build()) # 3 x 3 = 9 parameter combinations
• ParamGridBuilder builds the full Cartesian product; the count is the product of the

list lengths. Section 3.9 covers how that number combines with the fold count.

Random search — Hyperopt with rand.suggest from hyperopt import fmin, rand, hp

• best = fmin(fn=objective, space=search_space, algo=rand.suggest, max_evals=40)
• Every trial draws fresh values from the distributions, so continuous hyperparameters

are never re-tested at the same point — the key advantage over a grid.

Bayesian search — Hyperopt with tpe.suggest from hyperopt import fmin, tpe

• best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=40)
• TPE builds density models of the good and the poor trials and proposes candidates that

maximise the ratio between them, so later trials exploit what earlier trials revealed.

Choosing a Strategy

Situation Strategy Reason

A few discrete values that must each be evaluated (e.g. 3 regularisation strengths × 3 mixing ratios) Grid Exhaustive, deterministic, easy to audit

Many hyperparameters, most of them unimportant Random Spends the budget on distinct values of the parameters that matter instead of re-testing them

Continuous ranges spanning orders of magnitude Random or Bayesian A grid cannot cover a continuum without exploding

Each evaluation is expensive (minutes to hours) Bayesian (TPE) Uses every completed trial to choose the next one, so the budget goes further

Full reproducibility and exhaustive coverage are contractual requirements Grid The only strategy that provably evaluates every specified combination

Tuning a distributed pyspark.ml model Grid via CrossValidator Spark ML tuners take a ParamGrid ; see Section 3.7 for why SparkTrials is wrong here

The dimensionality argument

With 5 hyperparameters at 4 values each, a grid is 4 5 = 1,024 4^5 = 1{,}024 4 5 = 1 , 024 evaluations. If only

• 2 of the 5 genuinely affect the score, the grid tests each important value 64 times
• under irrelevant variations of the others. Random search with the same budget tests
• 1,024 distinct values of every parameter — which is why random search dominates grid

search on high-dimensional spaces at equal cost.

• Defining the Search Space Correctly
• The strategy only matters if the space it samples is well posed, and Hyperopt's space

primitives are themselves examinable:

Primitive Draws Use for

hp.uniform(label, low, high) A continuous value, uniformly Parameters whose sensible range spans one order of magnitude, such as subsample between 0.5 and 1.0

hp.loguniform(label, low, high) A continuous value, uniform in log space Parameters spanning orders of magnitude, such as a learning rate from 0.001 to 0.3 or regParam from 1e-4 to 1e2

hp.quniform(label, low, high, q) A value rounded to a multiple of q Integer-like parameters such as max_depth ; cast the result with int() before passing it to the estimator

hp.choice(label, options) One element of a list Genuinely categorical parameters such as the kernel or the boosting type

• The log-uniform case is the one that changes results. Sampling a learning rate
• uniformly between 0.001 and 0.3 puts roughly 97% of the draws above 0.01, so the small
• values that often win are almost never tried. hp.loguniform spreads the budget evenly
• across each decade instead. Note that its bounds are given in natural-log units, so a
• range of 0.001 to 0.3 is expressed as hp.loguniform("lr", np.log(0.001), np.log(0.3)) . import numpy as np
• from hyperopt import hp, fmin, tpe, space_eval

• search_space = {
• "learning_rate": hp.loguniform("learning_rate", np.log(0.001), np.log(0.3)),
• "max_depth": hp.quniform("max_depth", 3, 12, 1),
• "booster": hp.choice("booster", ["gbtree", "dart"]),
• }

• best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=40)
• best_params = space_eval(search_space, best) # {'booster': 'gbtree', ...}
• The hp.choice return-value trap
• fmin returns the index into the option list for any hp.choice parameter, not the
• value itself. Reading best["booster"] therefore yields 0 , and passing that straight

into an estimator either fails or silently trains something else. hyperopt.space_eval(search_space, best) translates the returned dictionary back into

real hyperparameter values, and it is the step scenario questions leave out.

• Budget
• max_evals is the total number of trials, and it is the only real cost control. A
• useful default is roughly 10 to 20 evaluations per hyperparameter for TPE, which needs
• some completed trials before its density models are informative — the first several

trials are effectively random regardless of the algorithm chosen.