3.9 Counting the Models Trained by Grid Search and Cross-Validation

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

3.9 Counting the Models Trained by Grid Search and Cross-Validation

Key Takeaways Explain This Topic AI

The number of parameter combinations is the product of the candidate-value counts across every tuned hyperparameter.

With K-fold cross-validation, total fits are combinations × K ; scikit-learn's GridSearchCV then adds one final refit on the full training set when refit=True .

Spark ML's CrossValidator fits numFolds × len(paramGrid) models, then refits the best combination once on the entire training set.

TrainValidationSplit fits len(paramGrid) models, one per combination, because there is a single validation split.

This arithmetic is what makes a grid over five hyperparameters with 5-fold CV computationally impossible — the cost is multiplicative, not additive.

Last updated: August 2026

• 3.9 Counting the Models Trained by Grid Search and Cross-Validation
• One exam objective is purely arithmetic: given a parameter grid and a validation
• scheme, state how many models are trained. It is easy marks, provided you apply the

formula rather than guessing.

The Formula

Step 1 — count the combinations. For hyperparameters p 1 … p n p_1 \dots p_n p 1 ​ … p n ​ with ∣ p i ∣ |p_i| ∣ p i ​ ∣ candidate values each, the grid contains C = ∏ i = 1 n ∣ p i ∣ C = \prod_{i=1}^{n} |p_i| C = i = 1 ∏ n ​ ∣ p i ​ ∣

combinations. A grid is a Cartesian product, so the counts multiply.

Step 2 — multiply by the validation scheme.

• Scheme Models fitted
• Single train/validation split C C C
• K K K -fold cross-validation C × K C \times K C × K
• K K K -fold CV with a final refit on all training data C × K + 1 C \times K + 1 C × K + 1
• Repeated K K K -fold, R R R repeats C × K × R C \times K \times R C × K × R
• Worked Example 1 — scikit-learn GridSearchCV

An SVM is tuned with GridSearchCV , 5-fold cross-validation, and this grid:

• C : [0.1, 1, 10] → 3 values
• kernel : ['linear', 'rbf'] → 2 values

gamma : [0.01, 0.1, 1] → 3 values C = 3 × 2 × 3 = 18 combinations C = 3 \times 2 \times 3 = 18 \text{ combinations} C = 3 × 2 × 3 = 18 combinations fits = 18 × 5 = 90 \text{fits} = 18 \times 5 = 90 fits = 18 × 5 = 90

90 models are trained during the search. With refit=True (the default), GridSearchCV then trains one more model — the best combination on the entire training

• set — for a total of 91 fits. When a question asks how many models the search trains,
• the answer is 90; when it asks how many fits occur in total including the refit, it is

91. Read the wording carefully.

• Worked Example 2 — Spark ML CrossValidator param_grid = (ParamGridBuilder()
• .addGrid(lr.regParam, [0.01, 0.1, 1.0]) # 3
• .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0]) # 3
• .build()) # 9 combinations

• crossval = CrossValidator(estimator=pipeline,
• estimatorParamMaps=param_grid,
• evaluator=evaluator,

numFolds=5) 9 combinations × 5 folds = 45 models 9 \text{ combinations} \times 5 \text{ folds} = 45 \text{ models} 9 combinations × 5 folds = 45 models

CrossValidator then refits the winning combination on the full training dataset, so

46 fits occur in total and cv_model.bestModel is that final refit.

Switching to TrainValidationSplit(trainRatio=0.8) with the same grid fits 9 models — one per combination — plus the final refit.

• Why the Arithmetic Matters
• Consider tuning XGBoost over five hyperparameters with four candidate values each,

under 5-fold cross-validation: 4 5 × 5 = 1,024 × 5 = 5,120 model fits 4^5 \times 5 = 1{,}024 \times 5 = 5{,}120 \text{ model fits} 4 5 × 5 = 1 , 024 × 5 = 5 , 120 model fits

• At two minutes per fit that is roughly 170 hours of sequential compute. This is the
• concrete reason the exam pairs this objective with the search-strategy objective: the

multiplicative blow-up is what motivates random and Bayesian search, and it is why TrainValidationSplit exists for very large data.

A budgeting checklist

Multiply the grid sizes to get C C C .

Multiply by K K K (and by repeats, if any).

Multiply by the average fit time.

Divide by the tuner's parallelism for an approximate wall-clock estimate.

If the number is unacceptable, cut the grid, cut K K K , or switch to Bayesian search —

not the fold count alone, since K < 3 K < 3 K < 3 makes the estimate noisy again.

• Common trap : parallelism changes how long the search takes , never how many
• models are trained . A question that adds parallelism=4 to a 45-model grid is

testing whether you conflate the two — the answer is still 45.

Variations That Change the Count

Several details alter the arithmetic, and each one appears in question stems.

• Conditional grids
• scikit-learn accepts a list of grid dictionaries, and the totals add rather than
• multiply, because parameters that do not apply to a kernel are simply not enumerated
• for it: param_grid = [
• {"kernel": ["linear"], "C": [0.1, 1, 10]}, # 3 combinations
• {"kernel": ["rbf"], "C": [0.1, 1, 10], "gamma": [0.01, 0.1]}, # 6 combinations
• ] # 9 in total
• Nine combinations under 5-fold cross-validation is 45 fits. A flat product over the
• union of the keys — kernel (2) × C (3) × gamma (2) — would count 12 combinations
• and 60 fits, over-counting the three linear configurations that gamma does not

apply to.

• Early stopping does not reduce the count
• Early stopping shortens each individual fit by halting boosting rounds; it does not

remove any combination from the grid. The number of models trained is unchanged.

Nested cross-validation multiplies again

An outer loop of K o u t e r K_{outer} K o u t er ​ folds wrapped around an inner tuning loop of K i n n e r K_{inner} K inn er ​ folds trains K o u t e r × ( C × K i n n e r + 1 ) K_{outer} \times (C \times K_{inner} + 1) K o u t er ​ × ( C × K inn er ​ + 1 ) models. With C = 9 C = 9 C = 9 , K i n n e r = 5 K_{inner} = 5 K inn er ​ = 5 , and K o u t e r = 5 K_{outer} = 5 K o u t er ​ = 5 , that is 5 × 46 = 230 5 \times 46 = 230 5 × 46 = 230 fits — which is why

nested CV is reserved for small datasets and publication-grade estimates.

• Hyperopt is budgeted, not enumerated
• fmin(..., max_evals=40) trains exactly 40 models regardless of how large the search
• space is, because Bayesian and random search sample rather than enumerate. If each

trial internally runs its own K K K -fold cross-validation, the total becomes 40 × K 40 \times K 40 × K . This contrast — a grid whose cost is set by the space, versus a

Hyperopt run whose cost is set by the budget — is the reason a large space pushes

practitioners toward fmin .