ExamFinch.com My Study

Study guide

3.8 Cross-Validation vs. a Train-Validation Split

Practice this exam

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

3.8 Cross-Validation vs. a Train-Validation Split

Key Takeaways Explain This Topic AI

K-fold cross-validation trains K models per parameter combination and averages the K holdout scores, giving a lower-variance estimate at K times the cost.

TrainValidationSplit evaluates each parameter combination once against a single holdout governed by trainRatio , which is far cheaper but noisier.

Cross-validation is preferred on small or medium datasets, where a single split's composition can swing the metric materially.

A single split is preferred on very large datasets, where the holdout is already big enough to be stable and K-fold cost is prohibitive.

Both are executed by passing an Estimator, a ParamGrid , and an Evaluator ; putting preprocessing inside the pipeline is what keeps each fold leak-free.

Last updated: August 2026

  • 3.8 Cross-Validation vs. a Train-Validation Split
  • Cross-Validation Paradigms on Databricks

Cross-validation estimates how well a predictive model generalizes to unseen independent data, mitigating evaluation variance. +-----------------------------------------------------------------------------+

  • | CROSS-VALIDATION METHODOLOGIES |
  • | |
  • | [1] K-FOLD CROSS-VALIDATION (numFolds = 5) |
  • | Fold 1: [ VAL ] [ TRAIN ] [ TRAIN ] [ TRAIN ] [ TRAIN ] --> Score 1 |
  • | Fold 2: [ TRAIN ] [ VAL ] [ TRAIN ] [ TRAIN ] [ TRAIN ] --> Score 2 |
  • | Fold 3: [ TRAIN ] [ TRAIN ] [ VAL ] [ TRAIN ] [ TRAIN ] --> Score 3 |
  • | Fold 4: [ TRAIN ] [ TRAIN ] [ TRAIN ] [ VAL ] [ TRAIN ] --> Score 4 |
  • | Fold 5: [ TRAIN ] [ TRAIN ] [ TRAIN ] [ TRAIN ] [ VAL ] --> Score 5 |
  • | Final Metric = Mean(Score 1..5) +/- StdDev |
  • | |
  • | [2] STRATIFIED K-FOLD |
  • | Guarantees identical class proportions (e.g. 2% positive) in every fold. |
  • | |
  • | [3] TIME-SERIES WALK-FORWARD VALIDATION (Expanding Window) |
  • | Split 1: [ Train: Month 1-3 ] -> [ Val: Month 4 ] |
  • | Split 2: [ Train: Month 1-4 ] -> [ Val: Month 5 ] |
  • | Split 3: [ Train: Month 1-5 ] -> [ Val: Month 6 ] |
  • | * Avoids temporal lookahead leakage: Never train on future to test past! |
  • +-----------------------------------------------------------------------------+
  • PySpark CrossValidator vs. TrainValidationSplit

In pyspark.ml.tuning , Databricks provides two primary tuning orchestrators:

Feature CrossValidator TrainValidationSplit

Data Partitioning Splits training set into K K K disjoint folds (e.g., K = 5 K=5 K = 5 ). Splits training set once into train/validation sets (e.g., 80/20 ratio via trainRatio=0.8 ).

Number of Models Fitted numFolds × len ( ParamGrid ) = K × M \text{numFolds} \times \text{len}(\text{ParamGrid}) = K \times M numFolds × len ( ParamGrid ) = K × M models. 1 × len ( ParamGrid ) = M 1 \times \text{len}(\text{ParamGrid}) = M 1 × len ( ParamGrid ) = M models.

Evaluation Variance Low (averages metrics across all K K K holdout folds). Higher (single validation split is sensitive to partitioning noise).

Computational Cost High ( K × K \times K × more expensive in CPU hours and cluster time). Low (evaluates parameter grid in a single pass).

Best For Small to medium datasets where robust validation is critical. Massive terabyte-scale datasets where K-fold CV is computationally prohibitive.

Benefits and Downsides, Stated Plainly

  • K-fold CrossValidator TrainValidationSplit
  • Models fitted numFolds × len(paramGrid) 1 × len(paramGrid)
  • Estimate variance Low — averages K holdout scores Higher — one split's composition drives the result
  • Uses all data for validation Yes, every row is validated exactly once No, only the holdout fraction
  • Compute cost K× 1×
  • Reports a spread Yes — the standard deviation across folds is itself diagnostic No

Best for Small to medium data; when the decision between candidates is close Very large data; expensive models; a first coarse sweep

  • The underrated benefit of K-fold is the standard deviation across folds . Two
  • candidate models with the same mean score but very different fold spreads are not
  • equally good; the high-variance one is unstable and will behave unpredictably in

production. A single split cannot reveal that at all.

  • The underrated cost is that K-fold multiplies every expense: cluster hours, wall-clock
  • time, and — with a large parameter grid — the risk of overfitting the validation

procedure itself by selecting from too many candidates.

A practical progression

Sweep coarsely with TrainValidationSplit to eliminate obviously poor regions.

Refine the survivors with CrossValidator ( numFolds=3 or 5 ).

Retrain the winning configuration on the full training pool.

Evaluate once on the untouched holdout test set.

  • Worked Implementation: CrossValidator over a Spark ML Pipeline from pyspark.ml import Pipeline
  • from pyspark.ml.feature import VectorAssembler, StandardScaler
  • from pyspark.ml.classification import LogisticRegression
  • from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
  • from pyspark.ml.evaluation import BinaryClassificationEvaluator
  • df = spark.table("lakehouse_gold.customer_features")
  • train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)
  • # Pipeline definition
  • assembler = VectorAssembler(inputCols=["f1", "f2", "f3"], outputCol="raw_features")
  • scaler = StandardScaler(inputCol="raw_features", outputCol="features")
  • lr = LogisticRegression(featuresCol="features", labelCol="label")
  • pipeline = Pipeline(stages=[assembler, scaler, lr])
  • # Build hyperparameter grid
  • param_grid = (
  • ParamGridBuilder()
  • .addGrid(lr.regParam, [0.01, 0.1, 1.0])
  • .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])
  • .build()
  • )
  • evaluator = BinaryClassificationEvaluator(
  • labelCol="label",
  • rawPredictionCol="rawPrediction",
  • metricName="areaUnderROC"
  • )
  • # Instantiate CrossValidator (5 folds * 9 param combinations = 45 models)
  • crossval = CrossValidator(
  • estimator=pipeline,
  • estimatorParamMaps=param_grid,
  • evaluator=evaluator,
  • numFolds=5,
  • parallelism=4,
  • seed=42
  • )

cv_model = crossval.fit(train_df)

  • # Evaluate best model on isolated holdout test set
  • test_predictions = cv_model.transform(test_df)
  • test_auc = evaluator.evaluate(test_predictions)
  • print(f"Isolated Holdout Test ROC-AUC: {test_auc:.4f}")
  • Stratified and Time-Ordered Variants
  • Stratified K-fold preserves the class proportions in every fold. With a 2%
  • positive rate, plain random folds can produce a fold with almost no positives, making
  • its score meaningless. scikit-learn's StratifiedKFold does this directly; in Spark
  • ML the usual approach is to stratify the split with sampleBy on the label before

handing folds to the tuner.

  • Walk-forward validation is mandatory for temporally ordered data. Random folds
  • place future rows in the training set and past rows in validation, which leaks the
  • future into the past and produces validation scores that never reproduce in

production.

Strict Holdout Test Set Isolation

A golden rule in production machine learning is the strict isolation of the holdout test set: +-----------------------------------------------------------------------------+

  • | STRICT DATA SPLITTING PROTOCOL |
  • | |
  • | FULL DATASET (Delta Table) |
  • | | |
  • | +----------------------------+-----------------------------+ |
  • | | (80% Training Pool) | (20%) |
  • | v v |
  • | TRAIN / VALIDATION POOL HOLDOUT TEST SET|
  • | +---------------------------------------------------+ +-------------+ |
  • | | CrossValidator / Hyperopt Tuning Executes HERE: | | STRICTLY | |
  • | | - Fold 1..K Train / Validation | | ISOLATED | |
  • | | - Feature Scaler & Imputer Fit | | DO NOT TOUCH| |
  • | | - Optimal Hyperparameters Selected | | DURING | |
  • | +---------------------------------------------------+ | TUNING! | |
  • | | | |
  • | v | |
  • | FINAL RETRAINED BEST MODEL | |
  • | | | |
  • | +-------------------> [ EVALUATE ONCE ] <------------------+ |
  • | | |
  • | v |
  • | UNBIASED GENERALIZATION SCORE |
  • +-----------------------------------------------------------------------------+