Study guide
3.14 Model Complexity and the Bias-Variance Tradeoff
Databricks Certified Machine Learning Associate Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Databricks Machine Learning (38% of the Exam)
1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion
2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)
2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate
3 Chapter 3: Model Development (31% of the Exam)
3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff
4 Chapter 4: Model Deployment (12% of the Exam)
4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints
3.14 Model Complexity and the Bias-Variance Tradeoff
Key Takeaways Explain This Topic AI
Expected generalisation error decomposes into bias squared, variance, and irreducible noise; complexity trades the first against the second.
Underfitting (high bias) shows high training error, high validation error, and a small gap between them; overfitting (high variance) shows very low training error, high validation error, and a wide gap.
Increase capacity, add features, or reduce regularisation for high bias; add data, regularise, constrain depth, or apply early stopping for high variance.
L1 (Lasso) drives redundant coefficients to exactly zero; L2 (Ridge) shrinks all coefficients smoothly and stabilises multicollinearity.
Spark ML sets strength with regParam and the L1/L2 mix with elasticNetParam (0.0 = ridge, 1.0 = lasso); scikit-learn's C is inverted, so a smaller C means stronger regularisation.
Last updated: August 2026
- 3.14 Model Complexity and the Bias-Variance Tradeoff
- Bias-Variance Tradeoff & Learning Curve Diagnostics
The expected generalization error of any supervised model decomposes mathematically into three distinct components: Expected Error = Bias 2 + Variance + σ irreducible 2 \text{Expected Error} = \text{Bias}^2 + \text{Variance} + \sigma^2_{\text{irreducible}} Expected Error = Bias 2 + Variance + σ irreducible 2 +-----------------------------------------------------------------------------+
- | BIAS VS. VARIANCE LEARNING CURVES |
- | |
- | UNDERFITTING (HIGH BIAS) OVERFITTING (HIGH VARIANCE) |
- | Error Error |
- | ^ ^ |
- | | --- Validation Error | --- Validation Error |
- | | === Training Error | |
- | | | <--- LARGE GAP ---> |
- | | ------------------- (High Error) | ------------------- |
- | | =================== (High Error) | |
- | | <-- SMALL GAP --> | =================== (Low Error)|
- | +----------------------------> +----------------------------> |
- | 0 Training Set Size (N) 0 Training Set Size (N) |
- | |
- | Symptoms: Symptoms: |
- | - Training error is high. - Training error is near zero. |
- | - Validation error is high. - Validation error is high. |
- | - Small gap between curves. - Massive gap between curves. |
- +-----------------------------------------------------------------------------+
- Comprehensive Diagnostic and Remediation Matrix
Diagnostic State Primary Cause Observed Symptoms Remediation Strategies
Underfitting (High Bias) Model is too simple or constrained to capture underlying data structure. High training error, high validation error, small gap between train and validation performance. 1. Increase model capacity (deeper trees, polynomial features, neural layers).; 2. Engineer informative interaction features.; 3. Reduce regularization penalties (decrease λ \lambda λ , reduce reg_alpha / reg_lambda ).; 4. Switch to more expressive algorithm (e.g. from linear model to XGBoost).
Overfitting (High Variance) Model memorizes training sample noise and idiosyncrasies rather than true population signals. Very low training error, high validation error, large generalization gap between curves. 1. Acquire more training data.; 2. Apply feature selection / dimensionality reduction.; 3. Increase regularization penalties (increase L1/L2 penalties).; 4. Constrain tree complexity ( max_depth , min_samples_leaf , subsample ).; 5. Apply early stopping during boosting iterations.
Optimal Fit Balanced model complexity. Low training error, low validation error, narrow generalization gap. Maintain configuration; validate on untouched holdout test set.
- Reading the Diagnosis from Two Numbers
- Almost every bias-variance question reduces to comparing training error against
validation error:
- Training error Validation error Gap Diagnosis First move
- High High Small Underfitting / high bias More capacity, better features, less regularisation
- Very low High Large Overfitting / high variance More data, more regularisation, less capacity, early stopping
- Low Low Small Good fit Confirm on the untouched test set
- Low Much lower than training Negative Leakage or a broken split Investigate before anything else
- That last row is worth remembering: validation error meaningfully below training
- error is not a triumph, it is a bug — usually a target leak or an accidental overlap
between the splits.
- What more data does, and does not, fix
- Adding training rows reduces variance and therefore narrows the gap. It does
- essentially nothing for bias : a linear model on a genuinely non-linear relationship
- will fit that relationship no better with ten times the data. This is why the correct
answer to "training and validation error are both high" is never "collect more data".
Complexity knobs by model family
- Family Increase capacity (less bias) Decrease capacity (less variance)
- Linear models Add polynomial or interaction features; lower regParam Raise regParam ; drop features
- Decision trees Raise maxDepth ; lower minInstancesPerNode Lower maxDepth ; prune; raise the minimum leaf size
- Random forest Deeper trees; more features per split More trees (also stabilises); shallower trees
Gradient boosting More rounds; deeper trees; higher learning rate Early stopping; subsample ; colsample_bytree ; reg_alpha / reg_lambda
- Neural networks More layers or units Dropout; weight decay; early stopping
- Random forests are the instructive case: adding trees reduces variance without adding
- bias, because predictions are averaged over independent bagged learners. Adding boosting
- rounds does the opposite — it reduces bias and eventually increases variance, which is
why early stopping exists for boosting but not for bagging.
Regularization Mechanics: L1 (Lasso) vs. L2 (Ridge) vs. ElasticNet
Regularization injects a penalty term Ω ( w ) \Omega(\mathbf{w}) Ω ( w ) into the loss function L ( w ) \mathcal{L}(\mathbf{w}) L ( w ) to constrain coefficient magnitudes: min w L ( w ) + λ Ω ( w ) \min_{\mathbf{w}} \mathcal{L}(\mathbf{w}) + \lambda \Omega(\mathbf{w}) w min L ( w ) + λ Ω ( w )
- Property L1 (Lasso) L2 (Ridge)
- Penalty term Sum of absolute weights Sum of squared weights
- Constraint geometry Diamond, with corners sitting on the axes Sphere, smooth everywhere
- Effect on coefficients Drives redundant weights to exactly zero Shrinks every weight smoothly toward zero
- Side effect Embedded feature selection; a sparse model Stable behaviour under multicollinearity
- Prefer when Many features, most believed irrelevant Correlated features you want to keep but damp
- ElasticNet mixes the two penalties. It is the usual choice when features are both
- numerous and correlated: the L1 component removes the clearly useless ones while the L2
component stops the correlated survivors from thrashing between fits.
- Setting the Knobs in Code
- The parameter names differ between the two libraries this exam uses, and one of them
runs in the opposite direction:
Library and class Strength parameter Mix parameter Direction
Spark ML LinearRegression , LogisticRegression regParam elasticNetParam Higher regParam means more regularisation
- scikit-learn Ridge , Lasso , ElasticNet alpha l1_ratio Higher alpha means more regularisation
- scikit-learn LogisticRegression , SVC C penalty Higher C means less regularisation
- In Spark ML, elasticNetParam=0.0 is pure L2 (ridge), elasticNetParam=1.0 is pure L1
(lasso), and anything between is ElasticNet. In scikit-learn's LogisticRegression , C is the inverse of regularisation strength, so C=0.01 is a heavily regularised model
and C=100 is very nearly unregularised. A scenario stating that an analyst raised C to cure overfitting is describing a change that makes the overfitting worse.
A Worked Diagnosis
A random forest is evaluated on a churn dataset at three depths:
- Configuration Train AUC Validation AUC Gap
- maxDepth=3 0.71 0.70 0.01
- maxDepth=8 0.86 0.84 0.02
- maxDepth=20 0.99 0.79 0.20
- Depth 3 underfits: both numbers are low and nearly identical, so the model is
- capacity-limited and additional rows would not move it. Depth 20 overfits: training AUC
- is almost perfect while validation falls back below the depth-8 result, and the 0.20 gap
- is the variance made visible. Depth 8 is the configuration to select, and the honest
- estimate of its performance comes from a test set that played no part in this comparison
— reusing the validation set to both select and report inflates the number you publish.
A caution on the classic U-curve. The conceptual chart below shows total error
falling and then rising as complexity grows, which is the behaviour this exam tests.
- Very heavily over-parameterised models can show a second descent past the
- interpolation point, but that phenomenon sits outside this blueprint — answer the
classic tradeoff.