3.11 Using Common Regression Metrics

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

3.11 Using Common Regression Metrics

Key Takeaways Explain This Topic AI

RMSE is expressed in the target's own units and penalises large errors quadratically, so a few big misses dominate it.

MAE is also in target units but penalises linearly, making it the robust choice when outliers should not dominate the score.

R 2 R^2 R 2 is the fraction of variance explained relative to predicting the mean; it is unitless, can be negative, and is the only one of the three that is comparable across datasets.

MAPE expresses error as a percentage of the actual value, which breaks down when actual values approach zero.

RegressionEvaluator computes rmse (default), mse , mae , r2 , and mape from a predictions DataFrame.

Last updated: August 2026

• 3.11 Using Common Regression Metrics
• Regression Evaluation Metric Framework

Regression models estimate continuous targets y ∈ R y \in \mathbb{R} y ∈ R . Evaluation metrics quantify the distribution and magnitude of residuals e i = y i − y ^ i e_i = y_i - \hat{y}_i e i ​ = y i ​ − y ^ ​ i ​ : +-----------------------------------------------------------------------------+

• | REGRESSION METRIC COMPARISON |
• | |
• | METRIC FORMULA PROPERTIES / USAGE |
• | ------ ----------------------------------- -------------------------- |
• | MSE (1/n) * sum( (y_i - y_hat_i)^2 ) Penalizes large outlier |
• | errors quadratically; units|
• | are squared. |
• | |
• | RMSE sqrt( MSE ) Same units as target; |
• | sensitive to large outliers|
• | Standard ML benchmark. |
• | |
• | MAE (1/n) * sum( |y_i - y_hat_i| ) Same units as target; |
• | linear penalty; highly |
• | robust to extreme outliers.|
• | |
• | R^2 1 - [ SS_res / SS_tot ] Fraction of variance |
• | 1 - [ sum((y-y_hat)^2)/sum((y-y_bar)^2)] explained by model; |
• | <= 1.0 (1.0 = perfect, |
• | 0.0 = baseline mean, <0 bad|
• | |
• | MAPE (100%/n) * sum( |(y-y_hat)/y| ) Scale-independent relative |
• | percentage error. |
• +-----------------------------------------------------------------------------+
• What Each Metric Actually Penalises

• Metric Units Outlier sensitivity Comparable across datasets? Reach for it when
• MSE Target² Very high No You need a differentiable training loss
• RMSE Target High No Large errors are disproportionately costly
• MAE Target Low No All errors cost in proportion to their size
• R 2 R^2 R 2 None Moderate Yes Communicating explanatory power to stakeholders
• MAPE Percent Moderate Yes Errors are naturally relative, and actuals stay well above zero
• RMSE versus MAE, concretely

Take two models on ten predictions. Model A misses by 10 units on every prediction.

Model B is perfect on nine and misses by 100 on the tenth.

MAE : A = 10.0, B = 10.0. The two look identical.

RMSE : A = 10.0, B = 10,000 / 10 = 31.6 \sqrt{10{,}000/10} = 31.6 10 , 000/10 ​ = 31.6 . B is punished three times harder.

• Because RMSE ≥ MAE \text{RMSE} \ge \text{MAE} RMSE ≥ MAE always, the gap between them is itself a
• diagnostic: a large gap means the error distribution has a heavy tail. Which model is
• better is a business decision — if one catastrophic miss is far worse than ten small

ones, RMSE captures it; if cost accrues linearly with error, MAE is the honest score.

R 2 R^2 R 2 pitfalls R 2 = 1 − ∑ ( y i − y ^ i ) 2 ∑ ( y i − y ˉ ) 2 R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2} R 2 = 1 − ∑ ( y i ​ − y ˉ ​ ) 2 ∑ ( y i ​ − y ^ ​ i ​ ) 2 ​

R 2 = 0 R^2 = 0 R 2 = 0 means the model does no better than predicting the training mean.

R 2 < 0 R^2 < 0 R 2 < 0 is possible on a test set and means the model is worse than that baseline.

R 2 R^2 R 2 rises mechanically as features are added, which is why adjusted R 2 R^2 R 2 exists for

model comparison at differing feature counts.

Because the denominator is the variance of the actuals, R 2 R^2 R 2 is not comparable across

datasets with different target spreads even though it is unitless.

MAPE's failure mode MAPE = 100 n ∑ ∣ y i − y ^ i y i ∣ \text{MAPE} = \frac{100}{n}\sum \left| \frac{y_i - \hat{y}_i}{y_i} \right| MAPE = n 100 ​ ∑ ​ y i ​ y i ​ − y ^ ​ i ​ ​ ​

• Every term divides by the actual value, so a single row with y i y_i y i ​ near zero can send
• MAPE to thousands of percent, and y i = 0 y_i = 0 y i ​ = 0 makes it undefined. For demand data with
• frequent zeros, prefer SMAPE or a weighted absolute-percentage error — which is why

Databricks AutoML uses SMAPE as its forecasting default.

Computing Them from pyspark.ml.evaluation import RegressionEvaluator

for metric in ["rmse", "mae", "r2", "mse", "mape"]:

• value = RegressionEvaluator(labelCol="price", predictionCol="prediction",
• metricName=metric).evaluate(preds)
• print(f"{metric}: {value:.4f}")
• RegressionEvaluator defaults to rmse when metricName is omitted — worth

remembering when a code snippet leaves it out.

In scikit-learn the equivalents live in sklearn.metrics : from sklearn.metrics import (root_mean_squared_error, mean_squared_error,

• mean_absolute_error, r2_score,
• mean_absolute_percentage_error)

• rmse = root_mean_squared_error(y_test, y_pred) # current API
• mse = mean_squared_error(y_test, y_pred) # returns MSE, not RMSE
• mae = mean_absolute_error(y_test, y_pred)
• r2 = r2_score(y_test, y_pred)
• mape = mean_absolute_percentage_error(y_test, y_pred) # a fraction, not a percent
• Two API details matter. mean_squared_error returns MSE , not RMSE — older code
• reached for squared=False , but that argument was deprecated in favour of the
• dedicated root_mean_squared_error function, so take the square root explicitly or use
• the newer function. And mean_absolute_percentage_error returns a fraction (0.12),

not a percentage (12%), so reports must scale it.

• Reporting Regression Results Honestly
• Always state the units. "RMSE 42,000" is meaningless without "dollars", and it is

meaningless in a different way if the model was trained on a log target (Section 3.13).

Quote a baseline. The mean predictor's RMSE — the standard deviation of the target —

is the number a model must beat, and it is what R 2 = 0 R^2 = 0 R 2 = 0 corresponds to.

Report more than one metric. RMSE and MAE together reveal the shape of the error

distribution, and R 2 R^2 R 2 adds a scale-free view. A single metric hides the tail.

Evaluate on the untouched holdout. Cross-validated scores are used for model selection ; the final number reported to stakeholders comes from the test split

scored exactly once.