3.12 Choosing the Most Appropriate Metric for the Objective

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

3.12 Choosing the Most Appropriate Metric for the Objective

Key Takeaways Explain This Topic AI

Start from the asymmetry of the business costs: whichever error type is more expensive determines whether precision, recall, or a balance is optimised.

When false negatives dominate the cost — disease screening, fraud, equipment failure — maximise recall or F 2 F_2 F 2 ​ .

When false positives dominate — spam filtering, automated account closure, costly interventions — maximise precision or F 0.5 F_{0.5} F 0.5 ​ .

For rare events, prefer PR-AUC over ROC-AUC and never report raw accuracy; for well-calibrated probabilities used in expected-value decisions, use log loss.

For regression, pick RMSE when large errors are disproportionately costly, MAE when cost is proportional to error, and R 2 R^2 R 2 only for communicating explanatory power.

Last updated: August 2026

• 3.12 Choosing the Most Appropriate Metric for the Objective
• Metric selection is not a statistical preference — it is a translation of the business
• cost structure into a single number the tuning loop can optimise. The exam poses this

as a scenario, and the scenario always contains the decisive clue.

The Decision Procedure

Classification or regression? Determined by the target.

What does each error type cost? Write down the cost of a false positive and of a

false negative. Whichever is larger is what the metric must punish.

How rare is the positive class? Below roughly 5–10% prevalence, accuracy and

ROC-AUC both flatter the model; use PR-AUC and F β F_\beta F β ​ .

• Is a hard decision made, or is the score consumed downstream? A hard decision
• calls for a threshold metric; a score feeding an expected-value calculation calls for

a calibration metric such as log loss.

Is the metric being reported to stakeholders? Prefer one expressed in units they

already reason about — dollars, minutes, percentage points.

Scenario to Metric

Scenario Dominant cost Metric

Cancer screening: a missed tumour is fatal, a false alarm means one more blood test False negatives Recall , or F 2 F_2 F 2 ​

Spam filter: a lost legitimate email is far worse than a spam message getting through False positives Precision , or F 0.5 F_{0.5} F 0.5 ​

Fraud detection at 0.2% prevalence, blocking a card costs customer friction Both, asymmetric PR-AUC to compare models, F β F_\beta F β ​ at the chosen threshold

• Credit limit decisions where the score feeds an expected-loss calculation Probability quality Log loss
• Ranking a recommendation slate — only the ordering is used Ordering quality ROC-AUC or ranking metrics
• Balanced classes, symmetric costs Neither Accuracy or F1
• Predicting house prices where a large miss is disproportionately costly Large errors RMSE
• Predicting delivery times where cost accrues per minute late Proportional MAE
• Explaining to executives how much of price variation the model captures Communication R 2 R^2 R 2
• Demand forecasting across products at very different scales Relative error SMAPE (MAPE breaks near zero)
• Traps
• Accuracy under imbalance. At 99.8% negatives, "predict negative always" scores

99.8%. Any answer choosing accuracy on a rare-event problem is wrong.

ROC-AUC under extreme imbalance. The false-positive-rate denominator is F P + T N FP + TN FP + TN ; with an enormous T N TN TN , hundreds of false positives barely move it. PR-AUC

ignores true negatives entirely and exposes the problem.

Optimising a metric you do not act on. If the deployed system applies a fixed

threshold, tuning to ROC-AUC optimises a property the system never uses.

Metrics on a transformed target. RMSE on ln ⁡ ( y + 1 ) \ln(y+1) ln ( y + 1 ) is not RMSE in dollars — see

Section 3.13.

MAPE with near-zero actuals. Mathematically undefined at zero and explosive near

it.

Comparing R 2 R^2 R 2 across datasets. The denominator is dataset-specific, so a higher R 2 R^2 R 2 on a different dataset does not mean a better model.

Connecting the Metric to the Tuning Loop

Whatever you choose has to become the objective the search minimises: # Hyperopt minimises, so a metric to maximise is negated

def objective(params):

• model = fit(params)
• f2 = fbeta_score(y_val, model.predict(X_val), beta=2.0) # recall-weighted

return {"loss": -f2, "status": STATUS_OK} # Spark ML: the Evaluator passed to CrossValidator IS the selection criterion

• evaluator = BinaryClassificationEvaluator(labelCol="label",
• rawPredictionCol="rawPrediction",
• metricName="areaUnderPR")
• A frequent real-world failure — and a plausible exam distractor — is selecting the best

model by one metric while reporting another. The evaluator handed to CrossValidator is what decides bestModel , so it must be the metric the business actually cares

about.

One Metric to Optimise, Several to Watch

Tuning requires a single scalar, but reporting should not be reduced to one number.

A workable convention is to nominate one optimisation metric and a set of guardrail metrics that must stay within agreed bounds:

• Role Example on a fraud model Behaviour
• Optimisation metric PR-AUC Maximised by the search
• Guardrail False positives per 10,000 transactions Must stay under the operations team's review capacity
• Guardrail p95 inference latency Must stay under the payment authorisation budget
• Guardrail Recall on the highest-value segment Must not regress against the incumbent

A candidate that wins on the optimisation metric but breaches a guardrail is rejected.

This is exactly the shape of the automated validation gate described in Section 1.1,

and it is why metric choice is an MLOps decision as much as a statistical one.

• Translating a stated cost into β \beta β
• F β F_\beta F β ​ treats recall as β \beta β times as important as precision, so the direction

of the cost asymmetry picks β \beta β directly:

• Stated cost relationship Choose
• False negatives are much more expensive β > 1 \beta > 1 β > 1 — typically F 2 F_2 F 2 ​
• Costs are roughly symmetric β = 1 \beta = 1 β = 1 — the ordinary F 1 F_1 F 1 ​
• False positives are much more expensive β < 1 \beta < 1 β < 1 — typically F 0.5 F_{0.5} F 0.5 ​
• The exam does not ask for a numeric β \beta β ; it asks you to read the asymmetry in the

scenario and pick the metric that punishes the expensive error.