ExamFinch.com My Study

Study guide

3.10 Using Common Classification Metrics

Practice this exam

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

3.10 Using Common Classification Metrics

Key Takeaways Explain This Topic AI

Precision is T P / ( T P + F P ) TP/(TP+FP) TP / ( TP + FP ) — the cost of false alarms; recall is T P / ( T P + F N ) TP/(TP+FN) TP / ( TP + FN ) — the cost of misses; F1 is their harmonic mean.

F β F_\beta F β ​ tilts the balance: F 2 F_2 F 2 ​ weights recall twice as heavily as precision, F 0.5 F_{0.5} F 0.5 ​ does the reverse.

Log loss scores calibrated probabilities rather than thresholded labels and punishes confident wrong predictions severely.

ROC-AUC measures ranking quality across all thresholds; PR-AUC does the same while ignoring true negatives, which is why it is preferred for rare events.

Spark ML computes these through BinaryClassificationEvaluator ( areaUnderROC , areaUnderPR ) and MulticlassClassificationEvaluator ( f1 , accuracy , weightedPrecision , weightedRecall , logLoss ).

Last updated: August 2026

3.10 Using Common Classification Metrics

Model evaluation is the objective mathematical discipline of assessing how effectively a trained algorithm generalizes to unseen data, meets business operational requirements, and balances structural errors. Selecting inappropriate metrics or misinterpreting learning curve diagnostics can lead to deploying poorly performing models into mission-critical production systems. This section details the complete catalog of classification and regression evaluation metrics, the nuances of evaluating models with transformed targets, and systematic protocols for diagnosing and remediating bias and variance.

Classification Evaluation Metric Framework

Binary and multiclass classification models output discrete class assignments or continuous probability vectors that must be evaluated against ground-truth labels using a confusion matrix: +-----------------------------------------------------------------------------+

  • | CONFUSION MATRIX TAXONOMY |
  • | |
  • | ACTUAL POSITIVE (1) ACTUAL NEGATIVE (0) |
  • | PREDICTED POSITIVE (1) True Positive (TP) False Positive (FP) |
  • | Correct Detection Type I Error (Alarm) |
  • | |
  • | PREDICTED NEGATIVE (0) False Negative (FN) True Negative (TN) |
  • | Type II Error (Missed) Correct Rejection |
  • +-----------------------------------------------------------------------------+
  • Mathematical Metric Formulations

Accuracy: Measures overall proportion of correct classifications: Accuracy = TP + TN TP + TN + FP + FN \text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} Accuracy = TP + TN + FP + FN TP + TN ​

Limitation: Highly misleading under class imbalance.

Precision (Positive Predictive Value): Measures the accuracy of positive predictions: Precision = TP TP + FP \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} Precision = TP + FP TP ​

Optimization Goal: Maximize when False Positives are costly (e.g., spam filtering, VIP transaction freezing).

Recall / Sensitivity (True Positive Rate): Measures the proportion of actual positives successfully identified: Recall = TP TP + FN \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} Recall = TP + FN TP ​

Optimization Goal: Maximize when False Negatives are costly (e.g., medical pathology diagnosis, critical equipment failure).

Specificity (True Negative Rate): Measures the proportion of actual negatives correctly rejected: Specificity = TN TN + FP \text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} Specificity = TN + FP TN ​

F1-Score: Harmonic mean of precision and recall, balancing both dimensions: F 1 = 2 ⋅ Precision ⋅ Recall Precision + Recall = 2 TP 2 TP + FP + FN F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2\text{TP}}{2\text{TP} + \text{FP} + \text{FN}} F 1 ​ = 2 ⋅ Precision + Recall Precision ⋅ Recall ​ = 2 TP + FP + FN 2 TP ​

F β F_\beta F β ​ -Score: Weighted harmonic mean that assigns β \beta β times as much importance to recall as to precision: F β = ( 1 + β 2 ) ⋅ Precision ⋅ Recall β 2 Precision + Recall F_\beta = (1 + \beta^2) \cdot \frac{\text{Precision} \cdot \text{Recall}}{\beta^2 \text{Precision} + \text{Recall}} F β ​ = ( 1 + β 2 ) ⋅ β 2 Precision + Recall Precision ⋅ Recall ​

F 2 F_2 F 2 ​ Score ( β = 2 \beta=2 β = 2 ): Weights recall twice as heavily as precision (ideal for fraud and medical detection).

F 0.5 F_{0.5} F 0.5 ​ Score ( β = 0.5 \beta=0.5 β = 0.5 ): Weights precision twice as heavily as recall (ideal for automated trading or spam filters).

Log Loss / Cross-Entropy: Evaluates calibrated probability outputs rather than hard thresholded labels: LogLoss = − 1 N ∑ i = 1 N [ y i ln ⁡ ( p ^ i ) + ( 1 − y i ) ln ⁡ ( 1 − p ^ i ) ] \text{LogLoss} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \ln(\hat{p}_i) + (1-y_i) \ln(1-\hat{p}_i) \right] LogLoss = − N 1 ​ i = 1 ∑ N ​ [ y i ​ ln ( p ^ ​ i ​ ) + ( 1 − y i ​ ) ln ( 1 − p ^ ​ i ​ ) ]

Property: Heavily penalizes confident incorrect predictions (e.g., predicting p ^ = 0.99 \hat{p}=0.99 p ^ ​ = 0.99 when actual y = 0 y=0 y = 0 ).

Threshold Metrics vs. Ranking Metrics

Classification metrics fall into two families, and mixing them up is a frequent error.

Family Depends on a threshold? Members Answers

Threshold metrics Yes — computed from hard labels Accuracy, precision, recall, specificity, F1, F β F_\beta F β ​ "How good are the decisions at this cutoff?"

Ranking metrics No — computed across all cutoffs ROC-AUC, PR-AUC (average precision) "How well does the model order positives above negatives?"

Probability metrics No — computed from the probabilities themselves Log loss, Brier score "How well calibrated are the probabilities?"

Two practical consequences:

  • A model can have excellent ROC-AUC and terrible F1 simply because the default 0.5
  • threshold is wrong for its calibration. The remedy is threshold tuning (Section 3.2),

not retraining.

Reporting accuracy on an imbalanced problem is meaningless: predicting the majority

class for everything already achieves the prevalence rate.

Computing Them on Databricks from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator

  • roc = BinaryClassificationEvaluator(labelCol="label", rawPredictionCol="rawPrediction",
  • metricName="areaUnderROC").evaluate(preds)
  • pr = BinaryClassificationEvaluator(labelCol="label", rawPredictionCol="rawPrediction",
  • metricName="areaUnderPR").evaluate(preds)
  • f1 = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction",
  • metricName="f1").evaluate(preds)

Note which column each evaluator consumes. BinaryClassificationEvaluator needs rawPredictionCol (scores or probabilities) because it sweeps thresholds; MulticlassClassificationEvaluator needs predictionCol (hard labels) because its

metrics are threshold metrics. Passing the wrong column is a classic snippet bug.

Multiclass averaging

For more than two classes, precision and recall must be averaged across classes:

Macro — unweighted mean over classes; every class counts equally, so rare classes

matter as much as common ones.

Weighted — mean weighted by class support; dominated by the common classes. This

is what Spark's weightedPrecision and weightedRecall compute.

Micro — pooled counts across classes; for single-label multiclass this equals

accuracy.

  • Worked Implementation: Classification Metrics in scikit-learn import numpy as np
  • from sklearn.metrics import (
  • confusion_matrix,
  • precision_score,
  • recall_score,
  • f1_score,
  • fbeta_score,
  • roc_auc_score,
  • average_precision_score
  • )
  • y_true = np.array([0, 1, 0, 0, 1, 1, 0, 1, 0, 1])
  • y_pred = np.array([0, 1, 0, 0, 0, 1, 0, 1, 1, 1])
  • y_prob = np.array([0.1, 0.9, 0.2, 0.3, 0.45, 0.85, 0.15, 0.78, 0.65, 0.92])
  • cm = confusion_matrix(y_true, y_pred)
  • prec = precision_score(y_true, y_pred)
  • rec = recall_score(y_true, y_pred)
  • f1 = f1_score(y_true, y_pred)
  • f2 = fbeta_score(y_true, y_pred, beta=2.0) # Recall weighted 2x over precision
  • roc_auc = roc_auc_score(y_true, y_prob)
  • pr_auc = average_precision_score(y_true, y_prob)
  • print("Confusion Matrix:\n", cm)
  • print(f"Precision: {prec:.3f}, Recall: {rec:.3f}, F1: {f1:.3f}, F2: {f2:.3f}")
  • print(f"ROC-AUC: {roc_auc:.3f}, PR-AUC: {pr_auc:.3f}")