7.1 Evaluation Metrics for Predictive Models

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

7.1 Evaluation Metrics for Predictive Models

Key Takeaways Explain This Topic AI

Precision is TP / (TP + FP), recall is TP / (TP + FN), and F1 is their harmonic mean, which balances both on imbalanced data.

AUC PR is more informative than AUC ROC when the positive class is rare, because ROC curves can look strong when negatives dominate.

RMSE penalizes large errors more heavily than MAE, and MAPE is undefined when any actual value is zero.

Agent Platform model evaluation compares batch inference results against ground-truth data and can run from Model Registry or as a pipeline component.

Raising a classifier's confidence threshold generally increases precision and lowers recall.

Last updated: September 2026

Section 2.3 of the exam guide includes evaluating predictive and gen AI solutions (for example, model evaluation metrics and LLM-as-a-judge) . This section covers predictive models. Section 7.2 covers gen AI. On the exam, the right metric is the one that matches the business cost of errors and the data distribution , not the one with the most familiar name.

Classification: The Confusion Matrix First

Predicted positive Predicted negative
Actual positive True positive (TP) False negative (FN)
Actual negative False positive (FP) True negative (TN)

Metric Formula Answers
Accuracy (TP + TN) / total How often is the model right overall? Misleading with imbalanced classes

Precision TP / (TP + FP) When the model flags something, how often is it correct?

Recall (true positive rate) TP / (TP + FN) Of all real positives, how many did the model catch?

F1 2 × P × R / (P + R) A balance of precision and recall

False positive rate FP / (FP + TN) How often are negatives wrongly flagged?

Worked example: why accuracy misleads

A fraud model scores 10,000 transactions, and 100 are truly fraudulent. It flags 150, of which 80 are real fraud.

TP = 80, FP = 70, FN = 20, TN = 9,830
Accuracy = (80 + 9,830) / 10,000 = 99.1%
Precision = 80 / 150 = 53.3%
Recall = 80 / 100 = 80.0%
F1 = 2 × 0.533 × 0.800 / (0.533 + 0.800) = 0.64

A model that flags nothing reaches 99.0% accuracy and catches zero fraud. With rare positives, look at precision, recall, F1, and AUC PR , not accuracy.

Threshold-Independent Metrics

Metric What it summarizes Use when

AUC ROC True positive rate vs. false positive rate across all thresholds Classes are reasonably balanced, or you care about ranking quality overall

AUC PR (average precision) Precision vs. recall across thresholds The positive class is rare (fraud, defects, churn). Less inflated by many true negatives

Log loss Penalty for confident wrong probabilities Probabilities themselves are used, as in pricing and risk scores

Choosing the threshold

Most classifiers output a score. The confidence threshold turns it into a decision. Raising it usually raises precision and lowers recall . Pick it from business costs:

Missed fraud costs $500 and a false alarm costs $5 in review time : favor recall and accept a lower threshold.

An email marketing model where every false positive annoys a customer : favor precision.

Regulated screening with a required minimum recall (for example, 95%): choose the highest precision that still meets the recall floor. That is exactly what AutoML's "precision at recall" objective optimizes.

Multi-class problems report per-class precision and recall plus micro averages (pooled counts, dominated by large classes) and macro averages (average of per-class scores, which treats rare classes equally). Agent Platform also reports "at 1" metrics, which consider only the top-scoring label.

Regression Metrics

Metric Behavior Choose when
MAE Average absolute error, in the target's units Errors should count linearly. Robust to outliers
RMSE Square root of mean squared error Large errors are especially costly

RMSLE RMSE on log(1 + value), penalizing under-prediction more Relative error matters and targets span orders of magnitude. Needs non-negative values

R² Share of variance explained (0-1 in Agent Platform's definition) Comparing fit quality across models
MAPE Average absolute percentage error Stakeholders think in percentages. Undefined when any actual value is 0

Worked example: errors of +2, -2, and +10 give MAE = (2 + 2 + 10) / 3 = 4.67 and RMSE = √((4 + 4 + 100) / 3) = √36 = 6.0 . The one large error pulls RMSE well above MAE.

Forecasting, Ranking, and Clustering

Forecasting adds WAPE (total absolute error divided by total actuals, stable when many values are small or intermittent), RMSPE , and quantile metrics for probabilistic forecasts.

Recommendation and ranking: BigQuery ML matrix factorization with implicit feedback reports recall, mean squared error, normalized discounted cumulative gain (NDCG) , and average rank. NDCG rewards putting relevant items near the top of a list.

Clustering: BigQuery ML k-means reports the Davies-Bouldin index (lower is better) and mean squared distance to centroids. Clusters still need a business check to confirm they're meaningful segments.

Beyond Aggregate Metrics

Slice metrics: evaluate by region, device, language, or demographic group. An overall AUC of 0.90 can hide a segment at 0.70 (fairness, Chapter 18).

Baseline comparison: always compare with a simple model and the current production model on the same test data .

Calibration: check that predicted probabilities match observed rates when scores drive pricing or risk.

Offline vs. online: offline metrics decide whether a model is ready to test. Online A/B tests measure the business KPI (Chapter 13).

Common Metric Traps on the Exam

Trap Better choice
Reporting accuracy for a 1%-positive fraud model Precision, recall, F1, AUC PR
Using AUC ROC to compare models on a very rare class AUC PR
Optimizing MAE when a few huge misses cause outages RMSE
Reporting MAPE on intermittent demand with zero days WAPE or MAE

Tuning the threshold on the test set, then reporting test metrics Choose the threshold on validation data, then report on untouched test data

Comparing two models evaluated on different test windows Same test data for every candidate
Model Evaluation on Agent Platform

Agent Platform's predictive model evaluation takes a trained model , batch inference output , and ground truth , and computes metrics such as precision, recall, AuPRC, AuROC, log loss, confusion matrices, MAE, RMSE, and MAPE. You can:

Create evaluations from Model Registry in the console and compare them across models or versions.

Add the model evaluation component to an Agent Platform Pipeline , usually after a batch inference component, so every retraining run is evaluated the same way (Chapter 15).

Run evaluations periodically on fresh labeled production data. This is continuous evaluation , which signals when to retrain.

AutoML also produces its own evaluation metrics during training. That is separate from the model evaluation service described here.