Study guide
7.1 ML Evaluation Metrics & Validation Strategies
AWS Certified Machine Learning Engineer — Associate MLA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Introduction & Exam Strategy
1.1 AWS MLA-C01 Blueprint, Format & Strategy 1.2 The Machine Learning Engineering Lifecycle on AWS
2 Domain 1: Data Ingestion & Storage Architecture for ML
2.1 Ingesting Batch and Streaming Data for ML 2.2 High-Performance Storage for Training and Inference 2.3 Data Cataloging, Governance, and Querying
3 Domain 1: Feature Engineering & Data Transformation
3.1 Distributed Feature Engineering with AWS Glue & DataBrew 3.2 Interactive Data Preparation with SageMaker Data Wrangler 3.3 Feature Management with SageMaker Feature Store
4 Domain 1: Data Quality, Integrity & Ground Truth Labeling
4.1 Data Validation, Schema Drift, and Quality Rules 4.2 Handling Missing Values, Outliers & Class Imbalance 4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus
5 Domain 2: Model Selection & Framework Architecture
5.1 SageMaker Built-in Algorithms & Selection Criteria 5.2 Custom Models with Script Mode and BYOC 5.3 Foundation Models, JumpStart, and Amazon Bedrock
6 Domain 2: SageMaker Model Training & Hyperparameter Optimization
6.1 SageMaker Training Jobs & Cost Optimization 6.2 Distributed Training Strategies & Architectures 6.3 Hyperparameter Optimization with Automatic Model Tuning (AMT)
7 Domain 2: Model Evaluation, Tracking & Model Registry
7.1 ML Evaluation Metrics & Validation Strategies 7.2 Experiment Tracking with SageMaker Experiments 7.3 Model Governance with SageMaker Model Registry
8 Domain 3: SageMaker Inference Options & Endpoint Architecture
8.1 Real-Time, Serverless, and Asynchronous Inference 8.2 High-Throughput Offline Inference with Batch Transform 8.3 Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)
9 Domain 3: Deployment Strategies & Infrastructure Scaling
9.1 Production Deployment Patterns & Traffic Shifting 9.2 Endpoint Auto-Scaling & Hardware Acceleration
10 Domain 3: MLOps Pipelines & Workflow Orchestration
10.1 SageMaker Pipelines Core Architecture 10.2 Advanced Pipeline Patterns & Integration 10.3 End-to-End MLOps CI/CD & Automated Retraining
11 Domain 4: Production Model Monitoring with SageMaker Model Monitor
11.1 Data Quality & Model Quality Monitoring 11.2 Model Bias & Model Explainability Drift 11.3 Operational Observability with Amazon CloudWatch
12 Domain 4: Responsible AI, Bias Detection & Governance
12.1 Pre-Training and Post-Training Bias with SageMaker Clarify 12.2 Model Documentation & Auditing with Model Cards & Dashboard 12.3 Safety, Guardrails & Generative AI Governance
13 Domain 4: ML Security, Network Isolation & Cost Optimization
13.1 IAM Roles and Least Privilege for ML 13.2 Network Isolation, VPC Endpoints & Data Encryption 13.3 ML Cost Optimization & Resource Governance
7.1 ML Evaluation Metrics & Validation Strategies
Key Takeaways Explain This Topic AI
Classification metric selection depends strictly on error cost: Precision minimizes False Positives (spam filtering, non-critical alerts), Recall minimizes False Negatives (medical diagnosis, financial fraud), while F-beta allows configurable trade-offs (beta=2 favors recall, beta=0.5 favors precision).
Precision-Recall AUC (PR-AUC / Average Precision) is significantly superior to ROC-AUC on imbalanced datasets because ROC-AUC is distorted by large True Negative counts, presenting misleadingly optimistic performance.
Regression metrics penalize residuals differently: MAE applies a linear penalty robust to outliers, while RMSE applies a quadratic penalty that heavily penalizes large errors, making it essential when extreme forecast deviations are costly.
Generative AI and NLP metrics evaluate complementary properties: Perplexity measures model fluency and uncertainty, BLEU calculates n-gram precision for translation, ROUGE (1/2/L) computes n-gram recall and longest common subsequence for summarization, and BERTScore evaluates contextual semantic similarity.
Validation schemes must prevent data leakage: Stratified K-Fold preserves class ratios for imbalanced tabular data, whereas Time-Series Cross-Validation strictly enforces temporal ordering (expanding or rolling windows) with zero future-lookahead leakage.
Last updated: August 2026
ML Evaluation Metrics & Validation Strategies
Evaluating machine learning models is not simply about computing an overall accuracy score. In production systems and on the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, selecting the correct evaluation metric and validation scheme directly determines whether a model achieves its real-world business objective or fails catastrophically due to class imbalance, temporal data leakage, or unpenalized high-cost errors.
An ML engineer must understand how to mathematically derive and interpret classification metrics from the confusion matrix, when to choose PR-AUC over ROC-AUC , how multi-class averaging techniques function, when to penalize regression outliers with RMSE versus MAE , how to assess retrieval and ranking systems with MRR and NDCG , how to evaluate generative AI/LLM models using ROUGE , BLEU , and BERTScore , and how to design validation splits that reflect production inference conditions.
1. Classification Metrics & The Confusion Matrix
All supervised binary classification evaluations originate from the Confusion Matrix , which tabulates the model's predicted labels against the actual ground-truth labels across four fundamental quadrants. +-----------------------------------------------------------------------------------------+
- | THE CONFUSION MATRIX |
- | |
- | ACTUAL CLASS |
- | Positive (1) Negative (0) |
- | +------------------------+------------------------+ |
- | Positive (1) | True Positive (TP) | False Positive (FP) | |
- | PREDICTED | (Hit: Fraud detected) | (Type I Error / Alarm)| |
- | CLASS +------------------------+------------------------+ |
- | Negative (0) | False Negative (FN) | True Negative (TN) | |
- | | (Type II: Fraud missed| (Correct Rejection) | |
- | +------------------------+------------------------+ |
- +-----------------------------------------------------------------------------------------+
- Mathematical Formulation of Core Metrics
Metric Mathematical Formula Focus & Optimization Objective Common Business Use Cases
Accuracy T P + T N T P + T N + F P + F N \frac{TP + TN}{TP + TN + FP + FN} TP + TN + FP + FN TP + TN Overall proportion of correct predictions across all classes. Balanced datasets where false positives and false negatives carry identical business costs.
Precision (Positive Predictive Value) T P T P + F P \frac{TP}{TP + FP} TP + FP TP Measures exactness: out of all instances predicted as positive, how many were actually positive? Minimizes False Positives (FP). Spam email classification, customer churn outreach with high promotional cost, content moderation flags.
Recall / Sensitivity (True Positive Rate) T P T P + F N \frac{TP}{TP + FN} TP + FN TP Measures completeness: out of all actual positive instances, how many did the model find? Minimizes False Negatives (FN). Cancer/disease detection, financial fraud detection, manufacturing defect detection, cybersecurity intrusion.
Specificity (True Negative Rate) T N T N + F P \frac{TN}{TN + FP} TN + FP TN Measures the proportion of actual negative instances correctly identified as negative. Medical screening where confirming healthiness avoids invasive follow-up testing.
False Positive Rate (FPR) F P T N + F P = 1 − Specificity \frac{FP}{TN + FP} = 1 - \text{Specificity} TN + FP FP = 1 − Specificity Proportion of actual negatives incorrectly classified as positive. Threshold calibration in alarm and monitoring systems.
F1-Score 2 × Precision × Recall Precision + Recall 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} 2 × Precision + Recall Precision × Recall Harmonic mean of Precision and Recall. Balances both metrics; punishes extreme discrepancies between them. General imbalanced classification benchmarks where both FP and FN must be controlled.
The F β F_\beta F β -Score: Weighted Harmonic Mean
When business requirements prioritize either Precision or Recall without completely ignoring the other, use the generalized F β F_\beta F β -score : F β = ( 1 + β 2 ) × Precision × Recall ( β 2 × Precision ) + Recall F_\beta = (1 + \beta^2) \times \frac{\text{Precision} \times \text{Recall}}{(\beta^2 \times \text{Precision}) + \text{Recall}} F β = ( 1 + β 2 ) × ( β 2 × Precision ) + Recall Precision × Recall
F 2 F_2 F 2 Score ( β = 2 \beta = 2 β = 2 ): Weights Recall twice as heavily as Precision . Used when missing a positive case (FN) is far more severe than a false alarm (FP) (e.g., loan default prediction, early sepsis detection).
F 0.5 F_{0.5} F 0.5 Score ( β = 0.5 \beta = 0.5 β = 0.5 ): Weights Precision twice as heavily as Recall . Used when a false alarm (FP) is much more disruptive or costly than a missed case (FN) (e.g., automated transactional blockings, legal document discovery filters). # Computing Classification Metrics with Scikit-Learn
from sklearn.metrics import classification_report, fbeta_score, precision_score, recall_score
- y_true = [0, 1, 0, 0, 1, 0, 1, 1, 0, 1]
- y_pred = [0, 1, 1, 0, 1, 0, 0, 1, 0, 1]
- precision = precision_score(y_true, y_pred) # 4 / (4 + 1) = 0.80
- recall = recall_score(y_true, y_pred) # 4 / (4 + 1) = 0.80
- f2 = fbeta_score(y_true, y_pred, beta=2.0) # Recall-weighted
- f05 = fbeta_score(y_true, y_pred, beta=0.5) # Precision-weighted
- print(classification_report(y_true, y_pred, target_names=['Normal', 'Fraud']))
- 2. Threshold-Independent Metrics & Imbalanced Data: ROC-AUC vs. PR-AUC
Binary classifiers output continuous probabilities (e.g., P ( y = 1 ) ∈ [ 0 , 1 ] P(y=1) \in [0, 1] P ( y = 1 ) ∈ [ 0 , 1 ] ). Assigning a discrete label requires choosing a decision threshold (default 0.5 0.5 0.5 ). Evaluating performance across all possible decision thresholds requires curve-based metrics. +-----------------------------------------------------------------------------------------+
- | ROC-AUC VS. PR-AUC COMPARISON |
- | |
- | RECEIVER OPERATING CHARACTERISTIC (ROC) PRECISION-RECALL (PR) CURVE |
- | |
- | TPR Precision |
- | 1.0 | .---''''' 1.0 |''''---. |
- | | .' | `.. |
- | | .' Random Classifier | `.. Baseline = P/(P+N|
- | |.' (AUC = 0.5) | `----------------- |
- | 0.0 +--------------------- 0.0 +-------------------------------- |
- | 0.0 1.0 0.0 1.0 |
- | FPR Recall |
- | |
- | * ROC X-Axis: FPR = FP / (TN + FP) * PR X-Axis: Recall = TP / (TP + FN) |
- | * ROC Y-Axis: TPR = TP / (TP + FN) * PR Y-Axis: Precision = TP / (TP+FP) |
- | * Distorted by large TN (Class Imbalance) * Unaffected by TN; sensitive to FP |
- +-----------------------------------------------------------------------------------------+
- When to Use ROC-AUC vs. PR-AUC
Receiver Operating Characteristic - Area Under Curve (ROC-AUC):
Measures the probability that the model ranks a randomly chosen positive instance higher than a randomly chosen negative instance.
The Imbalance Flaw: The x-axis is F P R = F P T N + F P FPR = \frac{FP}{TN + FP} FPR = TN + FP FP . If the negative class is massive (e.g., 99.9% non-fraud vs. 0.1% fraud), T N TN TN is enormous. A large surge in False Positives ( F P FP FP ) will barely change the denominator ( T N + F P ) (TN + FP) ( TN + FP ) , keeping F P R FPR FPR close to zero and yielding an artificially inflated, optimistic ROC-AUC score (e.g., 0.98 0.98 0.98 ).
Precision-Recall - Area Under Curve (PR-AUC / Average Precision):
Plots Precision ( y y y -axis) against Recall ( x x x -axis) across all decision thresholds.
Why PR-AUC Wins on Imbalanced Data: The PR curve does not include True Negatives ( T N TN TN ) in either axis. If the model generates False Positives on rare positive data, Precision immediately collapses, directly reflecting the drop in performance.
Important
Exam Rule for Imbalanced Datasets: Whenever an exam scenario describes heavy class imbalance (e.g., ad click prediction with 0.05 % 0.05\% 0.05% click-through rate, credit card fraud with 0.1 % 0.1\% 0.1% positive rate), PR-AUC (Precision-Recall AUC / Average Precision) is the correct metric to select over ROC-AUC.
3. Multi-Class Averaging Techniques
When evaluating multi-class classification problems (e.g., image categorization into 10 classes), metrics like Precision, Recall, and F1 must be aggregated across all classes using one of three standard strategies: +-----------------------------------------------------------------------------------------+
- | MULTI-CLASS AVERAGING STRATEGIES |
- | |
- | 1. MACRO AVERAGING: |
- | - Compute metric independently for each class. |
- | - Calculate unweighted arithmetic mean: Macro_F1 = (F1_A + F1_B + F1_C) / 3 |
- | - Key Trait: Treats all classes equally; highlights poor performance on MINORITY. |
- | |
- | 2. MICRO AVERAGING: |
- | - Globally sum TP, FP, FN across all classes first: Total_TP, Total_FP, Total_FN. |
- | - Compute metric from aggregate sums. |
- | - Key Trait: Dominated by MAJORITY class performance; Micro_F1 = Micro_Accuracy. |
- | |
- | 3. WEIGHTED AVERAGING: |
- | - Compute metric independently for each class. |
- | - Calculate weighted average proportional to class sample size (Support). |
- | - Weighted_F1 = (N_A * F1_A + N_B * F1_B + N_C * F1_C) / Total_N |
- +-----------------------------------------------------------------------------------------+
- Log Loss / Cross-Entropy Loss
For multi-class probabilistic models, Log Loss measures the accuracy of predicted probability distributions by penalizing confident incorrect predictions exponentially: Log Loss = − 1 N ∑ i = 1 N ∑ c = 1 C y i , c log ( p i , c ) \text{Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{C} y_{i,c} \log(p_{i,c}) Log Loss = − N 1 i = 1 ∑ N c = 1 ∑ C y i , c lo g ( p i , c )
Lower Log Loss indicates better calibrated probability outputs.
If a model predicts P ( y = 1 ) = 0.99 P(y=1) = 0.99 P ( y = 1 ) = 0.99 on an instance where actual y = 0 y=0 y = 0 , the penalty approaches infinity.
4. Regression Evaluation Metrics
Regression models predict continuous quantities (e.g., home prices, temperature, server load). Choosing the appropriate regression metric depends on how residuals ( y i − y ^ i y_i - \hat{y}_i y i − y ^ i ) should be penalized. +-----------------------------------------------------------------------------------------+
- | REGRESSION METRICS COMPARISON |
- | |
- | Metric Formula Outlier Sensitivity Scale / Units |
- | ------ ------- ------------------- ------------- |
- | MAE (1/N) * sum(|y - y_hat|) Low (Linear) Same as Target ($) |
- | MSE (1/N) * sum((y - y_hat)^2) Very High (Quadratic) Squared Units ($^2) |
- | RMSE sqrt( MSE ) High (Quadratic penal)Same as Target ($) |
- | MAPE (100%/N) * sum(|(y-y_hat)/y|) Moderate Percentage (%) |
- | R^2 1 - (SS_res / SS_tot) Relative to Baseline Dimensionless (-inf,1|
- +-----------------------------------------------------------------------------------------+
Deep-Dive on Regression Metrics:
Mean Absolute Error (MAE):
MAE = 1 N ∑ i = 1 N ∣ y i − y ^ i ∣ \text{MAE} = \frac{1}{N} \sum_{i=1}^N |y_i - \hat{y}_i| MAE = N 1 ∑ i = 1 N ∣ y i − y ^ i ∣
Linear error penalty: an error of 10 10 10 is penalized exactly twice as much as an error of 5 5 5 .
Best used when: Dataset contains anomalous noisy outliers that should not disproportionately skew model evaluation.
Root Mean Squared Error (RMSE):
RMSE = 1 N ∑ i = 1 N ( y i − y ^ i ) 2 \text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2} RMSE = N 1 ∑ i = 1 N ( y i − y ^ i ) 2
Quadratic error penalty: an error of 10 10 10 contributes 100 100 100 to the sum ( 4 × 4\times 4 × an error of 5 5 5 ).
Best used when: Large errors are unacceptable or catastrophic in production (e.g., flight arrival delays, supply chain stockout forecasts).
Mean Absolute Percentage Error (MAPE):
MAPE = 100 % N ∑ i = 1 N ∣ y i − y ^ i y i ∣ \text{MAPE} = \frac{100\%}{N} \sum_{i=1}^N \left|\frac{y_i - \hat{y}_i}{y_i}\right| MAPE = N 100% ∑ i = 1 N y i y i − y ^ i
Expresses error as a percentage relative to the true magnitude.
Limitation: Undefined when actual y i = 0 y_i = 0 y i = 0 , and heavily penalizes over-forecasts when y i y_i y i is close to zero.
Coefficient of Determination ( R 2 R^2 R 2 Score):
R 2 = 1 − ∑ ( y i − y ^ i ) 2 ∑ ( y i − y ˉ ) 2 = 1 − S S res S S tot R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2} = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}} R 2 = 1 − ∑ ( y i − y ˉ ) 2 ∑ ( y i − y ^ i ) 2 = 1 − S S tot S S res
Measures the proportion of variance in the target variable explained by the features.
R 2 = 1.0 R^2 = 1.0 R 2 = 1.0 : Perfect fit. R 2 = 0.0 R^2 = 0.0 R 2 = 0.0 : Predicts no better than the mean. R 2 < 0.0 R^2 < 0.0 R 2 < 0.0 : Worse than predicting the mean.
5. Ranking & Recommendation Metrics
Search engines, e-commerce product recommenders, and retrieval-augmented generation (RAG) vector retrievals require ranking metrics to measure the quality of ordered lists. +-----------------------------------------------------------------------------------------+
- | RANKING & RETRIEVAL METRICS |
- | |
- | 1. PRECISION@K: |
- | - Fraction of top-K retrieved items that are relevant. |
- | - Precision@5 = (Relevant items in top 5) / 5 |
- | |
- | 2. MEAN RECIPROCAL RANK (MRR): |
- | - Evaluates the position of the FIRST relevant item found. |
- | - Reciprocal Rank = 1 / Rank_of_first_relevant_result |
- | - If first relevant item is at rank 3, RR = 1/3 = 0.33. |
- | - MRR is the average Reciprocal Rank across all queries. |
- | |
- | 3. NORMALIZED DISCOUNTED CUMULATIVE GAIN (NDCG@K): |
- | - Evaluates graded relevance (e.g., 0=Irrelevant, 1=Relevant, 2=Highly Relevant). |
- | - Discounted Cumulative Gain (DCG@K) penalizes relevant items ranked lower down: |
- | DCG@K = sum_{i=1}^K ( (2^{rel_i} - 1) / log_2(i + 1) ) |
- | - NDCG@K = DCG@K / IDCG@K (where IDCG is Ideal DCG of perfectly ordered list). |
- | - Score ranges from 0.0 to 1.0 (1.0 = optimal ranking). |
- +-----------------------------------------------------------------------------------------+
- 6. Generative AI & Large Language Model (LLM) Metrics
Evaluating natural language generation (NLG), summarization, translation, and foundation models requires specialized automated metrics. +-----------------------------------------------------------------------------------------+
- | GENAI & LLM EVALUATION METRICS |
- | |
- | Metric Type Core Mechanism Primary Use Case |
- | ------ ---- -------------- ---------------- |
- | Perplexity Intrinsic exp(Cross-Entropy Loss) Language modeling fluency |
- | BLEU Precision Modified n-gram precision Machine translation, QA exact |
- | ROUGE-1/2/L Recall / LCS n-gram recall & Longest Text summarization |
- | Common Subsequence (LCS) |
- | BERTScore Semantic Token contextual embedding Paraphrase & open-ended GenAI |
- | cosine similarity (BERT) |
- +-----------------------------------------------------------------------------------------+
In-Depth Generative AI Metric Breakdown:
Perplexity (PPL):
PPL = exp ( Cross-Entropy Loss ) = exp ( − 1 T ∑ t = 1 T log P ( w t ∣ w < t ) ) \text{PPL} = \exp(\text{Cross-Entropy Loss}) = \exp\left(-\frac{1}{T} \sum_{t=1}^T \log P(w_t \mid w_{<t})\right) PPL = exp ( Cross-Entropy Loss ) = exp ( − T 1 ∑ t = 1 T lo g P ( w t ∣ w < t ) )
Measures how "surprised" or uncertain an autoregressive model is when predicting the next token in a test corpus. Lower perplexity indicates higher fluency and predictive confidence.
BLEU (Bilingual Evaluation Understudy):
Calculates modified n n n -gram precision between candidate generated text and reference human text, combined with a Brevity Penalty (BP) to prevent models from outputting artificially short responses.
Best used for: Machine translation where exact token and phrase matches are desired.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation):
ROUGE-1 / ROUGE-2: Measures the recall overlap of unigrams and bigrams between the generated summary and reference summary.
ROUGE-L: Computes the Longest Common Subsequence (LCS) at the sentence level. It measures structural and word-order similarity without requiring consecutive n n n -gram matches.
Best used for: Text summarization where capturing all source facts is paramount.
BERTScore:
Computes pairwise cosine similarity between contextual token embeddings generated by pre-trained transformer models (e.g., RoBERTa, BERT).
Advantage: Overcomes lexical rigidity in BLEU and ROUGE by recognizing synonyms and paraphrases (e.g., recognizing that "automobile" and "car" represent the same concept).
7. Validation Schemes: Cross-Validation & Temporal Leakage
To ensure evaluation metrics accurately predict generalization performance on unseen real-world data, validation splitting strategies must match data generation dynamics. +-----------------------------------------------------------------------------------------+
- | VALIDATION SPLIT STRATEGIES |
- | |
- | 1. K-FOLD CROSS-VALIDATION (Standard Tabular / Independent Data): |
- | Fold 1: [ Test ] [ Train ] [ Train ] [ Train ] [ Train ] |
- | Fold 2: [ Train ] [ Test ] [ Train ] [ Train ] [ Train ] |
- | Fold 3: [ Train ] [ Train ] [ Test ] [ Train ] [ Train ] |
- | * Averages metric across K runs; reduces evaluation variance. |
- | |
- | 2. STRATIFIED K-FOLD (Imbalanced Classification): |
- | * Guarantees each fold maintains identical class proportions (e.g., 2% positive) |
- | |
- | 3. TIME-SERIES CROSS-VALIDATION (Temporal / Sequential Data): |
- | Split 1: [ Train: Jan-Mar ] ---> [ Test: Apr ] |
- | Split 2: [ Train: Jan-Apr ] -------> [ Test: May ] (Expanding Window) |
- | Split 3: [ Train: Jan-May ] -----------> [ Test: Jun ] |
- | * CRITICAL: Never shuffle time-series data! Prevents future lookahead leakage. |
- +-----------------------------------------------------------------------------------------+
- Offline Validation vs. Online Validation
Offline Validation: Performed on historical static datasets using cross-validation or held-out test splits. Fast, deterministic, and safe, but cannot observe user feedback loops, behavioral drift, or system latency constraints.
Online Validation (A/B Testing & Shadow Deployment): Evaluates live production traffic. Live users are routed between Model A and Model B, measuring real business conversions (e.g., click-through rate, checkout rate) and verifying operational latency under load.
8. Debugging Training Convergence (SageMaker Debugger)
Task Statement 2.3 also expects you to know how model convergence problems are debugged on AWS. Amazon SageMaker Debugger captures training tensors (weights, gradients, activations, losses) during SageMaker training jobs and evaluates built-in rules that detect non-convergence and training pathologies in near real time — for example VanishingGradient , ExplodingTensor , LossNotDecreasing , Overfit , SaturatedActivationFunction , and PoorWeightInitialization — emitting Amazon CloudWatch / EventBridge alerts or stopping the job when a rule fires. Typical exam signals: training loss flatlines or explodes, gradients vanish across deep layers, or validation loss diverges from training loss (overfitting) → attach Debugger rules to the Estimator to capture and diagnose the tensors.
Note
Availability note (2026): SageMaker Debugger is no longer open to new customers (existing customers retain access), and its framework-profiling features were deprecated from TensorFlow 2.11 / PyTorch 2.0 onward; AWS now steers interactive training analysis toward the TensorBoard application on SageMaker . The convergence concepts Debugger automates — vanishing/exploding gradients, loss plateaus, overfit detection — remain exam-relevant for Task 2.3.