Study guide

13.1 Testing for Overfitting and Underfitting

ISTQB Certified Tester AI Testing CT-AI

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts (CT-AI v2.0) 1.2 Scoring, K-levels & Chapter Weights

2 Introduction to AI: Systems and Capabilities

2.1 AI-Based vs Conventional Systems 2.2 Narrow AI, General AI and Super AI 2.3 Types of AI Technologies 2.4 Generative AI

3 AI Hardware, Hosting, Frameworks and Governance

3.1 Hardware for Machine Learning Systems 3.2 Development and Hosting of AI Models 3.3 Machine Learning Development Frameworks 3.4 Regulations and Standards for AI

4 Quality Characteristics and Acceptance Criteria

4.1 ISO/IEC 25059 AI Quality Characteristics 4.2 AI and Safety 4.3 Acceptance Criteria for AI-Based Systems

5 Machine Learning Forms, Workflow and Data Preparation

5.1 Forms of Machine Learning 5.2 The Machine Learning Workflow 5.3 Pretrained Models, Fine-Tuning and RAG 5.4 Data Preparation Activities

6 Datasets, Classification Metrics and Neural Networks

6.1 Training, Validation and Test Datasets 6.2 Calculating Classification Metrics 6.3 Structure of Deep Neural Networks 6.4 Neural Network Coverage Measures

7 Testing Challenges for AI-Based Systems

7.1 Locked vs Adaptive AI-Based Systems 7.2 Why Statistical Testing Is Needed 7.3 Test Oracles for AI-Based Systems

8 GenAI Testing, Test Levels and Risk-Based Testing

8.1 Testing Generative AI 8.2 Red Teaming for GenAI 8.3 Test Levels for Machine Learning Systems 8.4 Risk-Based Testing of ML Systems

9 Input Data Risks, Bias and Pipelines

9.1 Input Data Risks and Mitigations 9.2 Testing for Bias 9.3 Data Pipeline Testing

10 Representativeness, Constraints and Labels

10.1 Testing for Data Representativeness 10.2 Dataset Constraint Testing 10.3 Label Correctness Testing

11 Model Risks, Documentation and Performance Testing

11.1 ML Model Risks and Mitigations 11.2 ML Model Documentation and Review 11.3 Functional Performance Testing of Probabilistic MLS

12 Adversarial, Metamorphic and Drift Testing

12.1 Adversarial Testing 12.2 Metamorphic Testing 12.3 Drift Testing

13 Overfitting, A/B Testing and Back-to-Back Testing

13.1 Testing for Overfitting and Underfitting 13.2 A/B Testing for ML Systems 13.3 Back-to-Back Testing

14 ML Development and Deployment Testing

14.1 ML Development Risks and Mitigations 14.2 ML System Deployment Testing

13.1 Testing for Overfitting and Underfitting

Key Takeaways Explain This Topic AI

ML models have three fit outcomes: overfitting, underfitting, and right-fitting; testers check for them during training, evaluation, and tuning.

Overfitting learns sample noise rather than the underlying pattern, so generalization to unseen data is poor.

Detect overfitting on a separate test set that was not used in training and that includes uncommon examples; a significant drop versus validation is the signal.

Underfitting means the model is too simple or training data lacks features of an important input-output relationship, so metrics are poor on both training and validation.

Learning curves that stay high and close, without improvement, indicate underfitting; right-fit shows good train, validation, and test scores with a small generalization gap.

Last updated: September 2026

Why fit is a testing problem, not only a training notebook

Learning objective AI-6.1.8 is a K2 explain item: show how testers detect overfitting and underfitting , and how they recognize the third outcome, a right-fitting model. CT-AI v2.0 (ISTQB, 17 April 2026) puts this work inside model testing , not as an optional chart a data scientist might glance at after the last epoch. The syllabus is explicit about when the checks happen: during training , evaluation , and tuning . Those three activities together are model generation . If you wait until a frozen production model is already serving traffic, you have skipped the windows where a too-flexible network memorized quirks or a too-simple model never learned the job.

Every machine learning system (MLS) you test lands in one of three buckets. Overfitting means the model learned the training sample too well, including accidental noise, so it generalizes poorly to new data. Underfitting means the model never captured the underlying structure, so it is weak on data it was shown and weak on data held back. Right-fitting means the model learned the pattern that matters, with only a small gap between behavior on data used in model generation and behavior on data it has not seen. Your exam job is to name the three outcomes, name the three activities where you test for them, and name the evidence that separates overfit from underfit from right-fit.

  • The three outcomes in tester language
  • Overfitting: noise instead of pattern

Overfitting occurs when a model learns the training data too well . That phrase is not praise. The model captures noise in that sample rather than the underlying pattern that should apply to new cases. A claims-triage classifier that memorizes that last year's fraud ring always used the phrase "urgent wire today" will fire on that phrase and miss a new ring that uses different wording. Training metrics look excellent because the noise lived in the training file. Generalization to unseen data is poor.

That is why testing for overfitting is not "look at training accuracy and stop." Training success is the trap. A model that recites the training set is not a model that will survive next month's operational mix.

Underfitting: too simple, or the features were never there

Underfitting occurs in two syllabus-named ways, and multiple-choice items like to offer only one. First, the model is too simple to capture the structure of the data: a linear separator where the true relationship bends, a tiny decision tree that cannot represent a policy with several interacting conditions, a network with too little capacity for the task. Second, the training data does not contain features that reflect an important relationship between inputs and outputs . If the true driver of a medical-risk score is a lab ratio nobody collected, extra epochs will not invent that relationship. The observable result is the same either way: poor ML functional performance on both the training dataset and the validation dataset .

Right-fitting: a small generalization gap

A right-fitting model is the remaining outcome. It performs well on training data, on the validation data used during evaluation and tuning, and on an independent test set. The generalization gap — the drop from scores seen during model generation into truly unseen cases — stays small. Right-fit is not "perfect on every metric." It is "good against the agreed ML functional performance criteria, without the tell-tale splits that mark overfit or underfit."

Outcome What the model did What testers see Typical next action

Overfit Memorized sample noise, not the pattern Strong on data used in model generation; significantly worse on an independent test set than on validation Reduce capacity, regularize, improve data, retune; do not ship on validation scores alone

Underfit Too simple, or missing features of an important input-output relationship Poor metrics on training and validation ; learning curves stay high and close Add capacity or features, inspect data preparation; more epochs help only if curves were still moving

Right-fit Learned the useful pattern Good train, validation, and test scores with a small generalization gap Continue remaining model tests and deployment checks

When you test: training, evaluation, and tuning

Do not treat fit testing as a single gate after someone announces that training is finished. During training , you watch whether errors are actually falling. During evaluation , you measure ML functional performance on data that was not used to update weights in that step. During tuning — architecture choices, hyperparameters, feature sets — you can overfit the validation set itself by hunting for a configuration that only wins on that slice. That is why CT-AI still demands a separate test dataset that was not used during training (and, in a well-run workflow, was not used to steer tuning either). The three-way data split from earlier in the syllabus is the instrument. This learning objective is about the defects that split is meant to reveal.

If a team retunes after peeking at the test set, the test set has become another validation set. You can still run numbers. You can no longer claim you measured generalization on data that never influenced model generation.

Loading diagram... Three fit outcomes and the tests that distinguish them

How testers detect overfitting

To test for overfitting, evaluate the model's ML functional performance on a separate test dataset that was not used during training . That sentence is the detection method. Two extra requirements sit on the test set.

First, independence . If the rows you call "test" leaked into training or into the tuning loop, you are not measuring generalization. You are measuring a slightly shuffled training score.

Second, uncommon examples . The test dataset should include less common examples that were unlikely to have been used during training . Frequent cases can look fine even when the model only memorized common patterns. Rare but valid cases — an unusual product-return reason, an uncommon lab combination, a nighttime photo of a traffic sign, a loan applicant with a thin but legitimate credit file — are where memorization fails and a right-fitting model should still behave reasonably.

The decision rule is comparative: the model might be overfitting if it performs significantly worse on the test dataset than on the validation dataset . Notice the pairing. The contrast is test versus validation , not "training looked nice." Validation already participated in evaluation and tuning. A model can look acceptable on validation and still collapse on a true hold-out, especially when uncommon cases finally appear. "Significantly worse" is a tester judgment against the project's agreed metrics and thresholds, not a universal percentage the exam expects you to invent.

A worked picture: a spam filter reports 0.97 F1 on validation after a week of hyperparameter search. On a held-out test set that includes new phishing templates, odd character encodings, and low-volume sender domains, F1 drops to 0.71. That gap, on data the training loop never saw, is the overfitting signal. If instead F1 is 0.96 on validation and 0.95 on a similarly constructed test set, you do not have that signal. Other risks may remain. This is not the overfit pattern.

How testers detect underfitting

Underfitting is detected by evaluating ML functional performance metrics such as accuracy, precision, recall, or F1 score . If those metrics are consistently low on both the training set and the validation set , the model is underfitting. Both sides matter. Low validation with high training is a different story (often overfit during the training run). Low on both means the model never got a useful grip on the task.

Ask two follow-up questions when both sides are poor. Is the model too simple for the structure of the data? Or did data preparation omit features that carry an important input-output relationship? Testers do not have to retrain the model themselves, but they do have to report which of those two stories the evidence supports. A fraud model that never sees "claim amount divided by policy limit" cannot learn that ratio's effect. Extra training time will not create a missing column. A model with too few parameters cannot represent a policy that depends on several interacting conditions, even when every needed column is present.

Learning curves as a visual test

Visual inspection of learning curves is a second underfitting detector. Plot training error and validation error as training progresses. If training and validation errors remain high and relatively close together, without significant improvement , the picture is underfitting. The model is not pulling away from a bad start. A right-fitting run usually shows errors falling, then leveling, with a modest gap between the two curves. An overfitting run often shows training error still falling while validation error flattens or rises — a growing gap. For this learning objective, memorize the underfit curve in words: high, close, not improving .

Do not wait until the last epoch to glance at a single pair of numbers. The "without significant improvement" clause is about trajectory . A short, noisy start can look high and close; continued training that still does not move is the finding you log.

Putting a K2 answer on the page

A complete exam answer names five things. One: the three outcomes — overfit, underfit, right-fit. Two: testing during training, evaluation, and tuning, not only after freeze. Three: overfitting as noise-not-pattern with poor generalization, detected on an independent test set that includes uncommon examples, with a significant drop versus validation. Four: underfitting as too-simple or missing-features, with poor metrics on both training and validation, plus the high-and-close learning-curve signature. Five: right-fit as good train, validation, and test scores with a small generalization gap.

Exam traps for AI-6.1.8

Testing for fit only after deployment. The syllabus requires the check during training, evaluation, and tuning.

Declaring overfit because training accuracy is high. You need the independent test comparison, including uncommon examples.

Comparing only training versus test and ignoring that validation is the syllabus's contrast partner for the "significantly worse" rule.

Calling a model right-fit merely because training and validation errors are close. If both are high and stuck, that closeness is underfitting.

Treating underfitting as "the model is always too simple." Missing features of an important relationship is an equally valid cause.

Reusing the test set for tuning, then claiming you measured generalization.

Confusing this LO with drift testing. Fit is about how the model learned from the datasets you have now. Drift is about operational data changing later.