Study guide
6.2 Calculating Classification Metrics
ISTQB Certified Tester AI Testing CT-AI
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts (CT-AI v2.0) 1.2 Scoring, K-levels & Chapter Weights
2 Introduction to AI: Systems and Capabilities
2.1 AI-Based vs Conventional Systems 2.2 Narrow AI, General AI and Super AI 2.3 Types of AI Technologies 2.4 Generative AI
3 AI Hardware, Hosting, Frameworks and Governance
3.1 Hardware for Machine Learning Systems 3.2 Development and Hosting of AI Models 3.3 Machine Learning Development Frameworks 3.4 Regulations and Standards for AI
4 Quality Characteristics and Acceptance Criteria
4.1 ISO/IEC 25059 AI Quality Characteristics 4.2 AI and Safety 4.3 Acceptance Criteria for AI-Based Systems
5 Machine Learning Forms, Workflow and Data Preparation
5.1 Forms of Machine Learning 5.2 The Machine Learning Workflow 5.3 Pretrained Models, Fine-Tuning and RAG 5.4 Data Preparation Activities
6 Datasets, Classification Metrics and Neural Networks
6.1 Training, Validation and Test Datasets 6.2 Calculating Classification Metrics 6.3 Structure of Deep Neural Networks 6.4 Neural Network Coverage Measures
7 Testing Challenges for AI-Based Systems
7.1 Locked vs Adaptive AI-Based Systems 7.2 Why Statistical Testing Is Needed 7.3 Test Oracles for AI-Based Systems
8 GenAI Testing, Test Levels and Risk-Based Testing
8.1 Testing Generative AI 8.2 Red Teaming for GenAI 8.3 Test Levels for Machine Learning Systems 8.4 Risk-Based Testing of ML Systems
9 Input Data Risks, Bias and Pipelines
9.1 Input Data Risks and Mitigations 9.2 Testing for Bias 9.3 Data Pipeline Testing
10 Representativeness, Constraints and Labels
10.1 Testing for Data Representativeness 10.2 Dataset Constraint Testing 10.3 Label Correctness Testing
11 Model Risks, Documentation and Performance Testing
11.1 ML Model Risks and Mitigations 11.2 ML Model Documentation and Review 11.3 Functional Performance Testing of Probabilistic MLS
12 Adversarial, Metamorphic and Drift Testing
12.1 Adversarial Testing 12.2 Metamorphic Testing 12.3 Drift Testing
13 Overfitting, A/B Testing and Back-to-Back Testing
13.1 Testing for Overfitting and Underfitting 13.2 A/B Testing for ML Systems 13.3 Back-to-Back Testing
14 ML Development and Deployment Testing
14.1 ML Development Risks and Mitigations 14.2 ML System Deployment Testing
6.2 Calculating Classification Metrics
Key Takeaways Explain This Topic AI
Accuracy = (TP + TN) / (TP + TN + FP + FN) * 100%; precision = TP / (TP + FP) * 100%; recall (sensitivity) = TP / (TP + FN) * 100%; F1-score = 2 * (Precision * Recall) / (Precision + Recall) on a 0–100 scale.
Confusion-matrix axes may be swapped in a figure; the four cells true positive, false positive, false negative, and true negative still define every metric.
Accuracy can look excellent on imbalanced data while the positive class is almost unusable — a screening example can show 91.6% accuracy with 18% precision and an F1-score of 30.
Prefer recall when missing actual positives is the costly error; prefer precision when false positive alarms are the costly error; a high F1-score means both precision and recall are high.
AI-3.3.1 is a K3, two-point-style skill: you must compute the metrics from given TP, TN, FP, and FN, not only name them.
Last updated: September 2026
Classification models almost never get every case right. Labels are noisy, features overlap, and the learner is probabilistic. Learning objective AI-3.3.1 (K3) is not “list four metric names.” It is calculate common ML functional performance metrics from a given confusion matrix. On the exam this is treated as a two-point calculation item: you will be handed counts, you will apply the syllabus formulas, and you will interpret what the numbers mean for risk.
The four cells you must never mix up
For a binary problem, every prediction falls into one of four buckets:
True positive (TP): actual positive, predicted positive. The screening test flags a patient who really has the condition; the filter files a message that really is spam.
False positive (FP): actual negative, predicted positive. A healthy patient is flagged; a legitimate invoice is dumped in junk.
False negative (FN): actual positive, predicted negative. A sick patient is cleared; a phishing mail lands in the inbox.
True negative (TN): actual negative, predicted negative. A healthy patient is cleared; a normal mail stays in the inbox.
A confusion matrix is just a 2×2 arrangement of those four counts. Figures often put actual classes on one axis and predicted classes on the other. The axes may be swapped from one textbook or dashboard to the next. That presentation change does not invent new categories. You still extract TP, FP, FN, and TN, then plug them into the formulas. If an exam figure looks “upside down” compared with the last tutorial you watched, relabel the four cells before you divide. Do not invent a fifth cell.
Formulas exactly as the CT-AI v2.0 syllabus states them
Write these until they are automatic. Precision and recall are used as percentages (0–100) when you compute F1-score the way the syllabus writes it; using 0–1 proportions and then scaling yields the same F1-score.
Accuracy = (TP + TN) / (TP + TN + FP + FN) * 100%
Accuracy is the percentage of all classifications that were correct — both positives and negatives.
Precision = TP / (TP + FP) * 100%
Precision is the proportion of predicted positives that were actually positive. It answers: when the model raises an alarm, how sure can I be?
Recall (also called sensitivity ) = TP / (TP + FN) * 100%
Recall is the proportion of actual positives that were predicted positive. It answers: how confident can I be that positives are not being missed?
F1-score = 2 * (Precision * Recall) / (Precision + Recall)
F1-score is the harmonic mean of precision and recall, ranging from 0 to 100 . A value near 100 means both precision and recall are high, so false positives and false negatives have little impact. A low F1-score means the model struggles to identify positives — missing true cases, generating many false alarms, or both.
Accuracy can be high while F1-score is poor. That is the trap this K3 item is built to catch.
Worked example 1: medical screening on an imbalanced set
A clinic screens 1,000 people for a rare condition. Only 20 people truly have it (2% prevalence). A “model” that predicts negative for everyone scores:
- TP = 0, FP = 0, FN = 20, TN = 980
- Accuracy = (0 + 980) / 1000 * 100% = 98%
- Recall = 0 / (0 + 20) * 100% = 0%
Precision is undefined if you never predict positive (TP + FP = 0), which is already a testing smell: the positive class was never exercised.
Ninety-eight percent accuracy looks like a celebration slide. The system missed every sick patient . Testers who stop at accuracy would ship a useless screener.
Now a real detector flags 100 people. The confusion matrix is:
- Predicted positive Predicted negative
- Actual positive TP = 18 FN = 2
- Actual negative FP = 82 TN = 898
Check the total: 18 + 2 + 82 + 898 = 1,000.
- Accuracy = (18 + 898) / 1000 * 100% = 916 / 1000 * 100% = 91.6%
- Precision = 18 / (18 + 82) * 100% = 18 / 100 * 100% = 18%
- Recall = 18 / (18 + 2) * 100% = 18 / 20 * 100% = 90%
- F1-score = 2 * (18 * 90) / (18 + 90) = 3240 / 108 = 30
Read that out loud. Accuracy is still in the nineties. Recall is high: 90% of the truly sick people were flagged. Precision is 18%: five out of six alarms are false . F1-score is only 30 because the harmonic mean punishes the weak precision. In screening, missing a true case (FN) may be the unacceptable harm, so a team might still prefer recall and accept a pile of follow-up tests. They must not quote 91.6% accuracy as if the positive predictions were trustworthy.
Worked example 2: spam filter where false alarms bury real mail
A mailbox has 200 labeled messages. 60 are spam; 140 are legitimate. The filter’s matrix is:
- TP = 36 (spam caught)
- FP = 4 (legitimate mail marked spam)
- FN = 24 (spam that reached the inbox)
- TN = 136 (legitimate mail left alone)
Total: 36 + 4 + 24 + 136 = 200.
- Accuracy = (36 + 136) / 200 * 100% = 172 / 200 * 100% = 86%
- Precision = 36 / (36 + 4) * 100% = 36 / 40 * 100% = 90%
- Recall = 36 / (36 + 24) * 100% = 36 / 60 * 100% = 60%
- F1-score = 2 * (90 * 60) / (90 + 60) = 10800 / 150 = 72
Here false positives are expensive : an auto-deleted invoice or a missed customer email is worse, for many users, than deleting one extra phishing note by hand. Precision is the metric that matches that harm model. Recall of 60% means two in five spam messages still arrive; that may be acceptable if the alternative is burying payroll notices. F1-score 72 sits between the two: it is a compromise statistic, not a substitute for naming which error type the product can tolerate.
If you reversed the costs — a filter for a high-risk fraud queue where missing spam-like fraud is catastrophic — you would lean the other way, toward recall , the same way the clinic did. The formulas do not choose the product risk; they quantify the two error directions so you can.
Side-by-side so the imbalance lesson sticks
- Setting Accuracy Precision Recall F1-score Costly error
- Always-negative screener 98% n/a (no predicted positives) 0% collapses Missed disease
- Real screener (18 / 82 / 2 / 898) 91.6% 18% 90% 30 False alarms vs missed disease
- Spam filter (36 / 4 / 24 / 136) 86% 90% 60% 72 Buried legitimate mail
Accuracy ranked the always-negative screener first . That ranking is why CT-AI makes you calculate the rest.
How to sit a two-point calculation item
Label TP, FP, FN, TN from the figure even if predicted/actual are swapped.
Write the four syllabus formulas before plugging in numbers.
Compute accuracy, precision, and recall as percentages.
Compute F1-score from those precision and recall percentage values unless the item clearly uses 0–1 proportions — the syllabus form is the percentage form, and both agree if you are consistent.
Sanity-check: precision uses FP in the denominator; recall uses FN . Swapping FP and FN is the most common arithmetic way to lose both points.
Interpret: if classes are imbalanced, do not let accuracy be the only sentence in your answer.
You will see library helpers in hands-on work. The exam still expects you to calculate by hand from the four counts. Practice until 2 × (P × R) / (P + R) is muscle memory.