9.1 Input Data Risks and Mitigations

ISTQB Certified Tester AI Testing CT-AI

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts (CT-AI v2.0) 1.2 Scoring, K-levels & Chapter Weights

2 Introduction to AI: Systems and Capabilities

2.1 AI-Based vs Conventional Systems 2.2 Narrow AI, General AI and Super AI 2.3 Types of AI Technologies 2.4 Generative AI

3 AI Hardware, Hosting, Frameworks and Governance

3.1 Hardware for Machine Learning Systems 3.2 Development and Hosting of AI Models 3.3 Machine Learning Development Frameworks 3.4 Regulations and Standards for AI

4 Quality Characteristics and Acceptance Criteria

4.1 ISO/IEC 25059 AI Quality Characteristics 4.2 AI and Safety 4.3 Acceptance Criteria for AI-Based Systems

5 Machine Learning Forms, Workflow and Data Preparation

5.1 Forms of Machine Learning 5.2 The Machine Learning Workflow 5.3 Pretrained Models, Fine-Tuning and RAG 5.4 Data Preparation Activities

6 Datasets, Classification Metrics and Neural Networks

6.1 Training, Validation and Test Datasets 6.2 Calculating Classification Metrics 6.3 Structure of Deep Neural Networks 6.4 Neural Network Coverage Measures

7 Testing Challenges for AI-Based Systems

7.1 Locked vs Adaptive AI-Based Systems 7.2 Why Statistical Testing Is Needed 7.3 Test Oracles for AI-Based Systems

8 GenAI Testing, Test Levels and Risk-Based Testing

8.1 Testing Generative AI 8.2 Red Teaming for GenAI 8.3 Test Levels for Machine Learning Systems 8.4 Risk-Based Testing of ML Systems

9 Input Data Risks, Bias and Pipelines

9.1 Input Data Risks and Mitigations 9.2 Testing for Bias 9.3 Data Pipeline Testing

10 Representativeness, Constraints and Labels

10.1 Testing for Data Representativeness 10.2 Dataset Constraint Testing 10.3 Label Correctness Testing

11 Model Risks, Documentation and Performance Testing

11.1 ML Model Risks and Mitigations 11.2 ML Model Documentation and Review 11.3 Functional Performance Testing of Probabilistic MLS

12 Adversarial, Metamorphic and Drift Testing

12.1 Adversarial Testing 12.2 Metamorphic Testing 12.3 Drift Testing

13 Overfitting, A/B Testing and Back-to-Back Testing

13.1 Testing for Overfitting and Underfitting 13.2 A/B Testing for ML Systems 13.3 Back-to-Back Testing

14 ML Development and Deployment Testing

14.1 ML Development Risks and Mitigations 14.2 ML System Deployment Testing

9.1 Input Data Risks and Mitigations

Key Takeaways Explain This Topic AI

Input data testing confirms that training, testing, and prediction data are of sufficient quality for the intended machine learning use.

The toolbox includes reviews, statistical bias tests, exploratory data analysis, and static plus dynamic tests of the data pipeline.

Training defects and algorithmic unfairness are mitigated by testing for bias; untrustworthy sources and poor data management are mitigated by data provenance testing.

Poisoned training data is mitigated by stacking A/B testing, provenance testing, exploratory data analysis, and red-team attacks.

Constraint, feature, representativeness, label-correctness, and pipeline tests each map to a distinct family of input-data defects.

Last updated: September 2026

Machine learning systems fail in a distinctive way: they can look numerically impressive and still be unsafe, unfair, or simply wrong for the people who will live with the predictions. The usual software reflex — the code compiled, so the input must be fine — does not survive contact with training sets, hold-out sets, and live prediction feeds. ISTQB Certified Tester AI Testing (CT-AI) version 2.0 learning objective AI-5.1.1 (K2) asks you to give examples of test approaches that reduce input-data risk. This section is that map. You will learn the objective of input data testing, the mix of techniques it includes, and a risk-to-mitigation table you can retrieve when a scenario stem names a defect and four tempting techniques.

The objective: quality across training, testing, and prediction data

The objective of input data testing is to confirm that the data a machine learning system uses for training , testing , and prediction is of sufficient quality for the intended use. That sentence has three nouns on purpose. A meticulously reviewed training extract does not rescue a test set sampled from a single hospital, and neither set rescues a production feed whose units silently changed last Tuesday. Testers therefore ask the same quality question at three moments in the lifecycle:

• while examples are assembled and used to fit the model
• while separate examples are used to evaluate the model in the laboratory
• while live records are scored to support a real decision

Sufficient quality is not a slogan. It means the data are fit for the decisions the system will influence: complete enough for the use cases, consistent enough to learn from, traceable enough to trust, and free enough of hostile or accidental contamination that the model is not being taught a lie.

Input data testing is a toolbox , not a single script. It includes:

• Reviews of collection plans, labelling guides, pipeline design, and dataset documentation
• Statistical techniques , including tests that look for bias and other non-random structure

Exploratory data analysis (EDA) of training data and related sets — distributions, clusters, holes, and odd groupings

• Static testing of pipeline code, schemas, and configuration
• Dynamic testing of the pipeline as data actually moves from source to model

If you remember one exam sentence, remember this: quality is a property of the whole data story, not of a single file name.

A risk-to-mitigation table you can recall under time pressure

CT-AI does not ask you to invent a unique test for every ugly spreadsheet. It asks you to pair a recognizable risk with a named mitigation. Treat the table as a diagnostic. The left column is the failure a question stem might describe. The right column is the test family you should reach for first.

Potential input-data risk Test approach that mitigates it

Defects in the training examples, or unfairness introduced by the algorithm, model, or development framework Testing for bias

• Data taken from untrustworthy sources, or data that is poorly managed after it arrives Data provenance testing
• Poisoned training data A/B testing, data provenance testing, EDA, and red-team attacks
• Internally inconsistent records, values outside allowed ranges, or wrong data types Dataset constraint testing
• Sub-optimal feature selection Feature testing
• Imbalanced classes, skewed distributions, or missing values Data representativeness testing

Training that covers only a subset of use cases, or value ranges that do not span operational reality Data representativeness testing

Poor labelling guidelines, ambiguous examples, or inconsistent labels Label correctness testing

Pipeline design or integration failures, quality defects in pipeline outputs, operational performance degradation, security breaches, or uncontrolled change Data pipeline testing

Read the rows as a diagnostic, not as a ritual. Real incidents often trip more than one row. Poisoned data can also create unfairness. A broken pipeline can manufacture missingness that then looks like a representativeness problem. On the exam, pick the mitigation that addresses the primary defect the stem named.

Training defects and algorithmic unfairness

When training examples under-represent a group, or when an algorithm applies a decision threshold that systematically harms that group, the resulting model is not merely a bit inaccurate. It treats people differently in a non-random way that is often unlawful. The matching mitigation is testing for bias , which section 9.2 develops. Testers look for unfair differences tied to sensitive attributes such as gender, age, or race. Do not stop at a healthy overall accuracy number. Overall accuracy can hide a group that is almost never approved, never detected, or never offered the safer option. If the stem says the training set or the algorithm produced systemic unfairness, do not answer add more features or tighten the schema. Answer testing for bias.

Untrustworthy sources and poor data management

If nobody can say where a column came from, who edited it, which license covered it, or whether a third-party dump was screened, you do not have a dataset — you have a rumor with a filename. Data provenance testing asks whether origin, custody, transformations, and access history are recorded and believable. A provenance review might reject a scraped forum corpus used as ground-truth clinical labels, or a hand-merged workbook whose authors cannot reconstruct which rows came from which year. Poor management after collection — silent overwrites, undocumented recodes, shared passwords on the labelling bucket — is the same risk wearing a different badge. The test is not whether accuracy looks high. The test is whether you can trust how this data got here.

Poisoned training data

Poisoning is hostile or reckless contamination of the examples the model will treat as truth. An attacker might inject mislabeled images, flip a thin slice of loan outcomes, or stuff a sensor stream with plausible-looking outliers that pull a decision boundary. The named mitigations stack rather than compete:

• A/B testing can show that a new training batch quietly changed live behavior compared with a previous batch
• Data provenance testing can show that the batch should never have been trusted
• EDA can surface clusters, spikes, or label patterns that do not belong in the intended population
• Red-team attacks try to insert poison the way an adversary would, then check whether controls notice

On the exam, if the stem says someone tampered with training examples, do not answer retrain with a larger model. Answer this stacked set.

Loading diagram... Input-data risks mapped to named CT-AI mitigations

Internally inconsistent, out-of-range, and mistyped values

A vital-sign field that stores high, 118, and 11800 in the same column is not a modeling inconvenience. It is a broken contract. Dataset constraint testing checks rules that should hold if the data are correct: types, ranges, uniqueness, required fields, and cross-field comparisons such as a discharge timestamp that must not precede admission. Constraints can apply to a single cell — this value must be an integer between 0 and 120 — or across many cells — this column must not be ninety percent duplicates; this total must not exceed a known cap. Nobody reviews a million rows by eye. The checks belong in the pipeline and should fail loudly when a type flips or a range explodes.

Sub-optimal feature selection

Features are the questions the model is allowed to ask. Omit the signal that actually drives the outcome, and the model will invent a proxy. Include an identifier that merely memorizes training IDs, and the model will look brilliant in the lab and collapse in the field. Feature testing examines whether selected features are useful, non-leaking, stable across time, and worth their cost. A classic trap is using a post-decision field — the approval code you are trying to predict — as an input. Another is dropping a rare-but-critical sensor because it was missing in the first sample, then discovering that sensor is exactly what operations uses during incidents. When the stem is about choosing, dropping, or leaking predictors, the named mitigation is feature testing, not constraint testing.

Imbalance, skew, missingness, narrow use cases, and truncated ranges

These symptoms are cousins, which is why two table rows share data representativeness testing . An imbalanced classifier may see ninety-eight percent no-event rows and learn to say no. A skewed set may be the product of naive augmentation that cloned the majority class until the minority vanished from view. Missingness may hide an entire clinic, region, device generation, or language. Training that only saw weekday daytime tickets will not represent overnight incidents. A temperature feature that only spanned a narrow indoor band will not represent a warehouse in winter. Representativeness testing asks whether training, validation, and test sets resemble the operational population — including the ugly tails and the use cases nobody photographed for the original collection plan. If the stem says the model never saw a use case or a value range, stay on this row even if the pipeline itself looks tidy.

Labelling failures

Supervised learning treats labels as the teacher. If the labelling guide is vague, the examples are inherently ambiguous, or annotators disagree without a resolution path, the model learns noise as if it were physics. Label correctness testing is the mitigation: expert review of samples, multiple independent annotations, agreement measures, rule checks, and mining of high-loss or low-confidence points. When a scenario mentions inconsistent tags, an undefined boundary, or a sloppy guide, do not call it feature testing, and do not call it provenance testing unless the stem also attacks origin. The primary defect is the label.

Pipeline design, integration, quality, performance, security, and change

Even a perfect warehouse dump is useless if the pipe that delivers it drops fields, reorders columns, lets malware ride an upload, or ships an unreviewed config into production. Data pipeline testing , developed in section 9.3, covers design reviews, component and integration tests, system tests including fault injection and back-to-back comparison, production A/B testing and monitoring, and configuration-management reviews of code, configs, and datasets. Use this row when the stem is about ingestion, transforms, routing, load, security, or uncontrolled change — not when the only issue is a biased threshold sitting inside an otherwise healthy pipe.

Using the map on CT-AI questions

K2 items typically give you a short story and four named techniques. Translate the story into a table row before you look at the options. A vendor file with no origin log is provenance. Approvals that differ by age after the same credit history are testing for bias. Night-shift cases that never appeared in training are representativeness. A new transform that silently changed units is data pipeline testing. If two rows both seem plausible, pick the one that names the defect the question actually wrote down. The rest of this chapter then deepens two of those families: testing for bias, and data pipeline testing.