Study guide
3.2 Perturbation & metamorphic testing
ISTQB Certified Tester — Testing with Generative AI CT-GenAI
Track your progress as you read
Finished sections get a check mark.
1 GenAI Foundations for Testers
1.1 Transformers, tokenization, embeddings & context window 1.2 Foundation models, temperature/sampling & prompting basics 1.3 RAG & agent architectures testers must understand
2 Quality Attributes for GenAI
2.1 Faithfulness, factuality, relevance, coherence & fluency 2.2 Safety, toxicity & bias 2.3 Robustness, privacy, latency & cost
3 Test Design for Non-Determinism
3.1 Controlling non-determinism (seeds, temperature, golden datasets) 3.2 Perturbation & metamorphic testing 3.3 Adversarial prompts & red-teaming
4 GenAI Risks & Mitigation
4.1 Hallucinations: detection & mitigation 4.2 Prompt injection, jailbreaks & training-data extraction 4.3 Governance frameworks (NIST AI RMF, EU AI Act, model cards)
5 Test Infrastructure & Tooling
5.1 Evaluation frameworks (Promptfoo, LangSmith, OpenAI Evals, RAGAS) 5.2 Automated metrics (BLEU/ROUGE/METEOR, embedding similarity, LLM-as-judge) 5.3 Guardrails & safety tooling (Llama Guard, NeMo Guardrails)
6 Organizational Adoption
6.1 Responsible AI governance & regression testing for GenAI 6.2 Continuous evaluation, synthetic test data & cost monitoring
3.2 Perturbation & metamorphic testing
Key Takeaways Explain This Topic AI
Metamorphic and perturbation testing sidestep the oracle problem by checking relationships between related inputs' outputs, not one output against a known answer.
A metamorphic relation links a source input/output to a transformed follow-up; the outputs must satisfy an expected relation (invariance or a directional change).
Common invariance relations: paraphrasing the prompt, adding irrelevant context, or reordering list items should not change a factual answer or decision.
Perturbation testing applies small meaning-preserving edits (typos, synonyms, casing) and asserts output stability, measured as a robustness rate.
Both techniques need no hand-labelled oracle and pair with the golden dataset; apply one transformation at a time and log every failing pair as a regression case.
Last updated: July 2026
The oracle problem and how to sidestep it
Section 3.1 assumed you already had a reference answer to compare against. Often you do not: nobody has hand-labelled the "correct" output for every possible input, and for open-ended generation a single ground truth may not even exist. This is the oracle problem . Metamorphic and perturbation testing sidestep it by checking the relationship between outputs across related inputs. You do not need to know the right answer; you only need to know how the answer should change , or stay the same, when you change the input in a defined way.
Metamorphic testing
A metamorphic relation (MR) is a rule that links a source input/output pair to a follow-up input/output pair. You transform the source input in a controlled way, run the model again, and assert that the two outputs satisfy the expected relation. If they do not, you have found a defect — without ever writing a golden answer.
Common relations for GenAI fall into two families:
Invariance relations — the transformation should not change the meaning of the output.
Directional / equivalence relations — the transformation should change the output in a predictable direction, or preserve a specific property.
Worked examples make this concrete:
Paraphrase invariance. Source: "What is the capital of France?" Follow-up: "Which city is France's capital?" The factual answer must stay semantically equal ("Paris"). If the paraphrase flips the answer, the model is unstable to surface wording.
Irrelevant-context invariance. Insert a neutral, unrelated sentence ("It rained yesterday.") before a factual question. A robust system's factual answer should not change; a changed answer reveals susceptibility to distraction.
Negation / antonym relation. Change "list the advantages of X" to "list the disadvantages of X"; the follow-up output should be substantively different , not a copy. Sameness signals the model ignored the semantic flip.
Translation round-trip. Translate English to German to English; the meaning should be preserved, so large drift flags a translation defect.
Ordering invariance. For "summarise these three reviews," reordering the reviews should not materially change the summary's sentiment or key points.