Study guide

3.2 Perturbation & metamorphic testing

ISTQB Certified Tester — Testing with Generative AI CT-GenAI

Track your progress as you read

Finished sections get a check mark.

1 GenAI Foundations for Testers

1.1 Transformers, tokenization, embeddings & context window 1.2 Foundation models, temperature/sampling & prompting basics 1.3 RAG & agent architectures testers must understand

2 Quality Attributes for GenAI

2.1 Faithfulness, factuality, relevance, coherence & fluency 2.2 Safety, toxicity & bias 2.3 Robustness, privacy, latency & cost

3 Test Design for Non-Determinism

3.1 Controlling non-determinism (seeds, temperature, golden datasets) 3.2 Perturbation & metamorphic testing 3.3 Adversarial prompts & red-teaming

4 GenAI Risks & Mitigation

4.1 Hallucinations: detection & mitigation 4.2 Prompt injection, jailbreaks & training-data extraction 4.3 Governance frameworks (NIST AI RMF, EU AI Act, model cards)

5 Test Infrastructure & Tooling

5.1 Evaluation frameworks (Promptfoo, LangSmith, OpenAI Evals, RAGAS) 5.2 Automated metrics (BLEU/ROUGE/METEOR, embedding similarity, LLM-as-judge) 5.3 Guardrails & safety tooling (Llama Guard, NeMo Guardrails)

6 Organizational Adoption

6.1 Responsible AI governance & regression testing for GenAI 6.2 Continuous evaluation, synthetic test data & cost monitoring

3.2 Perturbation & metamorphic testing

Key Takeaways Explain This Topic AI

Metamorphic and perturbation testing sidestep the oracle problem by checking relationships between related inputs' outputs, not one output against a known answer.

A metamorphic relation links a source input/output to a transformed follow-up; the outputs must satisfy an expected relation (invariance or a directional change).

Common invariance relations: paraphrasing the prompt, adding irrelevant context, or reordering list items should not change a factual answer or decision.

Perturbation testing applies small meaning-preserving edits (typos, synonyms, casing) and asserts output stability, measured as a robustness rate.

Both techniques need no hand-labelled oracle and pair with the golden dataset; apply one transformation at a time and log every failing pair as a regression case.

Last updated: July 2026

The oracle problem and how to sidestep it

Section 3.1 assumed you already had a reference answer to compare against. Often you do not: nobody has hand-labelled the "correct" output for every possible input, and for open-ended generation a single ground truth may not even exist. This is the oracle problem . Metamorphic and perturbation testing sidestep it by checking the relationship between outputs across related inputs. You do not need to know the right answer; you only need to know how the answer should change , or stay the same, when you change the input in a defined way.

Metamorphic testing

A metamorphic relation (MR) is a rule that links a source input/output pair to a follow-up input/output pair. You transform the source input in a controlled way, run the model again, and assert that the two outputs satisfy the expected relation. If they do not, you have found a defect — without ever writing a golden answer.

Common relations for GenAI fall into two families:

Invariance relations — the transformation should not change the meaning of the output.

Directional / equivalence relations — the transformation should change the output in a predictable direction, or preserve a specific property.

Worked examples make this concrete:

Paraphrase invariance. Source: "What is the capital of France?" Follow-up: "Which city is France's capital?" The factual answer must stay semantically equal ("Paris"). If the paraphrase flips the answer, the model is unstable to surface wording.

Irrelevant-context invariance. Insert a neutral, unrelated sentence ("It rained yesterday.") before a factual question. A robust system's factual answer should not change; a changed answer reveals susceptibility to distraction.

Negation / antonym relation. Change "list the advantages of X" to "list the disadvantages of X"; the follow-up output should be substantively different , not a copy. Sameness signals the model ignored the semantic flip.

Translation round-trip. Translate English to German to English; the meaning should be preserved, so large drift flags a translation defect.

Ordering invariance. For "summarise these three reviews," reordering the reviews should not materially change the summary's sentiment or key points.