Study guide

3.1 Controlling non-determinism (seeds, temperature, golden datasets)

ISTQB Certified Tester — Testing with Generative AI CT-GenAI

Track your progress as you read

Finished sections get a check mark.

1 GenAI Foundations for Testers

1.1 Transformers, tokenization, embeddings & context window 1.2 Foundation models, temperature/sampling & prompting basics 1.3 RAG & agent architectures testers must understand

2 Quality Attributes for GenAI

2.1 Faithfulness, factuality, relevance, coherence & fluency 2.2 Safety, toxicity & bias 2.3 Robustness, privacy, latency & cost

3 Test Design for Non-Determinism

3.1 Controlling non-determinism (seeds, temperature, golden datasets) 3.2 Perturbation & metamorphic testing 3.3 Adversarial prompts & red-teaming

4 GenAI Risks & Mitigation

4.1 Hallucinations: detection & mitigation 4.2 Prompt injection, jailbreaks & training-data extraction 4.3 Governance frameworks (NIST AI RMF, EU AI Act, model cards)

5 Test Infrastructure & Tooling

5.1 Evaluation frameworks (Promptfoo, LangSmith, OpenAI Evals, RAGAS) 5.2 Automated metrics (BLEU/ROUGE/METEOR, embedding similarity, LLM-as-judge) 5.3 Guardrails & safety tooling (Llama Guard, NeMo Guardrails)

6 Organizational Adoption

6.1 Responsible AI governance & regression testing for GenAI 6.2 Continuous evaluation, synthetic test data & cost monitoring

3.1 Controlling non-determinism (seeds, temperature, golden datasets)

Key Takeaways Explain This Topic AI

GenAI is non-deterministic: the same prompt can yield different valid outputs, so exact-match assertions produce flaky failures rather than real defects.

Reduce test variance with temperature 0 (greedy decoding), a fixed seed where supported, and a pinned dated model version instead of a 'latest' alias.

Replace exact-match oracles with tolerance-based checks: semantic-similarity thresholds, fact presence, schema validation, and LLM-as-judge scoring.

A golden dataset is a curated input to reference-output test set scored in aggregate (e.g. at least 95% of cases pass) as the regression baseline.

Re-run and version the golden dataset on every prompt, model, parameter, or retrieval change; a drop in aggregate score signals a regression.

Last updated: July 2026

Why generative AI is non-deterministic

A conventional program is deterministic: the same input produces the same output every time, so a tester can assert assertEquals(expected, actual) with confidence. Large language models break this contract. A GenAI system samples its next token from a probability distribution, so the same prompt submitted twice can return two different, yet both acceptable, answers. "Summarise this paragraph" might yield a five-word summary on one run and a two-sentence summary on the next. Sources of variance include probabilistic sampling, floating-point non-associativity across GPUs, load-balanced routing to different hardware, and silent vendor model updates behind the same API endpoint.

This non-determinism is exactly why classic expected-value assertions fail. An exact string comparison against a single reference answer reports a false failure whenever the model phrases a correct answer differently. The test is flaky not because the code is broken, but because the oracle is too strict. The tester's job therefore shifts from asking "is the output identical to the expected string?" to "is the output acceptable within a defined tolerance?"

Reducing variance so tests become repeatable

Before you can measure quality reliably you must shrink the noise. Several controls make a GenAI system as reproducible as possible:

Temperature 0 / greedy decoding. Temperature scales the sampling randomness. At temperature 0 the model always picks the highest-probability token (greedy decoding), which minimises run-to-run variation. Use it for regression tests where you want the most stable output. It does not guarantee bit-identical results across hardware, and it can mask output-diversity issues you may want to test separately.

Fixed seed where supported. Some APIs accept a random-seed parameter; pinning it makes sampling reproducible for a given backend. Treat it as best-effort, because providers often document the seed as non-guaranteed.

Pinned model version. Always test against an explicit, dated model snapshot (for example model-2026-05 ) rather than a floating "latest" alias. A silent upgrade is the single most common cause of a suite that passed yesterday and fails today with no code change.

Fixed prompt template, parameters, and context. Hold the system prompt, max-tokens, top-p, and any retrieved context constant so that the only variable under test is the one you intend to exercise.