Study guide

6.2 Continuous evaluation, synthetic test data & cost monitoring

ISTQB Certified Tester — Testing with Generative AI CT-GenAI

Track your progress as you read

Finished sections get a check mark.

1 GenAI Foundations for Testers

1.1 Transformers, tokenization, embeddings & context window 1.2 Foundation models, temperature/sampling & prompting basics 1.3 RAG & agent architectures testers must understand

2 Quality Attributes for GenAI

2.1 Faithfulness, factuality, relevance, coherence & fluency 2.2 Safety, toxicity & bias 2.3 Robustness, privacy, latency & cost

3 Test Design for Non-Determinism

3.1 Controlling non-determinism (seeds, temperature, golden datasets) 3.2 Perturbation & metamorphic testing 3.3 Adversarial prompts & red-teaming

4 GenAI Risks & Mitigation

4.1 Hallucinations: detection & mitigation 4.2 Prompt injection, jailbreaks & training-data extraction 4.3 Governance frameworks (NIST AI RMF, EU AI Act, model cards)

5 Test Infrastructure & Tooling

5.1 Evaluation frameworks (Promptfoo, LangSmith, OpenAI Evals, RAGAS) 5.2 Automated metrics (BLEU/ROUGE/METEOR, embedding similarity, LLM-as-judge) 5.3 Guardrails & safety tooling (Llama Guard, NeMo Guardrails)

6 Organizational Adoption

6.1 Responsible AI governance & regression testing for GenAI 6.2 Continuous evaluation, synthetic test data & cost monitoring

6.2 Continuous evaluation, synthetic test data & cost monitoring

Key Takeaways Explain This Topic AI

Continuous evaluation measures quality on live traffic, watching signals such as drift, guardrail hit-rate, groundedness, user feedback, latency, and cost.

Canary and A/B releases compare a new prompt or model against the current one on live metrics before full rollout, reusing the pre-release eval harness.

User feedback loops surface real failures that should be triaged and promoted into the golden dataset so regression tests keep improving.

LLM-generated synthetic test data expands coverage cheaply but risks bias amplification and wrong oracles, so it must be validated and only complement real data.

Treat cost as a first-class test dimension: track tokens and cost per eval run, set budgets and alerts, and use a tiered smoke-versus-full strategy.

Last updated: July 2026

Continuous evaluation, synthetic test data & cost monitoring

A GenAI system that passed every pre-release gate can still degrade in production: real user inputs differ from your test set, retrieval sources change, and the hosted model may be updated without notice. Continuous evaluation (production monitoring) is the discipline of measuring quality on live traffic, not only before release. The aim is to detect drift, quality regressions, and rising unsafe output while the problems are still small.

Signals to monitor

  • Signal What it tells you Example metric
  • Quality / model drift Behavior moving away from the tested baseline Rolling eval score on sampled live outputs

Guardrail hit-rate How often safety filters fire % responses blocked or redacted; a spike flags a new attack or prompt bug

  • User feedback Perceived quality from real users Thumbs up/down, regenerate rate, escalation-to-human rate
  • Groundedness (RAG) Whether answers are supported by sources % answers with valid citations; faithfulness score
  • Latency & availability Operational health p95 response time, error/timeout rate
  • Cost Spend trajectory Tokens per request, cost per 1,000 requests
  • Canary, A/B, and feedback loops

You rarely change everything at once. A canary release routes a small slice of traffic to the new prompt or model and compares its live metrics to the current version before full rollout. A/B testing runs two versions in parallel to see which performs better on quality and business metrics. Both reuse the same evaluation harness built for pre-release testing, now applied online. User feedback loops — explicit ratings plus implicit signals such as regeneration or abandonment — supply a stream of real failure cases; the most valuable are triaged and promoted into the golden dataset, so tomorrow's regression tests cover today's real problems.

Detecting drift without a fixed answer key

In production you rarely have a labeled correct answer for live traffic, so continuous evaluation leans on reference-free techniques: an LLM-as-judge scoring responses against a rubric, groundedness checks that verify each claim against retrieved sources, and statistical alarms on shifts in input distribution or output length. Because judging every request is expensive, teams sample — evaluating a representative slice of live traffic on a rolling window and alerting when the rolling score crosses a control limit, rather than reacting to any single output.