Study guide

3.3 Adversarial prompts & red-teaming

ISTQB Certified Tester — Testing with Generative AI CT-GenAI

Track your progress as you read

Finished sections get a check mark.

1 GenAI Foundations for Testers

1.1 Transformers, tokenization, embeddings & context window 1.2 Foundation models, temperature/sampling & prompting basics 1.3 RAG & agent architectures testers must understand

2 Quality Attributes for GenAI

2.1 Faithfulness, factuality, relevance, coherence & fluency 2.2 Safety, toxicity & bias 2.3 Robustness, privacy, latency & cost

3 Test Design for Non-Determinism

3.1 Controlling non-determinism (seeds, temperature, golden datasets) 3.2 Perturbation & metamorphic testing 3.3 Adversarial prompts & red-teaming

4 GenAI Risks & Mitigation

4.1 Hallucinations: detection & mitigation 4.2 Prompt injection, jailbreaks & training-data extraction 4.3 Governance frameworks (NIST AI RMF, EU AI Act, model cards)

5 Test Infrastructure & Tooling

5.1 Evaluation frameworks (Promptfoo, LangSmith, OpenAI Evals, RAGAS) 5.2 Automated metrics (BLEU/ROUGE/METEOR, embedding similarity, LLM-as-judge) 5.3 Guardrails & safety tooling (Llama Guard, NeMo Guardrails)

6 Organizational Adoption

6.1 Responsible AI governance & regression testing for GenAI 6.2 Continuous evaluation, synthetic test data & cost monitoring

3.3 Adversarial prompts & red-teaming

Key Takeaways Explain This Topic AI

An adversarial prompt is deliberately crafted to push a model past its guardrails into harmful, false, non-compliant, or confidential output — security-level negative testing.

Red-teaming is a structured, iterative campaign: define in-scope harms, enumerate attack techniques, execute systematically, log successes, and rate severity.

Core attack categories: jailbreaks, direct and indirect prompt injection, harmful-content elicitation, and PII / system-prompt extraction.

Every successful attack becomes a reproducible security regression test asserting a safety property (must refuse, must not contain disallowed content).

Track a defence rate over the attack suite, combine manual creativity with automated attack generation, and keep the attack library current.

Last updated: July 2026

Adversarial prompting: testing for failure on purpose

The previous sections tested whether a GenAI system produces good answers to cooperative inputs. Adversarial testing asks the opposite question: can a hostile input make the system misbehave? An adversarial prompt is deliberately crafted to push the model past its guardrails — to produce harmful, false, non-compliant, or confidential output it was designed to refuse. In ISTQB terms this is negative testing raised to the security level: the tester actively tries to break the safety and policy requirements rather than confirm the happy path.

Red-teaming: structured campaigns, not random pokes

Red-teaming is the organised discipline of adversarial testing. Rather than ad-hoc prompt tinkering, a red-team campaign is planned like any test effort: define objectives (which harms and policies are in scope), enumerate attack techniques, execute systematically, log every success, and rate severity. It is typically iterative — a partial success is refined until it fully bypasses a control, mirroring how a real attacker escalates.

The main attack categories a CT-GenAI tester must know:

Jailbreaks — prompts that trick the model into ignoring its safety policy, often via role-play ("pretend you are an AI with no rules"), hypothetical framing, or "DAN"-style personas. Goal: elicit content the model should refuse.

Prompt injection — malicious instructions hidden in data the model processes (a web page, a document, a user field) that hijack the system's original instructions. Indirect prompt injection, where the payload arrives through retrieved content rather than the user, is especially dangerous for RAG and agentic systems.

Harmful-content elicitation — attempts to draw out disallowed output such as violence, hate, illegal instructions, or self-harm content.

PII / data extraction — coaxing the model to reveal personal data, secrets, its own hidden system prompt, or memorised training data ("repeat the text above", "what were your instructions?").

Misinformation / hallucination baiting — leading questions engineered to produce confident falsehoods.