Study guide

5.2 Structured Output and Evaluation

Claude Certified Architect - Foundations Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam and Ecosystem Overview

1.1 Certification Context 1.2 Study Map

2 Chapter 2: Agentic Architecture and Orchestration

2.1 Workflows vs Agents 2.2 Agentic Loops and the Claude Agent SDK 2.3 Multi-Agent Orchestration and Session Management

3 Chapter 3: Claude Code Configuration and Workflows

3.1 Claude Code Workflows

4 Chapter 4: Tool Design and MCP Integration

4.1 Tool Design and MCP Integration

5 Chapter 5: Prompt Engineering and Structured Output

5.1 Prompt Engineering 5.2 Structured Output and Evaluation

6 Chapter 6: Context Management and Reliability

6.1 Context, Caching, and Compaction 6.2 Model Selection and Reliability

5.2 Structured Output and Evaluation

Key Takeaways Explain This Topic AI

When output feeds code, design the JSON schema first, then write the prompt around that contract — schema-first prevents output that reads well but breaks the parser.

Programmatic enforcement (strict schema / structured outputs) beats prompt-based pleading; pair it with a validate-and-retry loop and an explicit failure path.

Constrain values with enums and mark required fields so the model cannot drift into free text or omit critical data.

An eval set must cover known successes and known failures — tool errors, invalid JSON, stale retrieval, ambiguous requests, and safety-sensitive prompts.

Run regression evals before and after any model upgrade, prompt edit, or new integration so a silent quality drop is caught before users see it.

Last updated: June 2026

Schema-First Habit

If downstream code must parse the answer, design the schema before the prompt . Enumerate required fields, enum values, nested objects, and — critically — what happens on a validation failure. Then write the prompt around that contract. This prevents the classic failure the exam loves: output that reads beautifully to a human but breaks JSON.parse or a strict validator in production.

A schema-first object for a support-ticket extractor might require category (enum: billing, technical, account), priority (enum: low, medium, high), summary (string), and needs_human (boolean). Every field is named, every constrained field is an enum, and required lists the fields that must always appear. With the contract fixed, the prompt simply instructs Claude to fill it, and the validator rejects anything that does not conform.

Keep the schema as simple as the task allows. Strict structured-output schemas support common JSON types, enums, and nested objects, but some constraints (numeric ranges, string-length limits, recursive structures) are best enforced in your own validation code after parsing rather than expressed in the schema. The exam's point is the ordering discipline: fix the contract, generate against it, then validate the parsed object — not the precise list of supported keywords.

Programmatic Enforcement vs Prompt-Based Guidance

The exam draws a sharp line between asking the model for a format and enforcing it. Programmatic enforcement is stronger and is the preferred answer:

  • Approach Mechanism Reliability
  • Prompt-based guidance "Please respond in JSON matching this shape" Weakest — the model can still drift
  • Few-shot format steering Examples show the exact shape Better, but no guarantee

Structured outputs / strict schema The API constrains the response to a JSON schema, or a tool with strict: true validates inputs exactly Strongest — output is guaranteed to match

Validate-and-retry Parse and validate in code; on failure, re-prompt with the error The safety net around any of the above

The robust production shape is: enforce the schema with structured outputs or strict tool use, then validate the parsed object in code and retry with the validation error fed back if it somehow fails. Always define the failure path : after N failed retries, return a safe default, escalate to a human, or surface a typed error — never ship unvalidated output downstream.

Evaluate Failure Paths, Not Just Demos

Evaluation is how an architect proves a system works and keeps working. The most common CCA-F trap is an answer that only evaluates happy paths. A real eval set deliberately includes hard cases:

  • Ambiguous or underspecified user requests
  • Tool calls that error or time out
  • Invalid or unparseable JSON output
  • Stale or empty retrieval results
  • Adversarial prompts and safety-sensitive scenarios

A system that passes only on clean demo inputs is not production-ready. State the testable failure mode first , then pick the control that detects or mitigates it; that converts a vague "make it better" goal into observable behavior a team can measure and alert on.

Grading Methods

Different outputs need different graders, and the exam expects you to match them:

  • Output type Grading method
  • Exact value (classification, extracted field) Exact-match / programmatic assertion
  • Structured object Schema validation plus field-by-field checks
  • Open-ended prose Rubric-based scoring, often LLM-as-judge with a fresh context

Faithfulness to sources Citation checks; plant a question whose answer is absent and confirm the system abstains

Programmatic checks are cheap and deterministic; reserve LLM-as-judge for genuinely open-ended quality where a rubric is the only practical grader.

Regression Evals as a Change Gate

Evals earn their keep when they run as a regression gate . Maintain a small, version-controlled eval set and re-run it before and after any change that could move quality: a model upgrade , a prompt edit , a new tool or MCP integration , or a retrieval change. If a score drops, the change is blocked or flagged before it reaches users. Anthropic frames evals as the safety net that catches problems before real users do — and the exam consistently rewards the answer that adds a regression eval over the answer that simply adds another retry or reads the output by hand.

Close the loop: sample real production traffic into the eval set so the suite reflects how users actually behave, and alert when a live metric (schema-validation failure rate, refusal rate, citation-missing rate, p95 latency, cost per request) drifts past a threshold. An eval suite that never grows from real failures slowly stops representing reality.

Structured Output and Eval Decision Table

  • Symptom in the scenario First control to reach for
  • Output sometimes breaks the parser Structured outputs / strict schema + validate-and-retry
  • Field drifts into arbitrary free text Constrain it with an enum and mark it required
  • Quality dropped after a prompt edit Regression eval set run as a change gate
  • System only tested on clean demos Add failure-path cases to the eval set
  • Open-ended answer needs scoring Rubric / LLM-as-judge with fresh context
  • Need to prove an upgrade is safe Run the eval suite before and after, compare scores

Match the symptom to the control, define the failure path explicitly, and prefer programmatic enforcement over hoping the prompt holds — that discipline is exactly what this half of Domain 4 measures.