Study guide
11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows)
AWS Certified Generative AI Developer - Professional AIP-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & Foundation Model Fundamentals
1.1 AIP-C01 Exam Structure, Logistics & Passing Strategy 1.2 AIP-C01 Scoring & Professional Decision Strategy 1.3 Foundation Model Landscape & Model Selection on Amazon Bedrock 1.4 Model Abstraction, Controlled Switching & Resilience 1.5 Customization Spectrum: Prompting vs RAG vs Fine-Tuning vs Continued Pre-training 1.6 Custom Model Deployment & Lifecycle Management
2 Chapter 2: Bedrock Knowledge Bases & RAG Architectures
2.1 Managed RAG Architecture with Bedrock Knowledge Bases 2.2 Knowledge Source Integration, Synchronization & Freshness 2.3 Data Ingestion, Parsing & Chunking Strategies 2.4 Embedding Models & Vector Representations 2.5 Vector Index Scale, Migration & Retrieval Evaluation 2.6 Vector Database Selection: OpenSearch Serverless vs Aurora pgvector vs Managed Stores 2.7 Vector Store Security, Multi-Tenancy & Connectivity
3 Chapter 3: Advanced Vector Search, Indexing & Data Compliance
3.1 Hybrid Search, Metadata Filtering & Semantic Reranking 3.2 Metadata Filtering, Reranking & Query Transformation 3.3 Fine-Tuning & Continued Pre-Training Data Pipeline Engineering 3.4 Data Validation, Multimodal Processing & Model Formatting 3.5 Data Privacy, Data Residency & Compliance Governance 3.6 Privacy Preservation, Source Lineage & Audit Evidence
4 Chapter 4: Prompt Engineering & Bedrock Runtime APIs
4.1 Advanced Prompt Engineering Techniques & Inference Parameters 4.2 Inference Parameters, Prompt Governance & Regression 4.3 Converse API vs InvokeModel API & Message Formatting 4.4 Tool Calling, FM API Resilience & Model Routing 4.5 Streaming Architectures & Asynchronous Event Processing 4.6 Resilient Streaming, Deployment & Asynchronous Processing
5 Chapter 5: Amazon Bedrock Agents & Tool Integration
5.1 Agentic Architecture, ReAct Reasoning Loops & Orchestration 5.2 Agent Frameworks, MCP & Lifecycle Management 5.3 Action Groups, OpenAPI Schemas & AWS Lambda Integration 5.4 Agent Memory, Session State & Multi-Agent Collaboration Patterns 5.5 Persistent State & Multi-Agent Coordination 5.6 Advanced Agent Prompt Templates & Flow Customization 5.7 Agent Safeguards, Tool Reliability & Cost Control
6 Chapter 6: Workflow Orchestration & Application Integration
6.1 Amazon Bedrock Prompt Flows & Visual Orchestration 6.2 Step Functions Orchestration & Human-in-the-Loop Workflows 6.3 Enterprise Integration: API Gateway, AppSync, and EventBridge 6.4 Hybrid Integration, GenAI Gateways & CI/CD Tooling
7 Chapter 7: AI Safety & Guardrails for Amazon Bedrock
7.1 Guardrails Architecture, Denied Topics & Content Filters 7.2 Content Safety, Defense in Depth & Threat Detection 7.3 Sensitive Information Filters & PII Redaction/Masking 7.4 Contextual Grounding Checks & Hallucination Prevention
8 Chapter 8: Security, Access Control & Network Architecture
8.1 IAM Least Privilege & Resource Policies for Bedrock 8.2 PrivateLink VPC Endpoints & Network Isolation 8.3 Encryption with AWS KMS & Cross-Account Access Patterns
9 Chapter 9: Governance, Auditing & Operational Logging
9.1 SageMaker Model Cards & AI Governance Frameworks 9.2 Lineage, Fairness, Transparency & Continuous Governance 9.3 Invocation Logging, CloudTrail Auditing & Observability 9.4 Responsible AI, Bias Mitigation & Red Teaming Methodologies
10 Chapter 10: Performance, Throughput & Cost Optimization
10.1 On-Demand vs Provisioned Throughput & Capacity Planning 10.2 Capacity Measurement, Scaling Patterns & Resilience 10.3 Batch Inference & Bedrock Intelligent Prompt Routing 10.4 Intelligent Prompt Routing & Application Eligibility 10.5 Prompt Caching, Token Optimization & Latency Reduction
11 Chapter 11: Testing, Evaluation & Troubleshooting GenAI Solutions
11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows) 11.2 Human, RAG & Continuous Evaluation 11.3 Evaluation Metrics & RAG Quality Benchmarking (Ragas, ROUGE, BLEU) 11.4 RAG and Agent Performance Evaluation 11.5 CloudWatch Metrics, Throttling & Operational Observability 11.6 Quota Diagnosis, Retry Control & Operational Troubleshooting 11.7 Troubleshooting Common GenAI Failure Modes & Agent Traces 11.8 Prompt Maintenance, Content Handling & Deployment Validation
11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows)
Key Takeaways Explain This Topic AI
CreateEvaluationJob is the current Bedrock evaluation API operation.
Judge models need calibration against expert-reviewed examples because judge scores inherit bias and prompt sensitivity.
Evaluate retrieval and generation separately so remediation targets the actual failing stage.
Last updated: September 2026
11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows)
Evaluation determines whether a model or RAG system meets a defined task, safety, and business contract. Current Amazon Bedrock evaluations support programmatic model evaluation, human-based evaluation, judge-model evaluation, and RAG evaluation. They can evaluate supported Bedrock resources and, for some workflows, responses produced outside Bedrock. Use the current CreateEvaluationJob API; the older name CreateModelEvaluationJob is not the current control-plane operation.
Start with a testable acceptance contract
Define the task, population, risks, metrics, thresholds, dataset, and decision before running the job. Accuracy alone is insufficient for a customer assistant that also needs faithfulness, safety, latency, and appropriate refusal. Include ordinary cases, rare edge cases, adversarial inputs, protected-group slices where relevant, and previously observed incidents.
Keep a final holdout set that prompt and model designers did not use for tuning. Version the dataset and preserve provenance, licensing, consent, redaction, and access controls. A test set copied from production logs can contain secrets or personal data and requires the same governance as the source workload.
Programmatic and judge-model evaluation
Programmatic evaluation runs a selected model over built-in or custom prompt data and produces task metrics. Available task types, datasets, and metrics vary by current service support. A custom JSONL dataset in Amazon S3 is the normal choice for domain-specific acceptance testing. The service role needs access to the exact input and output locations, permission to invoke selected models, and KMS permissions when customer-managed keys protect the data.
Judge-model evaluation uses another foundation model to score and explain candidate responses. It scales semantic review better than exact-match metrics, but it inherits judge bias, prompt sensitivity, and position effects. Calibrate it against expert-reviewed examples and periodically test agreement. Never treat a judge score as ground truth merely because it is numeric.
Evaluation approach Strength Main limitation
Exact or reference metrics Reproducible and inexpensive Penalizes valid paraphrases and misses unsupported claims
- Judge model Flexible semantic and rubric scoring Judge bias and prompt sensitivity
- Human review Nuanced domain and policy judgment Cost, speed, and inter-rater variance
- Online outcome Measures real user impact Confounding factors and production risk
- Metric design and uncertainty
Choose metrics from the failure cost. Exact match fits a narrow deterministic answer but penalizes harmless wording changes. Semantic judge scores capture meaning but require calibration. Toxicity or harmfulness metrics address safety but do not prove factual correctness. Structured applications also need schema validity and business-rule checks.
Report sample size and distribution, not just a mean. Slice results by language, task, risk class, prompt length, and user population. Review the worst failures and compute uncertainty where practical. A candidate whose average improves by one point but has a new critical safety failure does not pass.
Generative outputs can vary. Fix model and prompt configuration, repeat nondeterministic cases, and distinguish evaluator variance from candidate variance. Preserve raw response identifiers and evaluator explanations under appropriate access so reviewers can audit disputed scores.
Use custom datasets for the actual domain, while retaining a smaller general-capability set to detect broad regressions. Prevent data leakage by keeping the final set away from prompt tuning, fine-tuning, and judge-prompt development.
Evaluation job mechanics
The service role should trust Bedrock and grant only the S3 prefixes, KMS keys, and model invocation resources needed by the job. Keep input and output buckets in supported locations and validate Cross-Origin Resource Sharing settings when the documented workflow requires them. An AccessDenied failure is an IAM or key-policy investigation, not evidence that custom datasets are unsupported.
CreateEvaluationJob accepts an evaluation configuration, inference source, dataset metrics, output location, and role appropriate to the selected workflow. Use current API schemas from the SDK rather than copying an obsolete sample. Store the submitted configuration beside the report so later reviewers know what was actually scored.
Judge and RAG jobs incur model inference and evaluation cost. Limit development samples, then run the approved holdout once the rubric stabilizes. Repeatedly tuning against the final set converts it into another development set.
Evaluation datasets and reports are production assets. Apply bucket versioning or immutable release prefixes, encryption, restricted roles, retention, and deletion procedures. Avoid placing one customer's confidential prompts in a shared benchmark visible to unrelated teams.
Treat a failed evaluation job separately from a failed model candidate. IAM, dataset schema, unsupported metrics, or evaluator access can fail the job before a meaningful score exists. Do not convert infrastructure failure into a zero quality score or a release pass.