Study guide
11.2 Human, RAG & Continuous Evaluation
AWS Certified Generative AI Developer - Professional AIP-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & Foundation Model Fundamentals
1.1 AIP-C01 Exam Structure, Logistics & Passing Strategy 1.2 AIP-C01 Scoring & Professional Decision Strategy 1.3 Foundation Model Landscape & Model Selection on Amazon Bedrock 1.4 Model Abstraction, Controlled Switching & Resilience 1.5 Customization Spectrum: Prompting vs RAG vs Fine-Tuning vs Continued Pre-training 1.6 Custom Model Deployment & Lifecycle Management
2 Chapter 2: Bedrock Knowledge Bases & RAG Architectures
2.1 Managed RAG Architecture with Bedrock Knowledge Bases 2.2 Knowledge Source Integration, Synchronization & Freshness 2.3 Data Ingestion, Parsing & Chunking Strategies 2.4 Embedding Models & Vector Representations 2.5 Vector Index Scale, Migration & Retrieval Evaluation 2.6 Vector Database Selection: OpenSearch Serverless vs Aurora pgvector vs Managed Stores 2.7 Vector Store Security, Multi-Tenancy & Connectivity
3 Chapter 3: Advanced Vector Search, Indexing & Data Compliance
3.1 Hybrid Search, Metadata Filtering & Semantic Reranking 3.2 Metadata Filtering, Reranking & Query Transformation 3.3 Fine-Tuning & Continued Pre-Training Data Pipeline Engineering 3.4 Data Validation, Multimodal Processing & Model Formatting 3.5 Data Privacy, Data Residency & Compliance Governance 3.6 Privacy Preservation, Source Lineage & Audit Evidence
4 Chapter 4: Prompt Engineering & Bedrock Runtime APIs
4.1 Advanced Prompt Engineering Techniques & Inference Parameters 4.2 Inference Parameters, Prompt Governance & Regression 4.3 Converse API vs InvokeModel API & Message Formatting 4.4 Tool Calling, FM API Resilience & Model Routing 4.5 Streaming Architectures & Asynchronous Event Processing 4.6 Resilient Streaming, Deployment & Asynchronous Processing
5 Chapter 5: Amazon Bedrock Agents & Tool Integration
5.1 Agentic Architecture, ReAct Reasoning Loops & Orchestration 5.2 Agent Frameworks, MCP & Lifecycle Management 5.3 Action Groups, OpenAPI Schemas & AWS Lambda Integration 5.4 Agent Memory, Session State & Multi-Agent Collaboration Patterns 5.5 Persistent State & Multi-Agent Coordination 5.6 Advanced Agent Prompt Templates & Flow Customization 5.7 Agent Safeguards, Tool Reliability & Cost Control
6 Chapter 6: Workflow Orchestration & Application Integration
6.1 Amazon Bedrock Prompt Flows & Visual Orchestration 6.2 Step Functions Orchestration & Human-in-the-Loop Workflows 6.3 Enterprise Integration: API Gateway, AppSync, and EventBridge 6.4 Hybrid Integration, GenAI Gateways & CI/CD Tooling
7 Chapter 7: AI Safety & Guardrails for Amazon Bedrock
7.1 Guardrails Architecture, Denied Topics & Content Filters 7.2 Content Safety, Defense in Depth & Threat Detection 7.3 Sensitive Information Filters & PII Redaction/Masking 7.4 Contextual Grounding Checks & Hallucination Prevention
8 Chapter 8: Security, Access Control & Network Architecture
8.1 IAM Least Privilege & Resource Policies for Bedrock 8.2 PrivateLink VPC Endpoints & Network Isolation 8.3 Encryption with AWS KMS & Cross-Account Access Patterns
9 Chapter 9: Governance, Auditing & Operational Logging
9.1 SageMaker Model Cards & AI Governance Frameworks 9.2 Lineage, Fairness, Transparency & Continuous Governance 9.3 Invocation Logging, CloudTrail Auditing & Observability 9.4 Responsible AI, Bias Mitigation & Red Teaming Methodologies
10 Chapter 10: Performance, Throughput & Cost Optimization
10.1 On-Demand vs Provisioned Throughput & Capacity Planning 10.2 Capacity Measurement, Scaling Patterns & Resilience 10.3 Batch Inference & Bedrock Intelligent Prompt Routing 10.4 Intelligent Prompt Routing & Application Eligibility 10.5 Prompt Caching, Token Optimization & Latency Reduction
11 Chapter 11: Testing, Evaluation & Troubleshooting GenAI Solutions
11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows) 11.2 Human, RAG & Continuous Evaluation 11.3 Evaluation Metrics & RAG Quality Benchmarking (Ragas, ROUGE, BLEU) 11.4 RAG and Agent Performance Evaluation 11.5 CloudWatch Metrics, Throttling & Operational Observability 11.6 Quota Diagnosis, Retry Control & Operational Troubleshooting 11.7 Troubleshooting Common GenAI Failure Modes & Agent Traces 11.8 Prompt Maintenance, Content Handling & Deployment Validation
11.2 Human, RAG & Continuous Evaluation
Key Takeaways Explain This Topic AI
Programmatic human evaluations use a SageMaker flow definition and private work-team ARN.
Evaluate retrieval and response generation separately to target the actual failure stage.
Continuous evaluation reruns after model, prompt, Guardrail, retrieval, tool, or incident changes.
Last updated: September 2026
- 11.2 Human, RAG & Continuous Evaluation
- Human-based evaluation
Human review is appropriate for tone, usefulness, complex reasoning, policy interpretation, and high-risk sign-off. Current Bedrock workflows use a private work team. When created programmatically, the workflow uses a SageMaker AI flow definition and work-team ARN. The evaluation job connects prompts and responses to that human workflow.
Write a scoring rubric with observable anchors. 'Good' is not reproducible; 'all claims are supported by the cited policy and no required exception is omitted' is testable. Use multiple reviewers for subjective or high-risk tasks, measure agreement, adjudicate disagreements, and train reviewers with examples.
Do not claim that proprietary prompts can never leave an organization's boundary solely because a private workforce was selected. Verify data flow, worker identity, S3 and KMS access, retention, browser environment, contractual controls, and the organization's definition of its security boundary.
RAG evaluation
Evaluate retrieval separately from response generation. Bedrock RAG evaluations support retrieve-only and retrieve-and-generate jobs for supported sources. Retrieve-only metrics include whether context is relevant and covers expected evidence. Generate-stage metrics include correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision and coverage, harmfulness, stereotyping, and refusal, depending on the chosen workflow.
A retrieval failure requires changes to source data, chunking, embeddings, metadata, query transformation, top-k, or reranking. A generation failure with good evidence requires changes to prompt, model, Guardrails, or response validation. An end-to-end score alone obscures this distinction.
Continuous evaluation and release gates
Run a fixed regression suite for every material model, prompt, Guardrail, retrieval, or tool change. Add incident examples to a separate regression set after remediation. Compare the candidate with the active release using confidence intervals or repeated samples where nondeterminism matters. Require safety and policy thresholds as hard gates even when average quality improves.
A production gate can require:
Dataset and configuration validation.
Offline model and RAG metrics above thresholds.
Red-team and structured-output tests.
Human review for high-risk slices.
Canary traffic with rollback triggers.
Post-deployment monitoring and sampled reevaluation.
Evaluation reports should identify the model, configuration, dataset version, metric implementation, timestamp, sample size, uncertainty, failures, and approval decision. A green average without named failures is not sufficient evidence for a professional production release.
Human rubric and stakeholder report
A rubric defines each metric, scale, and anchor. For a five-point faithfulness score, describe what fully supported, partially supported, contradicted, and unverifiable answers look like. Show reviewers examples, randomize candidate order for pairwise tasks, and measure agreement. Adjudicate low-agreement items rather than averaging confusion.
Limit each reviewer to authorized data. Use a private work team, role-scoped S3 access, optional KMS controls, time-bounded task availability, and a retention plan. Review worker instructions for accidental disclosure and avoid placing secrets in free-text examples.
The final report includes the evaluated release, dataset version, model or source configuration, metrics, thresholds, slice results, uncertainty, human-review method, failed examples, known limitations, and release decision. Visualizations should help stakeholders find tradeoffs rather than conceal them in one composite score.
After a canary, join offline metrics with user feedback, task completion, overrides, incidents, latency, and cost. Production data can reveal distribution shift, but feedback is biased toward users who choose to respond. Sample deliberately and continue scheduled benchmark runs.
Feedback loop controls
User ratings are signals, not labels automatically suitable for training. Detect spam, selection bias, duplicate feedback, and complaints caused by factors outside the model. Route high-risk negative feedback to review, redact sensitive content, and obtain the rights needed before reuse.
Annotation workflows need a queue, instructions, reviewer identity, disagreement handling, and quality checks such as known control items. Preserve the connection among prompt, response, cited evidence, release version, and rating. Otherwise the team cannot tell which component improved.
Close the loop by creating a candidate fix and rerunning the fixed benchmark plus the broader regression suite. Do not update production prompts continuously from raw ratings without review, because malicious or unrepresentative feedback can steer behavior.
Track evaluation debt: known failed slices, deferred metrics, reviewer disagreement, and missing production evidence. An approval can be conditional, but the condition needs an owner, due date, traffic limit, and automated guardrail against silent expansion.
For high-impact workflows, retain a small adjudicated benchmark that remains stable across releases. It provides continuity when user traffic and feedback distributions change, while the larger evolving dataset captures new failure modes.