11.4 RAG and Agent Performance Evaluation

AWS Certified Generative AI Developer - Professional AIP-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & Foundation Model Fundamentals

1.1 AIP-C01 Exam Structure, Logistics & Passing Strategy 1.2 AIP-C01 Scoring & Professional Decision Strategy 1.3 Foundation Model Landscape & Model Selection on Amazon Bedrock 1.4 Model Abstraction, Controlled Switching & Resilience 1.5 Customization Spectrum: Prompting vs RAG vs Fine-Tuning vs Continued Pre-training 1.6 Custom Model Deployment & Lifecycle Management

2 Chapter 2: Bedrock Knowledge Bases & RAG Architectures

2.1 Managed RAG Architecture with Bedrock Knowledge Bases 2.2 Knowledge Source Integration, Synchronization & Freshness 2.3 Data Ingestion, Parsing & Chunking Strategies 2.4 Embedding Models & Vector Representations 2.5 Vector Index Scale, Migration & Retrieval Evaluation 2.6 Vector Database Selection: OpenSearch Serverless vs Aurora pgvector vs Managed Stores 2.7 Vector Store Security, Multi-Tenancy & Connectivity

3 Chapter 3: Advanced Vector Search, Indexing & Data Compliance

3.1 Hybrid Search, Metadata Filtering & Semantic Reranking 3.2 Metadata Filtering, Reranking & Query Transformation 3.3 Fine-Tuning & Continued Pre-Training Data Pipeline Engineering 3.4 Data Validation, Multimodal Processing & Model Formatting 3.5 Data Privacy, Data Residency & Compliance Governance 3.6 Privacy Preservation, Source Lineage & Audit Evidence

4 Chapter 4: Prompt Engineering & Bedrock Runtime APIs

4.1 Advanced Prompt Engineering Techniques & Inference Parameters 4.2 Inference Parameters, Prompt Governance & Regression 4.3 Converse API vs InvokeModel API & Message Formatting 4.4 Tool Calling, FM API Resilience & Model Routing 4.5 Streaming Architectures & Asynchronous Event Processing 4.6 Resilient Streaming, Deployment & Asynchronous Processing

5 Chapter 5: Amazon Bedrock Agents & Tool Integration

5.1 Agentic Architecture, ReAct Reasoning Loops & Orchestration 5.2 Agent Frameworks, MCP & Lifecycle Management 5.3 Action Groups, OpenAPI Schemas & AWS Lambda Integration 5.4 Agent Memory, Session State & Multi-Agent Collaboration Patterns 5.5 Persistent State & Multi-Agent Coordination 5.6 Advanced Agent Prompt Templates & Flow Customization 5.7 Agent Safeguards, Tool Reliability & Cost Control

6 Chapter 6: Workflow Orchestration & Application Integration

6.1 Amazon Bedrock Prompt Flows & Visual Orchestration 6.2 Step Functions Orchestration & Human-in-the-Loop Workflows 6.3 Enterprise Integration: API Gateway, AppSync, and EventBridge 6.4 Hybrid Integration, GenAI Gateways & CI/CD Tooling

7 Chapter 7: AI Safety & Guardrails for Amazon Bedrock

7.1 Guardrails Architecture, Denied Topics & Content Filters 7.2 Content Safety, Defense in Depth & Threat Detection 7.3 Sensitive Information Filters & PII Redaction/Masking 7.4 Contextual Grounding Checks & Hallucination Prevention

8 Chapter 8: Security, Access Control & Network Architecture

8.1 IAM Least Privilege & Resource Policies for Bedrock 8.2 PrivateLink VPC Endpoints & Network Isolation 8.3 Encryption with AWS KMS & Cross-Account Access Patterns

9 Chapter 9: Governance, Auditing & Operational Logging

9.1 SageMaker Model Cards & AI Governance Frameworks 9.2 Lineage, Fairness, Transparency & Continuous Governance 9.3 Invocation Logging, CloudTrail Auditing & Observability 9.4 Responsible AI, Bias Mitigation & Red Teaming Methodologies

10 Chapter 10: Performance, Throughput & Cost Optimization

10.1 On-Demand vs Provisioned Throughput & Capacity Planning 10.2 Capacity Measurement, Scaling Patterns & Resilience 10.3 Batch Inference & Bedrock Intelligent Prompt Routing 10.4 Intelligent Prompt Routing & Application Eligibility 10.5 Prompt Caching, Token Optimization & Latency Reduction

11 Chapter 11: Testing, Evaluation & Troubleshooting GenAI Solutions

11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows) 11.2 Human, RAG & Continuous Evaluation 11.3 Evaluation Metrics & RAG Quality Benchmarking (Ragas, ROUGE, BLEU) 11.4 RAG and Agent Performance Evaluation 11.5 CloudWatch Metrics, Throttling & Operational Observability 11.6 Quota Diagnosis, Retry Control & Operational Troubleshooting 11.7 Troubleshooting Common GenAI Failure Modes & Agent Traces 11.8 Prompt Maintenance, Content Handling & Deployment Validation

11.4 RAG and Agent Performance Evaluation

Key Takeaways Explain This Topic AI

Use retrieve-only and retrieve-and-generate metrics to distinguish discovery from generation failures.

Agent success includes correct tools, valid arguments, safe stopping, latency, cost, and business outcome.

A plausible final answer can conceal failed or unauthorized tool actions, so inspect task traces and outcomes.

Last updated: September 2026

11.4 RAG and Agent Performance Evaluation
Ragas Framework Deep-Dive (Retrieval Augmented Generation Assessment)

Ragas is an open-source evaluation framework specifically architected to benchmark RAG pipelines using "LLM-as-a-judge" methodologies. Ragas evaluates four core metrics that map directly to the RAG Triad: ┌───────────────────┐

│ User Query │
└─────────┬─────────┘
│
Context Precision│▲ Context Recall
(Retriever Evaluation)
│
▼
┌───────────────────┐
│ Retrieved Context │
└─────────┬─────────┘
│
Faithfulness │ Answer Relevance
(Generator Evaluation)
│
▼
┌───────────────────┐
│ Generated Answer │
└───────────────────┘
1. Faithfulness (Groundedness)

Definition: The ratio of claims made in the generated answer that can be directly verified against the retrieved context.

Scoring Process: The evaluator LLM parses the generated answer into discrete atomic claims. For each claim, the evaluator checks if it is logically entailed by the retrieved context chunks.

Formula: Faithfulness = ∣ Number of verifiable claims in answer supported by context ∣ ∣ Total claims generated in answer ∣ \text{Faithfulness} = \frac{|\text{Number of verifiable claims in answer supported by context}|}{|\text{Total claims generated in answer}|} Faithfulness = ∣ Total claims generated in answer ∣ ∣ Number of verifiable claims in answer supported by context ∣

Target: A score approaching 1.0 indicates zero hallucination.

2. Answer Relevance

Definition: Measures how pertinent the generated response is to the input query.

Scoring Process: Ragas instructs an evaluator model to generate N N N potential questions based exclusively on the generated answer. It then calculates the mean cosine similarity between the embeddings of the generated questions and the original user query.

Key Characteristic: Highly penalizes evasive answers, incomplete responses, or tangential verbosity, even if the answer is factually grounded.

3. Context Precision

Definition: Evaluates whether all ground-truth-relevant context chunks are ranked at the very top of the retrieved context list.

Calculation: Employs Mean Reciprocal Rank (MRR) and Average Precision principles. If relevant chunks are buried at positions 4 and 5 while positions 1 and 2 contain irrelevant filler, Context Precision drops significantly.

Impact: LLMs suffer from the "lost in the middle" phenomenon—information placed at the top or bottom of a context window is recalled far more reliably than information in the middle.

4. Context Recall

Definition: Measures whether the retrieved context contains all the necessary information required to answer the prompt, evaluated against a reference ground truth.

Formula: Context Recall = ∣ Number of reference ground-truth sentences attributable to context ∣ ∣ Total sentences in reference ground truth ∣ \text{Context Recall} = \frac{|\text{Number of reference ground-truth sentences attributable to context}|}{|\text{Total sentences in reference ground truth}|} Context Recall = ∣ Total sentences in reference ground truth ∣ ∣ Number of reference ground-truth sentences attributable to context ∣

Impact: Reflects the fundamental performance of the retrieval engine (embedding model + vector index).

Diagnostic Matrix: Pinpointing & Fixing RAG Failure Signatures

The true power of Ragas metrics lies in cross-metric diagnostic analysis. By inspecting metric combinations, developers can pinpoint the exact architectural component causing failures and apply targeted AWS remediations:

Diagnostic Metric Signature Root Cause Analysis Prescribed AWS Architectural Remediation

High Context Recall (0.95+) ; Low Faithfulness (<0.50) LLM Hallucination: The retriever found all necessary facts, but the generator model ignored the context or fabricated unsupported assertions. 1. Attach an Amazon Bedrock Guardrail with Contextual Grounding Checks (grounding threshold ≥ 0.75 \ge 0.75 ≥ 0.75 ).; 2. Lower model temperature to 0.0 or 0.1.; 3. Refine system prompts to strictly mandate: "Answer only using provided documents. State 'I do not know' if information is missing."

Low Context Precision (<0.40) ; High Faithfulness (0.90+) Retrieval Noise: The vector store retrieved too many irrelevant or tangentially related chunks. The model answered correctly from the few good chunks, but latency and token costs are inflated. 1. Implement a Reranker Model (e.g., Cohere Rerank on Bedrock) to re-order chunks by relevance.; 2. Add Metadata Filtering in Bedrock Knowledge Bases to constrain search scope.; 3. Reduce numberOfResults (top-K) parameter.

Low Context Recall (<0.40) ; Low Faithfulness (<0.50) Retriever Inadequacy: The retriever failed to locate relevant chunks from the vector database, forcing the LLM to either refuse or hallucinate answers. 1. Switch from pure vector search to Hybrid Search (combining dense vector embeddings with sparse BM25 lexical search).; 2. Optimize chunking strategy (e.g., transition from fixed-size 300-token chunking to hierarchical or semantic chunking).; 3. Upgrade the embedding model (e.g., to Amazon Titan Text Embeddings V2).

High Faithfulness (0.95+) ; Low Answer Relevance (<0.40) Model Drift / Prompt Evasion: The model grounded its answer in the context, but failed to address the user's actual question (e.g., returning generic boilerplate or answering a different question). 1. Implement automated Query Rewriting or query decomposition prior to retrieval.; 2. Provide few-shot exemplar demonstrations in the system prompt illustrating direct answers.; 3. Switch to a more capable reasoning foundation model (e.g., Claude 3.5 Sonnet).

Important

Exam Diagnostic Tip: On the AIP-C01 exam, if a question describes a scenario where "the retrieved passages contain the exact answer, but the model outputs unverified claims not present in the reference documents," the failure is Low Faithfulness (hallucination). The correct architectural fix on AWS is configuring Contextual Grounding Checks in Amazon Bedrock Guardrails , not altering the chunking strategy or vector database.

Native RAG and agent evaluation

Amazon Bedrock RAG evaluation can score retrieve-only behavior separately from retrieve-and-generate behavior. Current built-in concepts include context relevance and coverage for retrieval and correctness, completeness, helpfulness, coherence, faithfulness, citation quality, harmfulness, stereotyping, and refusal for generated responses. Supply ground truth where the selected metric requires it and preserve evaluator-model configuration.

Agent evaluation adds task-level evidence. Measure task completion, correct tool selection, parameter validity, tool success, unnecessary calls, latency, cost, stopping behavior, and the quality of multi-step outcomes. Amazon Bedrock agent evaluations can support managed assessment for eligible agents; custom traces and golden workflows cover application-specific tools. Do not grade an agent only on final prose—a plausible answer can hide a failed or unauthorized action path.