Study guide

6.5 Bedrock Model Evaluation, Monitoring, and Human Feedback

AWS AI Practitioner Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: AIF-C01 Orientation and Official Source Control

1.1 Credential Scope, Code, and Search Language 1.2 Exam Format, Cost, Score, and Delivery 1.3 Official Exam Guide, Skill Builder, and Practice Workflow 1.4 Target Candidate Boundaries and Out-of-Scope Tasks 1.5 Domain Weights and Study Prioritization 1.6 Exam Policy, Retake, Results, and Recertification

2 Chapter 2: AI/ML Foundations and Use-Case Fit

2.1 AI, ML, Deep Learning, and Core Terminology 2.2 Supervised, Unsupervised, and Reinforcement Learning 2.3 Data Types, Labels, Structure, and Quality 2.4 Inference Patterns: Batch, Real-Time, and Embedded AI 2.5 AI Use-Case Fit and No-AI Decisions 2.6 Classification, Regression, Clustering, Forecasting, and Recommendation 2.7 AI/ML Foundations Case Lab

3 Chapter 3: ML Lifecycle, Metrics, and Practitioner MLOps

3.1 Data Collection, EDA, Preprocessing, and Feature Concepts 3.2 Training, Evaluation, Deployment, and Monitoring Lifecycle 3.3 Model Sources: Managed APIs, Open Source, and Custom Models 3.4 Evaluation Metrics and Business Metrics 3.5 Practitioner MLOps: Repeatability, Monitoring, and Retraining 3.6 SageMaker Lifecycle Service Map 3.7 ML Lifecycle Case Lab

4 Chapter 4: Generative AI Foundations and Inference Concepts

4.1 Foundation Models, LLMs, Transformers, and Modalities 4.2 Tokens, Context Windows, Embeddings, and Vector Search 4.3 Inference Parameters: Temperature, Top-p, and Output Controls 4.4 Hallucination, Grounding, and Context Quality 4.5 GenAI Use-Case Fit and Risk Triage 4.6 Prompt Injection, Data Leakage, and User Input Risk 4.7 Generative AI Foundations Case Lab

5 Chapter 5: Prompting, Model Selection, Customization, and Evaluation

5.1 Prompt Engineering Patterns and Business Quality 5.2 Zero-Shot, Few-Shot, Templates, and Instruction Design 5.3 Model Selection: Capability, Latency, Cost, and Risk 5.4 RAG vs Fine-Tuning vs Prompting vs Custom Models 5.5 Model Evaluation, Human Review, and Red-Team Feedback 5.6 Cost, Performance, and Throughput Decision-Making 5.7 Prompting and Model Selection Case Lab

6 Chapter 6: Amazon Bedrock, RAG, Agents, and Guardrails

6.1 Bedrock Core Concepts, Model Access, and Managed FM Choice 6.2 Knowledge Bases, RAG, Data Sources, and Grounding 6.3 AI Agents, Tools, Orchestration, and Business Workflows 6.4 Guardrails, Content Filters, Denied Topics, and Sensitive Data 6.5 Bedrock Model Evaluation, Monitoring, and Human Feedback 6.6 Bedrock Cost, Latency, Throughput, and Operational Fit 6.7 Bedrock, RAG, and Agents Case Lab

7 Chapter 7: AWS Managed AI/ML Services and SageMaker Map

7.1 Managed AI Services vs Foundation Model Apps vs Custom ML 7.2 Text, Language, Search, and Document AI Services 7.3 Vision, Speech, Contact Center, and Personalization Services 7.4 Amazon Quick, Amazon Q Developer, Kiro, and Practitioner Fit 7.5 SageMaker Canvas, Studio, Clarify, Autopilot, and Data Wrangler 7.6 Data Foundation Services: S3, Glue, OpenSearch, and Quick Sight 7.7 AWS AI Service Selection Case Lab

8 Chapter 8: Responsible AI, Human Review, and Safety

8.1 Fairness, Bias, Transparency, and Explainability 8.2 Privacy, Safety, Human Review, and Accountability 8.3 Guardrails, Clarify, Human Review, and Content Safety Controls 8.4 Responsible AI Risk Registers and Governance Workflows 8.5 Monitoring, Feedback, Drift, and Incident Response 8.6 Responsible AI Case Lab

9 Chapter 9: Security, Compliance, Governance, and Cost Controls

9.1 Shared Responsibility, IAM, and Least Privilege for AI 9.2 Encryption, Secrets, Networking, and Data Privacy 9.3 Prompt Injection, Data Exfiltration, and GenAI Threat Modeling 9.4 Logging, Monitoring, CloudTrail, CloudWatch, and Config 9.5 Compliance: Artifact, Config, Macie, and Policy Evidence 9.6 AI Cost Controls, Pricing, Throughput, and Budget Governance 9.7 Security and Governance Case Lab

10 Chapter 10: Integrated AWS AI Business Scenario Labs

10.1 Customer Support GenAI Assistant Lab 10.2 Document Intelligence and Compliance Review Lab 10.3 Personalization, Forecasting, and Fraud Detection Lab 10.4 Enterprise Search, RAG, and Knowledge Management Lab 10.5 Responsible AI and Security Review Board Lab 10.6 Cost, Performance, and Operations Review Lab 10.7 Full AIF-C01 Business Simulation

11 Chapter 11: Final Review, Exam Readiness, and Recertification

11.1 Final 30-Day AIF-C01 Study Plan 11.2 Official Practice Resources and Weak-Domain Remediation 11.3 90-Minute Exam Timing, Flagging, and Guessing Workflow 11.4 Test-Day Checklist: Online or Test Center 11.5 Post-Exam Results, Retake, and Recertification Plan 11.6 AWS AI Practitioner Final Mixed Review

6.5 Bedrock Model Evaluation, Monitoring, and Human Feedback

Key Takeaways Explain This Topic AI

Bedrock evaluation helps compare models, prompts, knowledge bases, and RAG sources using automatic, LLM-as-judge, or human-review methods where supported.

A useful evaluation dataset contains realistic prompts, expected answers or ground truth, edge cases, refusal cases, and business scoring criteria.

Monitoring combines CloudWatch metrics, CloudTrail audit events, invocation logging decisions, application telemetry, and user feedback.

Human feedback is most important when outputs affect customers, policy interpretation, safety, legal risk, money movement, or regulated decisions.

Evaluation is continuous: model behavior, source data, prompts, guardrails, and user needs can drift after launch.

Last updated: May 2026

Evaluation before production

A model demo is not an evaluation. A demo shows that a prompt can produce a good answer once. An evaluation asks whether the model, prompt, retrieval design, and guardrails perform well enough across representative work. Amazon Bedrock evaluations can help assess Bedrock models, knowledge bases, and RAG sources. AWS documentation describes automatic evaluation, LLM-as-judge evaluation, RAG evaluation, and human worker evaluation options where supported. The practitioner goal is to choose the evaluation approach that matches risk.

Evaluation starts with a dataset. For a summarization workflow, the dataset might include transcripts and expected summary traits. For a RAG assistant, it should include user questions, expected retrieved passages, expected answers, and questions that should not be answered. For an agent, the test set should include missing parameters, authorization failures, unsafe requests, and downstream API errors. A dataset built only from happy paths will make almost any model look better than it is.

Evaluation target What to measure Useful evidence

Base model Quality, instruction following, latency, cost, tone, format reliability Same prompt set across candidate models.

Prompt template Consistency, refusal behavior, schema adherence, sensitivity to wording Versioned prompts and regression tests.

Knowledge Base or RAG source Retrieval relevance, answer correctness, citation usefulness, grounding Ground truth passages and expected answers.

Guardrail False positives, false negatives, blocked categories, user experience Safe and adversarial test prompts.

Agent workflow Correct action choice, parameter elicitation, confirmation, error handling Agent traces and business-system logs.

Automatic metrics can scale review, but they do not replace business judgment. LLM-based judging can compare responses or score helpfulness, relevance, or faithfulness, but the judge model is still a model. Human evaluation is slower and more expensive, yet it is valuable when domain experts must judge nuance. A benefits specialist, claims reviewer, technician, nurse, or compliance analyst may notice a flaw that an automated score misses.

RAG evaluation deserves special attention. If the answer is wrong, first ask whether the correct source was retrieved. A generator cannot reliably answer from missing evidence. Bedrock RAG evaluations can help compare retrieve-only behavior and retrieve-and-generate behavior. The results can guide changes to chunking, metadata filters, reranking, source cleanup, and prompt instructions. Do not switch to a larger model before checking retrieval quality.

Monitoring begins before launch. Amazon Bedrock integrates with CloudWatch for metrics and monitoring, CloudTrail for API activity auditing, and model invocation logging to CloudWatch Logs or Amazon S3 when enabled. Invocation logging can include request and response data, so the decision to enable it must consider sensitive information, retention, access, encryption, and incident response. Application teams should also capture business metrics such as deflection rate, user edits, approval rate, escalation rate, and complaint rate.

Monitoring and feedback checklist:

Define success metrics and unacceptable failure modes before pilot launch.

Store prompt, model, retrieval, and guardrail versions with each test result where feasible.

Track latency, token usage, throttling, errors, and user-facing failure rates.

Review samples of accepted, edited, rejected, blocked, and escalated outputs.

Protect invocation logs and avoid collecting sensitive text that the business does not need.

Use CloudTrail to audit who changed model, guardrail, knowledge base, or agent resources.

Schedule reevaluation after model changes, prompt changes, source updates, and policy changes.

Scenario: a legal operations team wants contract clause summaries. Evaluation should include real but sanitized clause types, known gold-standard summaries, examples with missing context, and red-team prompts asking for legal conclusions outside the approved scope. Human reviewers should score accuracy and risk. Monitoring should track how often attorneys edit summaries and which clause types produce disagreement. A high edit rate is not just a UX issue; it may indicate a model or prompt mismatch.

Scenario: a customer support team pilots a RAG assistant. Metrics should include retrieval relevance, citation click-through, answer helpfulness, escalation reduction, average handle time, and complaint categories. If users mark answers as unhelpful, the team should inspect retrieved passages. The problem might be stale articles, poor metadata, conflicting policy pages, or a prompt that over-summarizes. Model replacement is only one possible remedy.

Human feedback loops should be structured. Letting users thumbs-up or thumbs-down can identify broad trends, but expert review should label the reason for failure: incorrect fact, unsupported claim, bad tone, missing citation, privacy concern, policy violation, or action error. Those labels guide whether to adjust data, retrieval, prompt, guardrail, model choice, or workflow design. Without failure categories, feedback becomes noise.

For AWS Skill Builder practice, compare two models or two prompt variants with the same small rubric. Record which answers are correct, which are unsupported, and which are too slow or expensive. The habit to build is repeatability. If a stakeholder asks why one model was chosen, the team should have evidence beyond personal preference.