Study guide
8.5 Monitoring, Feedback, Drift, and Incident Response
AWS AI Practitioner Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: AIF-C01 Orientation and Official Source Control
1.1 Credential Scope, Code, and Search Language 1.2 Exam Format, Cost, Score, and Delivery 1.3 Official Exam Guide, Skill Builder, and Practice Workflow 1.4 Target Candidate Boundaries and Out-of-Scope Tasks 1.5 Domain Weights and Study Prioritization 1.6 Exam Policy, Retake, Results, and Recertification
2 Chapter 2: AI/ML Foundations and Use-Case Fit
2.1 AI, ML, Deep Learning, and Core Terminology 2.2 Supervised, Unsupervised, and Reinforcement Learning 2.3 Data Types, Labels, Structure, and Quality 2.4 Inference Patterns: Batch, Real-Time, and Embedded AI 2.5 AI Use-Case Fit and No-AI Decisions 2.6 Classification, Regression, Clustering, Forecasting, and Recommendation 2.7 AI/ML Foundations Case Lab
3 Chapter 3: ML Lifecycle, Metrics, and Practitioner MLOps
3.1 Data Collection, EDA, Preprocessing, and Feature Concepts 3.2 Training, Evaluation, Deployment, and Monitoring Lifecycle 3.3 Model Sources: Managed APIs, Open Source, and Custom Models 3.4 Evaluation Metrics and Business Metrics 3.5 Practitioner MLOps: Repeatability, Monitoring, and Retraining 3.6 SageMaker Lifecycle Service Map 3.7 ML Lifecycle Case Lab
4 Chapter 4: Generative AI Foundations and Inference Concepts
4.1 Foundation Models, LLMs, Transformers, and Modalities 4.2 Tokens, Context Windows, Embeddings, and Vector Search 4.3 Inference Parameters: Temperature, Top-p, and Output Controls 4.4 Hallucination, Grounding, and Context Quality 4.5 GenAI Use-Case Fit and Risk Triage 4.6 Prompt Injection, Data Leakage, and User Input Risk 4.7 Generative AI Foundations Case Lab
5 Chapter 5: Prompting, Model Selection, Customization, and Evaluation
5.1 Prompt Engineering Patterns and Business Quality 5.2 Zero-Shot, Few-Shot, Templates, and Instruction Design 5.3 Model Selection: Capability, Latency, Cost, and Risk 5.4 RAG vs Fine-Tuning vs Prompting vs Custom Models 5.5 Model Evaluation, Human Review, and Red-Team Feedback 5.6 Cost, Performance, and Throughput Decision-Making 5.7 Prompting and Model Selection Case Lab
6 Chapter 6: Amazon Bedrock, RAG, Agents, and Guardrails
6.1 Bedrock Core Concepts, Model Access, and Managed FM Choice 6.2 Knowledge Bases, RAG, Data Sources, and Grounding 6.3 AI Agents, Tools, Orchestration, and Business Workflows 6.4 Guardrails, Content Filters, Denied Topics, and Sensitive Data 6.5 Bedrock Model Evaluation, Monitoring, and Human Feedback 6.6 Bedrock Cost, Latency, Throughput, and Operational Fit 6.7 Bedrock, RAG, and Agents Case Lab
7 Chapter 7: AWS Managed AI/ML Services and SageMaker Map
7.1 Managed AI Services vs Foundation Model Apps vs Custom ML 7.2 Text, Language, Search, and Document AI Services 7.3 Vision, Speech, Contact Center, and Personalization Services 7.4 Amazon Quick, Amazon Q Developer, Kiro, and Practitioner Fit 7.5 SageMaker Canvas, Studio, Clarify, Autopilot, and Data Wrangler 7.6 Data Foundation Services: S3, Glue, OpenSearch, and Quick Sight 7.7 AWS AI Service Selection Case Lab
8 Chapter 8: Responsible AI, Human Review, and Safety
8.1 Fairness, Bias, Transparency, and Explainability 8.2 Privacy, Safety, Human Review, and Accountability 8.3 Guardrails, Clarify, Human Review, and Content Safety Controls 8.4 Responsible AI Risk Registers and Governance Workflows 8.5 Monitoring, Feedback, Drift, and Incident Response 8.6 Responsible AI Case Lab
9 Chapter 9: Security, Compliance, Governance, and Cost Controls
9.1 Shared Responsibility, IAM, and Least Privilege for AI 9.2 Encryption, Secrets, Networking, and Data Privacy 9.3 Prompt Injection, Data Exfiltration, and GenAI Threat Modeling 9.4 Logging, Monitoring, CloudTrail, CloudWatch, and Config 9.5 Compliance: Artifact, Config, Macie, and Policy Evidence 9.6 AI Cost Controls, Pricing, Throughput, and Budget Governance 9.7 Security and Governance Case Lab
10 Chapter 10: Integrated AWS AI Business Scenario Labs
10.1 Customer Support GenAI Assistant Lab 10.2 Document Intelligence and Compliance Review Lab 10.3 Personalization, Forecasting, and Fraud Detection Lab 10.4 Enterprise Search, RAG, and Knowledge Management Lab 10.5 Responsible AI and Security Review Board Lab 10.6 Cost, Performance, and Operations Review Lab 10.7 Full AIF-C01 Business Simulation
11 Chapter 11: Final Review, Exam Readiness, and Recertification
11.1 Final 30-Day AIF-C01 Study Plan 11.2 Official Practice Resources and Weak-Domain Remediation 11.3 90-Minute Exam Timing, Flagging, and Guessing Workflow 11.4 Test-Day Checklist: Online or Test Center 11.5 Post-Exam Results, Retake, and Recertification Plan 11.6 AWS AI Practitioner Final Mixed Review
8.5 Monitoring, Feedback, Drift, and Incident Response
Key Takeaways Explain This Topic AI
Responsible AI continues after launch through monitoring, feedback analysis, drift detection, guardrail review, cost review, and incident response.
Classic ML systems can drift when live data, labels, business conditions, or user behavior differ from the baseline used for training or evaluation.
Generative AI systems need monitoring for unsafe content, hallucination, prompt attacks, retrieval failures, sensitive data exposure, user feedback, and unexpected tool use.
AWS services such as CloudWatch, CloudTrail, Bedrock logs, open-source SageMaker AI monitoring solutions, and application telemetry can support different monitoring needs (SageMaker Model Monitor and Clarify are no longer open to new customers).
Incident response should define severity, containment, human communications, rollback, evidence preservation, root-cause analysis, and control updates.
Last updated: October 2026
Launch is the start of responsible operation
AI systems can degrade after launch even when they passed initial testing. Customer behavior changes. Product policies change. Source documents become stale. A foundation model version changes. A fraud pattern adapts. Users learn how to phrase prompts that bypass weak controls. A seasonal event shifts data distribution. Responsible AI therefore needs monitoring and incident response, not just a launch checklist.
Monitoring should be tied to the approved risk register. If the main risk is biased predictions, monitor bias and segment-level outcomes. If the main risk is hallucinated policy advice, monitor unsupported answers, citation quality, retrieval misses, and user disputes. If the main risk is PII leakage, monitor sensitive data detections, log access, and prompt patterns. If the main risk is unsafe tool action, monitor action calls, denied attempts, confirmations, and rollback rates.
Signal What it can reveal AWS or workflow source
Data drift Live input distribution differs from baseline Open-source SageMaker AI monitoring solutions, CloudWatch, or custom telemetry
Model quality drift Accuracy or other quality metric changes Ground truth labels, monitoring jobs, business KPIs
- Bias drift Group-level behavior shifts over time Scheduled bias-metric checks
- Feature attribution drift Model explanations change from baseline Scheduled SHAP attribution checks
Guardrail intervention rate Users or outputs trigger safety policies Bedrock Guardrails metrics or application logs where configured
Retrieval quality Sources are missing, stale, or irrelevant Knowledge base evaluation, citations, feedback, search telemetry
Security event Unusual access, prompt attack, or tool misuse CloudTrail, CloudWatch, IAM Access Analyzer, app logs
User feedback Users dispute, correct, or abandon outputs Application feedback, support tickets, reviewer decisions
Amazon SageMaker Model Monitor has monitored data quality and model quality for SageMaker AI models in production, but it is no longer open to new customers; AWS's replacement guidance uses open-source SageMaker AI monitoring solutions, Amazon Quick Sight dashboards, and Amazon CloudWatch. The usual pattern is the same: capture inference data, create a baseline, define a monitoring schedule, inspect reports, and alert on violations. A practitioner does not need to build the code, but should recognize that monitoring needs a baseline and a response plan.
Generative AI monitoring is broader than classic model accuracy. A Bedrock RAG assistant might need telemetry for prompt length, response length, latency, token cost, retrieved document IDs, citation presence, refusal rate, guardrail interventions, user feedback, and escalation outcomes. A Bedrock Agent might need traces for action selection, parameter collection, tool calls, downstream errors, confirmations, and human handoffs. Without this evidence, teams cannot tell whether a bad answer came from the model, prompt, retrieval corpus, permissions, or action design.
Monitoring also has privacy implications. Model invocation logs, prompt traces, reviewer notes, and feedback can contain sensitive information. Teams should decide what to log, how to redact or mask, how long to retain it, and who can access it. It is weak governance to collect every prompt forever without a purpose. It is also weak governance to collect no evidence for a high-risk system. The right balance depends on risk, policy, and audit needs.
Feedback loops should be deliberate. Users need a simple way to mark an answer incorrect, unsafe, incomplete, or helpful. Reviewers should record why they changed an output. Product owners should review trends rather than individual anecdotes only. Feedback should feed into prompt updates, source corrections, guardrail tuning, reviewer training, model evaluation, or feature rollback. Feedback that sits in a dashboard nobody owns is not a control.
AI incident response workflow:
Detect: identify the signal, such as unsafe answer, PII exposure, biased outcome, hallucination, or unauthorized action.
Triage: classify severity based on affected users, sensitivity, harm, recurrence, and legal or customer impact.
Contain: disable a prompt, tighten a guardrail, remove a data source, pause an agent action, or route all outputs to review.
Preserve evidence: save relevant logs, prompts, outputs, source documents, model IDs, guardrail versions, and reviewer actions according to policy.
Communicate: notify owners, support teams, legal or compliance, and affected users where required.
Remediate: fix prompts, data, permissions, guardrails, model choice, review workflow, or business process.
Validate: retest normal, edge, and misuse cases before restoring automation.
Learn: update the risk register, runbook, monitoring thresholds, and training.
Scenario: a RAG assistant starts citing outdated policy after a document migration. Monitoring shows rising user corrections and retrieval from old S3 prefixes. The incident response should remove stale documents, re-ingest the knowledge base, verify metadata filters, notify support teams, and review why source ownership failed. The answer is not simply choosing a larger model.
Scenario: a SageMaker AI fraud model begins flagging a new customer segment at a higher rate after a market campaign. The team should compare current data to the baseline, inspect model quality and bias drift, review whether the campaign changed the input distribution, and route affected decisions to analysts while investigating. Monitoring connects data drift to fairness and business process risk.
Scenario: a Bedrock Agent submits duplicate service requests because users refresh the page during slow responses. The incident is partly AI and partly application design. The fix may require idempotency keys, clearer confirmation, better timeout handling, and trace review. Responsible AI incident response includes downstream systems, not only model prompts.
CloudTrail and CloudWatch are important supporting services. CloudTrail can help answer who invoked or changed AWS resources, depending on the service and configuration. CloudWatch can collect metrics, logs, and alarms for application behavior, SageMaker monitoring jobs, and operational thresholds. AWS Config, AWS Artifact, and other governance services may support broader evidence collection, but monitoring still needs business-specific signals.
For AWS Skill Builder practice, read AI scenarios as operations stories. Ask what baseline exists, what signal detects drift, who receives the alarm, what gets paused, what evidence is kept, and what must be retested. A responsible system is one the organization can operate when behavior changes.