Study guide
11.5 CloudWatch Metrics, Throttling & Operational Observability
AWS Certified Generative AI Developer - Professional AIP-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & Foundation Model Fundamentals
1.1 AIP-C01 Exam Structure, Logistics & Passing Strategy 1.2 AIP-C01 Scoring & Professional Decision Strategy 1.3 Foundation Model Landscape & Model Selection on Amazon Bedrock 1.4 Model Abstraction, Controlled Switching & Resilience 1.5 Customization Spectrum: Prompting vs RAG vs Fine-Tuning vs Continued Pre-training 1.6 Custom Model Deployment & Lifecycle Management
2 Chapter 2: Bedrock Knowledge Bases & RAG Architectures
2.1 Managed RAG Architecture with Bedrock Knowledge Bases 2.2 Knowledge Source Integration, Synchronization & Freshness 2.3 Data Ingestion, Parsing & Chunking Strategies 2.4 Embedding Models & Vector Representations 2.5 Vector Index Scale, Migration & Retrieval Evaluation 2.6 Vector Database Selection: OpenSearch Serverless vs Aurora pgvector vs Managed Stores 2.7 Vector Store Security, Multi-Tenancy & Connectivity
3 Chapter 3: Advanced Vector Search, Indexing & Data Compliance
3.1 Hybrid Search, Metadata Filtering & Semantic Reranking 3.2 Metadata Filtering, Reranking & Query Transformation 3.3 Fine-Tuning & Continued Pre-Training Data Pipeline Engineering 3.4 Data Validation, Multimodal Processing & Model Formatting 3.5 Data Privacy, Data Residency & Compliance Governance 3.6 Privacy Preservation, Source Lineage & Audit Evidence
4 Chapter 4: Prompt Engineering & Bedrock Runtime APIs
4.1 Advanced Prompt Engineering Techniques & Inference Parameters 4.2 Inference Parameters, Prompt Governance & Regression 4.3 Converse API vs InvokeModel API & Message Formatting 4.4 Tool Calling, FM API Resilience & Model Routing 4.5 Streaming Architectures & Asynchronous Event Processing 4.6 Resilient Streaming, Deployment & Asynchronous Processing
5 Chapter 5: Amazon Bedrock Agents & Tool Integration
5.1 Agentic Architecture, ReAct Reasoning Loops & Orchestration 5.2 Agent Frameworks, MCP & Lifecycle Management 5.3 Action Groups, OpenAPI Schemas & AWS Lambda Integration 5.4 Agent Memory, Session State & Multi-Agent Collaboration Patterns 5.5 Persistent State & Multi-Agent Coordination 5.6 Advanced Agent Prompt Templates & Flow Customization 5.7 Agent Safeguards, Tool Reliability & Cost Control
6 Chapter 6: Workflow Orchestration & Application Integration
6.1 Amazon Bedrock Prompt Flows & Visual Orchestration 6.2 Step Functions Orchestration & Human-in-the-Loop Workflows 6.3 Enterprise Integration: API Gateway, AppSync, and EventBridge 6.4 Hybrid Integration, GenAI Gateways & CI/CD Tooling
7 Chapter 7: AI Safety & Guardrails for Amazon Bedrock
7.1 Guardrails Architecture, Denied Topics & Content Filters 7.2 Content Safety, Defense in Depth & Threat Detection 7.3 Sensitive Information Filters & PII Redaction/Masking 7.4 Contextual Grounding Checks & Hallucination Prevention
8 Chapter 8: Security, Access Control & Network Architecture
8.1 IAM Least Privilege & Resource Policies for Bedrock 8.2 PrivateLink VPC Endpoints & Network Isolation 8.3 Encryption with AWS KMS & Cross-Account Access Patterns
9 Chapter 9: Governance, Auditing & Operational Logging
9.1 SageMaker Model Cards & AI Governance Frameworks 9.2 Lineage, Fairness, Transparency & Continuous Governance 9.3 Invocation Logging, CloudTrail Auditing & Observability 9.4 Responsible AI, Bias Mitigation & Red Teaming Methodologies
10 Chapter 10: Performance, Throughput & Cost Optimization
10.1 On-Demand vs Provisioned Throughput & Capacity Planning 10.2 Capacity Measurement, Scaling Patterns & Resilience 10.3 Batch Inference & Bedrock Intelligent Prompt Routing 10.4 Intelligent Prompt Routing & Application Eligibility 10.5 Prompt Caching, Token Optimization & Latency Reduction
11 Chapter 11: Testing, Evaluation & Troubleshooting GenAI Solutions
11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows) 11.2 Human, RAG & Continuous Evaluation 11.3 Evaluation Metrics & RAG Quality Benchmarking (Ragas, ROUGE, BLEU) 11.4 RAG and Agent Performance Evaluation 11.5 CloudWatch Metrics, Throttling & Operational Observability 11.6 Quota Diagnosis, Retry Control & Operational Troubleshooting 11.7 Troubleshooting Common GenAI Failure Modes & Agent Traces 11.8 Prompt Maintenance, Content Handling & Deployment Validation
11.5 CloudWatch Metrics, Throttling & Operational Observability
Key Takeaways Explain This Topic AI
Use InvocationThrottles rather than the nonexistent InvocationsThrottled metric.
Do not rely on a nonexistent universal ModelInvocationLatency metric to subtract internal processing stages.
Quota accounting varies by model and endpoint, so diagnose the exact target before attributing a 429 to RPM or tokens.
Last updated: September 2026
11.5 CloudWatch Metrics, Throttling & Operational Observability
Amazon Bedrock runtime publishes CloudWatch metrics under the AWS/Bedrock namespace. Use the exact current metric names. Core metrics include Invocations, InvocationLatency, InvocationClientErrors, InvocationServerErrors, InvocationThrottles, InputTokenCount, and OutputTokenCount for supported operations. InvocationsThrottled and ModelInvocationLatency are not the documented runtime metric names and should not appear in alarms or exam answers.
Read latency with token context
InvocationLatency measures from request submission until the last token is received. A longer response naturally raises that metric. Diagnose latency together with input tokens, output tokens, output tokens per second, request mode, model target, and percentile. Bedrock documentation describes diagnosing latency changes through output-token rate; it does not expose a universal CloudWatch metric that cleanly subtracts raw transformer compute from Guardrail or serialization time.
Symptom Evidence to inspect Likely action
Higher latency and more output tokens OutputTokenCount, requested maximum, response length Tighten output instructions or limits; stream when appropriate
429 responses InvocationThrottles, target model, request and token rates Bound concurrency, back off with jitter, inspect exact quotas
4xx increase InvocationClientErrors and application logs Validate IAM, request schema, model ID, token limits, and Guardrail configuration
5xx or capacity errors InvocationServerErrors, response codes, request IDs Retry eligible transient errors and open support evidence if sustained
Cost increase without traffic growth Input/output/cache usage and prompt versions Find prompt bloat, loops, cache misses, or model changes
CloudWatch aggregate metrics do not contain the full cause. Correlate them with structured application logs, X-Ray segments around application services, model invocation logging when permitted, and request IDs. Do not place raw sensitive prompts into ordinary logs by default.
Metric-to-alarm examples
Build a throttle-rate metric from InvocationThrottles divided by attempted eligible calls using service and application telemetry, then alarm on a sustained rate tied to the service objective. InvocationThrottles alone counts what Bedrock reports but does not describe user impact after retries. Track retry success and final failures separately.
For latency, chart p50, p95, and p99 InvocationLatency with OutputTokenCount and operation. Streaming requests measure through the last token, so add application time-to-first-token when user perception matters. An output-token increase after a prompt release can explain both latency and cost without any service degradation.
Client errors should be grouped by application error code such as access denied, validation, unsupported model, context limit, or Guardrail configuration. Server and transient-capacity errors need request IDs, UTC timestamps, target identifiers, and retry outcomes for support escalation. Avoid high-cardinality dimensions in CloudWatch metrics; store detailed IDs in controlled logs and correlate them.
Annotate dashboards with model, prompt, Guardrail, deployment, and quota changes. Without change markers, operators can mistake an intentional longer answer for an infrastructure incident.
Logging-delivery metrics and content controls
Model invocation logging delivery has its own success and failure telemetry. Alarm on failed delivery when the organization relies on those logs for audit or troubleshooting. Verify the destination policy, KMS permissions, and service configuration; an inference success does not prove its optional payload log was delivered.
Invocation logging can contain prompts, responses, images, or other sensitive material depending on configuration. Enable only the modalities and destinations the policy permits, encrypt them, restrict readers, and apply retention. CloudTrail records management and supported data events, while model invocation logging captures configured inference payloads; neither should be described as a substitute for the other.
Use metric filters cautiously on payload logs. Natural language is high-cardinality and can contain personal data. Prefer structured application outcome fields and sampled controlled review for quality metrics.
Business and model telemetry
Add task completion, validated-output rate, retrieval success, tool success, fallback use, human escalation, and user correction beside service metrics. Correlate by release and model target without exposing sensitive payloads. This reveals cases where infrastructure remains healthy while model behavior deteriorates.
Use sampled quality evaluation rather than attempting to infer hallucination directly from InvocationLatency or token counts. Operational metrics can identify when and where to investigate; they do not by themselves judge truth or responsibility.
Alarm ownership
Every alarm needs an owner, runbook, severity, and expected action. Test notifications and permissions. An unactionable alarm that fires on normal retries trains operators to ignore the signal, while a missing-data state can conceal disabled telemetry. Review thresholds after load or model changes and record why they still represent the service objective.