Study guide
11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries
AWS Certified DevOps Engineer – Professional Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CI/CD Pipeline Architecture & Source Integration
1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models
2 Chapter 2: Automated Testing, Test Gates & Quality Verification
2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates
3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies
3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda
4 Chapter 4: CloudFormation, CDK & Reusable IaC Components
4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance
5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower
5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation
6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration
6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig
7 Chapter 7: High Availability & Multi-Region Resiliency
7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR
8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation
8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery
9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation
9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions
10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray
10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps
11 Chapter 11: Automated Observability & Event-Driven Monitoring
11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries
12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines
12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling
13 Chapter 13: Automated Remediation & Deployment Troubleshooting
13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures
14 Chapter 14: Identity, Access Management & Secrets at Scale
14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager
15 Chapter 15: Security Automation, Data Protection & Compliance Auditing
15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config
11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries
Key Takeaways Explain This Topic AI
Amazon CloudWatch Synthetics Canaries proactively monitor customer-facing endpoints and internal microservices 24/7 by executing automated Node.js (Puppeteer) or Python (Selenium) scripts at scheduled intervals as fast as once per minute.
Synthetics provides four out-of-the-box blueprints: Heartbeat Monitoring (endpoint liveness and status), API Canaries (multi-step HTTP requests with JSON payload validation), Broken Link Checkers (spidering web pages), and GUI Workflow Canaries (simulating user browser interactions).
Every canary execution writes detailed diagnostic artifacts to a dedicated Amazon S3 bucket—including step screenshots, HTTP Archive (HAR) files capturing complete network request-response timings, and execution logs—while emitting metrics to the CloudWatchSynthetics namespace.
VPC Canaries monitor private microservices behind internal ALBs or private API Gateways by provisioning Elastic Network Interfaces (ENIs) inside customer subnets, requiring appropriate security group egress and VPC endpoints (or NAT Gateway) to publish metrics to CloudWatch and artifacts to S3.
CloudWatch Synthetics integrates with AWS CodeDeploy and AWS X-Ray: canary alarms serve as automated deployment rollback gates to abort blue/green traffic shifting, while X-Ray trace headers (X-Amzn-Trace-Id) allow end-to-end distributed latency analysis of synthetic transactions.
Last updated: September 2026
Synthetic Monitoring vs. Passive Observability
Traditional infrastructure and application monitoring is inherently passive : metrics, logs, and traces are generated only when real end users interact with the system. While vital, passive observability suffers from three critical blind spots in enterprise environments:
Low-Traffic Masking : During off-peak hours, weekends, or scheduled maintenance windows, user traffic drops significantly. A severe infrastructure outage (such as an expired TLS certificate, DNS misconfiguration, or broken database connection) may generate zero alerts simply because no users are attempting to connect.
The 200 OK False Positive Trap : An application frontend may successfully return HTTP 200 OK with an empty HTML body, or fail to render because a client-side JavaScript bundle failed to load from a CDN. Server-side metrics show a 100% success rate, while 100% of human users experience a broken application.
Post-Deployment Verification Latency : During blue/green or canary deployments, relying on real user error rates to detect bugs forces customers to act as involuntary software testers.
Amazon CloudWatch Synthetics solves these dilemmas through proactive, continuous testing. Synthetics canaries are modular, automated scripts that run on a continuous schedule (e.g., every 1, 5, or 15 minutes, 24 hours a day, 7 days a week). Canaries simulate the exact actions of a real user—clicking buttons, navigating checkout flows, filling forms, and validating API responses—to verify availability, latency, and functional correctness before end users are impacted.
Canary Blueprints & Runtime Engines
Under the hood, CloudWatch Synthetics packages canary scripts as specialized AWS Lambda functions running in a managed runtime environment. Synthetics supports two primary headless browser and automation runtimes:
Node.js : Runs Chromium via Puppeteer (managed runtime syn-nodejs-puppeteer-* ).
Python : Runs Chromium via Selenium WebDriver (managed runtime syn-python-selenium-* ).
Out-of-the-Box Canary Blueprints
For standard monitoring tasks, AWS provides pre-built Canary Blueprints that require minimal configuration:
Blueprint Name Operational Scope Ideal Use Case
Heartbeat Monitor Probes a single URL; verifies HTTP status code, page load latency, and captures a screenshot of the loaded DOM. Ingress liveness monitoring, public website landing page checks, TLS certificate validity tracking.
API Canary Executes single or multi-step HTTP requests ( GET , POST , PUT , DELETE ) against REST or GraphQL endpoints; validates headers, status codes, and JSON response body schemas. Microservice API contract testing, authentication token verification, payment gateway integration testing.
Broken Link Checker Spiders an HTML page, extracting and probing every internal and external anchor link ( href ) up to a specified link count or crawl depth. Content management systems, documentation portals, regulatory disclosure link verification.
GUI Workflow Canary Executes a multi-step user journey script (e.g., login -> search catalog -> add to cart -> proceed to checkout) with user-defined clicks, form inputs, and DOM assertions. Critical business transaction flows, customer portal login paths, SaaS application journeys.
Custom Script Anatomy: synthetics.executeStep()
For advanced multi-step workflows, engineers author custom scripts using the Synthetics SDK. The fundamental building block is synthetics.executeStep() , which segments a complex journey into distinct, individually timed phases: const synthetics = require('Synthetics');
const log = require('SyntheticsLogger');
- const checkoutFlowCanary = async function () {
- const page = await synthetics.getPage();
- // Step 1: Navigate to Home and Verify Title
- await synthetics.executeStep('NavigateToHome', async function () {
- await page.goto('https://shop.example.com', { waitUntil: ['domcontentloaded', 'networkidle0'] });
- await synthetics.takeScreenshot('home_page_loaded', 'loaded');
- const title = await page.title();
- if (!title.includes('Online Store')) {
- throw new Error(`Unexpected page title: ${title}`);
- }
- });
- // Step 2: Search for Item
- await synthetics.executeStep('SearchCatalog', async function () {
- await page.type('#search-input', 'wireless keyboard');
- await Promise.all([
- page.waitForNavigation({ waitUntil: 'networkidle0' }),
- page.click('#search-button')
- ]);
- await synthetics.takeScreenshot('search_results', 'displayed');
- });
- // Step 3: API Inventory Check
- await synthetics.executeStep('VerifyInventoryAPI', async function () {
- const response = await synthetics.executeHttp({
- hostname: 'api.example.com',
- path: '/v1/inventory/item-1234',
- method: 'GET',
- headers: { 'Authorization': 'Bearer <token>' }
- });
- if (response.statusCode !== 200) {
- throw new Error(`Inventory API returned HTTP ${response.statusCode}`);
- }
- });
- };
- exports.handler = async () => {
- return await checkoutFlowCanary();
- };
- Canary Artifacts, S3 Lifecycle & CloudWatch Metrics
Every time a canary executes, CloudWatch Synthetics collects a rich set of diagnostic artifacts and stores them in a designated Amazon S3 bucket under the prefix canary/<region>/<canary-name>/<timestamp>/ .
Diagnostic Artifact Types
Step Screenshots : High-resolution PNG images captured at each step (or automatically captured on failure). These allow engineers to visually inspect UI layout failures, modal dialog traps, or unexpected error banners.
HTTP Archive (HAR) Files : A complete, detailed record of every HTTP/HTTPS network transaction executed by the browser during the run. HAR files capture DNS lookup duration, TCP connection time, SSL/TLS handshake latency, Time to First Byte (TTFB), content download time, HTTP request/response headers, and response sizes for every image, script, and API call.
Execution and Console Logs : Captures all browser console warnings, JavaScript errors, network failures, and explicit logs emitted via SyntheticsLogger .
Caution
S3 Cost Explosion Watchout : A single GUI canary running once per minute generates 1,440 executions per day, producing thousands of PNG screenshots and HAR files daily. Without automated lifecycle management, S3 storage costs escalate rapidly. Always configure an Amazon S3 Lifecycle Rule on the canary artifact bucket to transition objects to S3 Standard-IA or expire/delete artifacts after 30 to 90 days .
CloudWatch Synthetics Metrics
Canary execution results are published to the CloudWatchSynthetics namespace with the dimension CanaryName :
SuccessPercent : Percentage of canary runs that succeeded (100 or 0 for a single run; average over an evaluation window).
Duration : Total execution time in milliseconds.
Failed : Count of failed canary runs.
2xx , 4xx , 5xx : Count of HTTP status codes returned by endpoints probed during the run.
VPC Canaries: Monitoring Private Internal Microservices
By default, CloudWatch Synthetics canaries run in a managed VPC operated by AWS and can only probe publicly accessible internet endpoints. However, in enterprise microservice architectures, core internal APIs and administration portals reside in private VPC subnets behind internal Application Load Balancers or private API Gateways with no internet ingress.
VPC Canary Architecture & Networking Prerequisites
To probe private endpoints, you configure the canary as a VPC Canary :
The canary is configured with a VPC ID, two or more private Subnet IDs, and one or more Security Group IDs.
The underlying Lambda execution platform provisions Elastic Network Interfaces (ENIs) directly inside the designated private subnets.
- The canary executes its requests from an IP address within the private subnet IP space. Private Subnet (AZ-a)
- ┌─────────────────────────────────────────────────────────┐
- │ Canary ENI ──(HTTP :8080)──> Internal Microservice ALB │
- │ │ │
- │ ├─(HTTPS :443)──> S3 Gateway VPC Endpoint (Artifacts)│
- │ │ │
- │ └─(HTTPS :443)──> CloudWatch VPC Endpoints │
- │ - com.amazonaws.<reg>.monitoring │
- │ - com.amazonaws.<reg>.logs │
- └─────────────────────────────────────────────────────────┘
- The VPC Endpoint vs. NAT Gateway Requirement
A frequent DOP-C02 exam scenario involves a VPC canary that successfully probes the private internal microservice, but the canary run itself fails with a timeout or error status.
Root Cause : When running inside a customer VPC, the canary is bound to the VPC's routing rules. In addition to reaching the target microservice, the canary must communicate with AWS public service endpoints to:
- Write execution logs to Amazon CloudWatch Logs
- Publish metric data points to Amazon CloudWatch
- Upload screenshots and HAR files to Amazon S3
If the private subnet lacks internet egress and lacks VPC endpoints, the canary times out attempting to upload its artifacts and fails.
Remediation Options :
Option A (Air-Gapped / PrivateLink) : Provision a Gateway VPC Endpoint for Amazon S3 (free of charge) and Interface VPC Endpoints (AWS PrivateLink) for CloudWatch Monitoring ( monitoring.<region>.amazonaws.com ) and CloudWatch Logs ( logs.<region>.amazonaws.com ). Ensure the canary's security group allows outbound HTTPS (port 443) to these endpoints.
Option B (NAT Egress) : Ensure the private subnets have a valid route ( 0.0.0.0/0 ) targeting an Amazon NAT Gateway located in a public subnet with an attached Internet Gateway.
Automated Deployment Guardrails & CodeDeploy Integration
In modern CI/CD pipelines, CloudWatch Synthetics canaries serve as automated deployment safety gates that trigger automated rollbacks during canary or blue/green traffic shifting.
CodeDeploy Traffic Shifting & Rollback Triggers
When deploying application updates using AWS CodeDeploy (for ECS, Lambda, or EC2/On-Premises), traffic shifting is executed progressively:
CodeDeployDefault.LambdaCanary10Percent5Minutes (shifts 10% of traffic, waits 5 minutes, then shifts remaining 90%).
CodeDeployDefault.LambdaLinear10PercentEvery1Minute (shifts 10% every minute over 10 minutes).
CodeDeployDefault.ECSLinear10PercentEvery3Minutes .
To automate rollback on failure:
Configure a CloudWatch Synthetics canary to run every 1 minute against the application endpoint.
Create a CloudWatch Alarm on the canary's SuccessPercent metric ( SuccessPercent < 100 for 1 evaluation period of 1 minute) or Failed >= 1 .
In the AWS CodeDeploy Deployment Group configuration, navigate to Deployment Alarms and attach the CloudWatch Synthetics alarm, enabling Roll back when alarms are triggered .
During traffic shifting (when 10% of user traffic is hitting the new replacement version), if the synthetic canary encounters an HTTP 5xx error, broken DOM selector, or breached latency threshold, the alarm enters ALARM state.
CodeDeploy immediately aborts the deployment, terminates traffic shifting, reverts 100% of traffic back to the original stable version, and flags the deployment as Failed .
CodeDeploy Lifecycle Hooks with Synthetics
For pre-traffic verification, AWS CodeDeploy supports lifecycle event hooks:
Lambda Deployments : BeforeAllowTraffic hook invokes a validation Lambda function that triggers a one-off Synthetics canary execution against the newly deployed target revision before any production traffic shifts.
ECS Blue/Green Deployments : The replacement task set is registered to a test listener port (e.g., port 8443) on the ALB. The canary executes its validation test against port 8443 during the AfterAllowTestTraffic hook. If the canary passes, traffic shifts to port 443; if it fails, CodeDeploy triggers an automated rollback before production users ever touch the new build.
Distributed Tracing Integration with AWS X-Ray
When a synthetic canary fails or detects elevated latency, pinpointing the root cause across a distributed microservice architecture can be difficult. CloudWatch Synthetics natively integrates with AWS X-Ray :
Active Tracing Configuration : Set ActiveTracing: true in the canary configuration.
Trace Propagation : When active tracing is enabled, the Synthetics runtime automatically injects the X-Amzn-Trace-Id HTTP header into every synthetic HTTP request it originates.
Downstream Trace Correlation : As the synthetic request traverses the architecture—passing through an Application Load Balancer, Amazon API Gateway, ECS containers, AWS Lambda functions, and downstream DynamoDB or Aurora databases—each instrumented service propagates the trace ID.
Service Map Visualization : DevOps engineers open the AWS X-Ray console or CloudWatch ServiceLens to view the complete synthetic transaction call tree. If the canary's Duration metric spikes from 200 ms to 4,000 ms, the X-Ray service map visually highlights the exact downstream microservice or SQL query causing the latency bottleneck.
DOP-C02 Exam Watchouts & Troubleshooting
Scenario / Symptom Root Cause Solution
VPC Canary targeting an internal ALB fails with TimeoutError: Navigation timeout of 30000 ms exceeded The private subnet has no route to S3 or CloudWatch endpoints, so the canary cannot upload artifacts or publish metrics Add an S3 Gateway Endpoint and CloudWatch Monitoring/Logs Interface VPC Endpoints, or route outbound traffic through a NAT Gateway.
Canary fails on step execution with NodeNotFound: Element #checkout-btn not found The frontend application dynamically renders DOM elements via React/Vue, and the button had not finished rendering Use page.waitForSelector('#checkout-btn', { visible: true, timeout: 5000 }) before invoking page.click() .
CodeDeploy deployment does not roll back despite synthetic canary reporting SuccessPercent = 0 The CloudWatch Alarm was configured with TreatMissingData=ignore and an evaluation period of 15 minutes, delaying alarm state transition Set evaluation period to 1 minute, datapoints to alarm 1 of 1 , and ensure the alarm is registered in the CodeDeploy Deployment Group alarm configuration.
S3 artifact bucket storage costs spike unexpectedly after deploying multiple canaries Canaries running at 1-minute intervals generate millions of screenshot and HAR objects that are retained indefinitely Create an Amazon S3 Lifecycle rule on the artifact bucket prefix to transition objects to S3 Glacier or expire them after 30–60 days.
Canary execution passes, but X-Ray traces do not show downstream microservice segments Downstream microservices do not propagate the incoming X-Amzn-Trace-Id HTTP header or lack the AWS X-Ray daemon / OpenTelemetry collector Ensure downstream services inspect and forward the X-Amzn-Trace-Id header, and ensure ECS tasks have the AWS X-Ray daemon sidecar container running.