Study guide
2.2 Load Testing, Performance Benchmarking & Synthetic Canaries
AWS Certified DevOps Engineer – Professional Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CI/CD Pipeline Architecture & Source Integration
1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models
2 Chapter 2: Automated Testing, Test Gates & Quality Verification
2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates
3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies
3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda
4 Chapter 4: CloudFormation, CDK & Reusable IaC Components
4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance
5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower
5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation
6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration
6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig
7 Chapter 7: High Availability & Multi-Region Resiliency
7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR
8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation
8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery
9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation
9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions
10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray
10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps
11 Chapter 11: Automated Observability & Event-Driven Monitoring
11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries
12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines
12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling
13 Chapter 13: Automated Remediation & Deployment Troubleshooting
13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures
14 Chapter 14: Identity, Access Management & Secrets at Scale
14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager
15 Chapter 15: Security Automation, Data Protection & Compliance Auditing
15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config
2.2 Load Testing, Performance Benchmarking & Synthetic Canaries
Key Takeaways Explain This Topic AI
The Distributed Load Testing on AWS solution uses Amazon ECS on AWS Fargate to distribute tests involving tens of thousands of concurrent users across Regions, supporting JMeter, k6, Locust, and simple HTTP scenarios without managing EC2 hosts.
Performance benchmarking in staging environments must enforce quantifiable latency budgets (e.g., p95 and p99 percentiles) and error rate SLOs, failing pipeline stages when budgets are breached.
Amazon CloudWatch Synthetics canaries run automated scripts using Node.js (Puppeteer) or Python (Selenium) on AWS Lambda to monitor endpoints, APIs, and complex multi-step browser user workflows.
Synthetics canaries deployed into private VPC subnets require VPC endpoints (for S3, CloudWatch Logs, and CloudWatch) or NAT Gateways to upload screenshots, HAR files, and metrics.
CodeDeploy deployment groups integrate with CloudWatch alarms tied to Synthetics canaries to automatically halt canary/linear traffic shifting and execute instant rollbacks when degradation occurs.
Last updated: September 2026
Functional testing verifies whether software produces correct outputs for given inputs; non-functional testing verifies how the system behaves under operational stress. On the AWS DOP-C02 exam, performance testing and synthetic monitoring are tested as automated quality gates that guard production availability and latency Service Level Objectives (SLOs).
DevOps engineers must understand how to generate large-scale distributed load using managed container architectures, how to evaluate latency percentiles against strict performance budgets in staging, and how to use synthetic canaries as automated circuit breakers during progressive rollouts.
Distributed Load Testing on AWS Architecture
A single test client running JMeter or Locust on an EC2 instance or local developer machine quickly runs into CPU, memory, or network socket exhaustion (ephemeral port limits) when simulating tens of thousands of concurrent users. To solve this, AWS provides the Distributed Load Testing on AWS implementation—an open-source AWS Solutions implementation. [API Gateway / Web UI]
- │
- ▼
- [AWS Step Functions]
- │
- ┌──────────────┴──────────────┐
- ▼ ▼
- [Amazon DynamoDB] [Amazon S3 Bucket]
- (Test State & Config) (Scripts & Artifacts)
- │ │
- └──────────────┬──────────────┘
- ▼
- [Amazon ECS on AWS Fargate]
- ┌─────────────────────────┐
- │ Fargate Task (Worker 1) │ ──┐
- │ Fargate Task (Worker 2) │ ──┼──> [Target Application]
- │ Fargate Task (Worker N) │ ──┘
- └─────────────────────────┘
- │
- ▼
- [Amazon CloudWatch Metrics]
- (Real-Time Concurrency/Latency)
- Architectural Components
Orchestration Layer: AWS Step Functions orchestrates the end-to-end load test lifecycle: validating test parameters, launching containers, monitoring progress, and aggregating final test reports.
Execution Layer: Amazon ECS on AWS Fargate executes containerized test workers. The solution can deploy regional stacks and simulate tens of thousands of concurrent users across multiple Regions, but capacity depends on the test script, per-task CPU and memory, Fargate quotas, and calibration. It does not promise a fixed request rate or task count.
Supported Test Types:
Apache JMeter: Upload JMX test plans for complex HTTP/S scenarios.
Grafana k6: Upload JavaScript k6 test scripts.
Locust: Upload Python test scripts with dynamic user behavior.
Single HTTP Endpoint: Configure a simple endpoint test without supplying a script.
The solution runs these test types through the Taurus automation framework; Taurus is the orchestration layer rather than a fourth user-authored test type.
Storage & Telemetry: Test scenarios and raw results are stored in Amazon S3 ; test execution status is tracked in Amazon DynamoDB ; real-time response times, throughput, and HTTP status codes are pushed to Amazon CloudWatch Metrics .
Performance Benchmarking & Latency Budgets in Staging
To prevent performance regressions from reaching production, teams integrate automated performance benchmarking into staging pipeline stages. Staging environments must maintain architectural parity with production (e.g., equivalent database provisioned IOPS, similar caching tiers, and identical container resource allocations).
Why Averages Mask Catastrophic Failures: The Percentile Rule
Exam scenarios consistently emphasize evaluating latency percentiles rather than arithmetic averages:
Average Latency (Mean): Masks high-latency tail events. If 99 requests take 10ms and 1 request takes 10,000ms, the average is ~109ms. The system appears healthy while 1 in 100 users experiences complete failure.
P90 / P95 Latency: Measures the experience of the 90th or 95th percentile of users, highlighting micro-stalls and thread starvation.
P99 / P99.9 Latency (Tail Latency): Crucial for microservice architectures. In an e-commerce checkout page that issues 50 statistically independent backend calls, a one-percent per-call tail event appears at least once in about 39.5% of page requests: 1 - 0.99^50 . Real dependencies can also be correlated, so teams measure the end-to-end distribution instead of inferring it from an average. Tail latency directly drives user-perceived abandonment.
Automated Latency Gating in Pipelines
An automated performance gate script running inside AWS CodeBuild or AWS Step Functions queries CloudWatch or parses Locust/JMeter output against defined latency budgets : #!/usr/bin/env bash
set -euo pipefail
echo "Evaluating Staging Performance Benchmarks against SLOs..."
- # Extract latency metrics from test execution summary (in milliseconds)
- P95_LATENCY=$(jq '.metrics.p95' load-test-summary.json)
- P99_LATENCY=$(jq '.metrics.p99' load-test-summary.json)
- ERROR_RATE=$(jq '.metrics.error_rate' load-test-summary.json)
- MAX_ALLOWED_P95=150 # 150ms
- MAX_ALLOWED_P99=300 # 300ms
- MAX_ERROR_RATE=0.001 # 0.1%
- if (( $(echo "$P99_LATENCY > $MAX_ALLOWED_P99" | bc -l) )); then
- echo "ERROR: P99 latency ($P99_LATENCY ms) exceeded budget ($MAX_ALLOWED_P99 ms)!" >&2
- exit 1
- fi
- if (( $(echo "$ERROR_RATE > $MAX_ERROR_RATE" | bc -l) )); then
- echo "ERROR: Error rate ($ERROR_RATE) exceeded allowable threshold ($MAX_ERROR_RATE)!" >&2
- exit 1
- fi
- echo "All performance benchmarks satisfied. Proceeding to production promotion."
- exit 0
If any threshold is violated, the script exits with status 1, failing the CodeBuild action and blocking progression to production.
Amazon CloudWatch Synthetics Canaries
While load testing evaluates capacity under synthetic stress, synthetic monitoring continuously verifies application availability and user experience from the outside in. Amazon CloudWatch Synthetics allows you to create canaries —configurable, automated scripts that run on a schedule (e.g., every 1 or 5 minutes) to monitor endpoints and APIs.
Runtime Environments & Blueprints
Canaries run inside AWS Lambda micro-runtimes using headless browser frameworks:
Node.js (Puppeteer): Headless Chromium automation for JavaScript-heavy web applications.
Python (Selenium Webdriver): Python automation for web and API validation.
CloudWatch Synthetics provides built-in blueprints:
Heartbeat Monitoring: Performs periodic HTTP/S GET or POST calls to an endpoint, tracking status code, response time, and DNS latency.
API Canaries: Executes multi-step RESTful API sequences, sending JSON payloads, authenticating via OAuth/Cognito tokens, and validating response schema and headers.
GUI / Workflow Canaries: Fully automates end-user web interactions—clicking buttons, filling text fields, navigating checkouts, waiting for DOM elements, capturing user-perceived performance metrics, and taking full-page screenshots.
Canary Artifacts & Storage
Every canary run generates rich diagnostic artifacts stored in a dedicated Amazon S3 bucket and CloudWatch Logs :
Screenshots: PNG captures of the browser at designated interaction steps or upon failure.
HAR (HTTP Archive) Files: Complete network logs detailing every asset loaded, HTTP request/response headers, and timing breakdowns.
Execution Logs: Console outputs and stack traces from the Puppeteer/Selenium script.
Critical Architecture: Canaries Inside a VPC
By default, canaries run in an AWS-managed VPC with internet access. However, testing private staging environments, internal Application Load Balancers, or private API Gateways requires configuring the canary to run inside your private VPC subnets .
Caution
The In-VPC Canary Network Trap: When a canary is attached to a private VPC subnet, it uses an Elastic Network Interface (ENI). It can no longer access public AWS APIs by default. The canary must write screenshots and HAR logs to Amazon S3 and write telemetry to CloudWatch Logs. If your VPC lacks an Amazon S3 Gateway VPC Endpoint and a CloudWatch Logs Interface Endpoint (or NAT Gateway routing), the canary will fail with connection timeout errors, even if your application endpoint responded perfectly!
Integrating Canaries as CodeDeploy Deployment Gates
One of the most heavily tested patterns on the DOP-C02 exam is using CloudWatch Synthetics canaries as automated circuit breakers during AWS CodeDeploy progressive deployments.
Traffic Shifting Modes in CodeDeploy
When deploying to AWS Lambda, Amazon ECS, or EC2, CodeDeploy supports progressive traffic shifting:
Canary Deployments: A small percentage of traffic is routed to the new version for a specified time window, followed by a complete shift if healthy (e.g., Canary10Percent5Minutes or Canary10Percent15Minutes ).
Linear Deployments: Traffic shifts in equal increments over time until 100% is reached (e.g., Linear10PercentEvery1Minute or Linear10PercentEvery3Minutes ).
Automated Rollback Integration Flow
Targeted Testing via Hooks: During deployment, CodeDeploy executes lifecycle event hooks. For Lambda and ECS, the AfterAllowTestTraffic hook allows validation scripts or test traffic to hit the replacement revision before any production user traffic is routed.
Canary Monitoring During Traffic Shifting: As live production traffic shifts (e.g., 10% on the new revision), the CloudWatch Synthetics canary continuously executes synthetic transactions against the service.
Alarm Evaluation: A CloudWatch Alarm monitors the canary metrics published to the CloudWatchSynthetics namespace under the CanaryName dimension (for example, Failed >= 1 or SuccessPercent < 100 ).
Instant Rollback: The CodeDeploy Deployment Group is configured with alarmConfiguration . If the canary alarm enters ALARM state at any point during the traffic shifting window:
CodeDeploy immediately cancels the deployment.
100% of traffic is instantly redirected back to the original healthy revision.
Replacement tasks or instances are terminated.
CodePipeline marks the deployment stage as Failed .
Comparison: Testing & Monitoring Paradigms
Attribute Distributed Load Testing CloudWatch Synthetics (Canaries) Real User Monitoring (CloudWatch RUM)
Primary Objective Stress testing & capacity limits Continuous functional & latency validation Actual user telemetry & client-side errors
Execution Model Scheduled or on-demand batch runs Continuous recurring schedule (1–60 min) Continuous passive telemetry from real browsers
Compute Engine Amazon ECS on AWS Fargate AWS Lambda (Puppeteer / Selenium) Client-side JavaScript snippet in web app
Traffic Type Simulated massive concurrency Simulated single-stream synthetic user Genuine live end-user traffic
Deployment Gating Pre-production performance gate (Build/Test stage) Deployment circuit breaker & production rollback trigger Post-deployment SLA tracking & alert triggering
Exam Watchouts & Operational Pitfalls
Important
Canary Artifact S3 Bucket Permissions: The IAM execution role assigned to a CloudWatch Synthetics canary must possess explicit permissions to write to the designated artifact S3 bucket ( s3:PutObject , s3:GetBucketLocation ). If custom KMS encryption is enabled on the bucket, the role must also have kms:GenerateDataKey and kms:Decrypt .
Warning
Pre-Existing Alarms in CodeDeploy: If a CloudWatch alarm attached to a CodeDeploy deployment group is already in the ALARM state when the deployment begins, CodeDeploy will either fail the deployment immediately upon starting or ignore the alarm depending on configuration. Always ensure alarms are in OK status prior to initiating progressive deployments.
Note
Canary Test Data Idempotency: Synthetic canaries executing against staging or production systems must be strictly idempotent. If your canary simulates an e-commerce order checkout, it must either use dedicated synthetic test accounts with dummy payment gateways or automatically invoke a cancellation API to prevent phantom inventory reservations.