Study guide
11.1 Application & Infrastructure Health Checks with Route 53 & ALB
AWS Certified DevOps Engineer – Professional Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CI/CD Pipeline Architecture & Source Integration
1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models
2 Chapter 2: Automated Testing, Test Gates & Quality Verification
2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates
3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies
3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda
4 Chapter 4: CloudFormation, CDK & Reusable IaC Components
4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance
5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower
5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation
6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration
6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig
7 Chapter 7: High Availability & Multi-Region Resiliency
7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR
8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation
8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery
9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation
9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions
10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray
10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps
11 Chapter 11: Automated Observability & Event-Driven Monitoring
11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries
12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines
12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling
13 Chapter 13: Automated Remediation & Deployment Troubleshooting
13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures
14 Chapter 14: Identity, Access Management & Secrets at Scale
14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager
15 Chapter 15: Security Automation, Data Protection & Compliance Auditing
15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config
11.1 Application & Infrastructure Health Checks with Route 53 & ALB
Key Takeaways Explain This Topic AI
Application Load Balancer target group health checks evaluate endpoint viability using eight discrete parameters: Protocol, Port, Path, Interval (5-300s), Timeout (2-120s), HealthyThresholdCount (2-10), UnhealthyThresholdCount (2-10), and Matcher HTTP status codes (such as 200 or 200-399); Timeout must always be strictly less than Interval.
Target deregistration delay (connection draining) pauses traffic routing to terminating targets for a configurable window (default 300s, range 0-3600s) to allow in-flight requests to complete cleanly without client 502/504 errors.
Deep health checks that synchronously query downstream databases or third-party APIs introduce catastrophic cascading failure risks; load balancer routing and auto-scaling decisions must rely on shallow, in-memory process liveness checks (/healthz) paired with decoupled circuit breakers.
Route 53 Alias records configured with EvaluateTargetHealth=true evaluate ALB target group metrics internally without incurring Route 53 health check charges or requiring security group ingress rules for global Route 53 health checker IP ranges.
EC2 System Status Checks identify underlying AWS physical hardware and hypervisor failures (remediated via EC2 Auto Recovery), whereas Instance Status Checks identify guest OS and software corruption (remediated via EC2 Reboot or Auto Scaling replacement); Auto Scaling groups require HealthCheckType=ELB and an adequate HealthCheckGracePeriod to avoid premature instance recycling.
Last updated: September 2026
Application Load Balancer Target Group Health Checks
In modern cloud architectures, the Application Load Balancer (ALB) acts as the primary ingress controller, distributing HTTP, HTTPS, and gRPC traffic across dynamic target groups composed of Amazon EC2 instances, ECS container tasks, Lambda functions, or private IP addresses. To ensure high availability and prevent routing traffic to degraded compute targets, the ALB continuously executes automated health checks against registered targets.
Core Health Check Parameters
ALB target group health checks are configured at the target group level and apply uniformly to all registered targets within that group. DevOps engineers must master the behavior, constraints, and operational defaults of each parameter:
Parameter Allowable Range Default Operational Description
HealthCheckProtocol HTTP , HTTPS , gRPC HTTP Protocol used by the load balancer nodes to connect to the target health endpoint.
HealthCheckPort traffic-port or 1-65535 traffic-port Port probed by health checkers. Choosing traffic-port uses the port on which the target receives user traffic.
HealthCheckPath Valid URI string (max 1024 chars) / Destination path for HTTP/HTTPS probes (e.g., /healthz , /status ). Must return a matching status code.
HealthCheckIntervalSeconds 5 - 300 seconds 30 seconds Time elapsed between individual health check attempts from each load balancer node to a specific target.
HealthCheckTimeoutSeconds 2 - 120 seconds 5 seconds Duration the load balancer waits for a response before counting the probe as a failure. Must be strictly less than Interval.
HealthyThresholdCount 2 - 10 consecutive checks 5 checks Number of consecutive successful responses required to transition a target from unhealthy to healthy .
UnhealthyThresholdCount 2 - 10 consecutive checks 2 checks Number of consecutive failed responses required to transition a target from healthy to unhealthy .
Matcher HTTP codes: single, comma list, or ranges (e.g., 200 , 200-299 , 200,301,302 ) 200 HTTP response codes that the target must return to be considered successful. gRPC status codes use 0-99 .
Important
Configuration Constraint Trap : In CloudFormation and the AWS CLI, attempting to set HealthCheckTimeoutSeconds equal to or greater than HealthCheckIntervalSeconds triggers an immediate InvalidConfigurationRequest error. The timeout must always be strictly smaller than the interval (e.g., Interval: 10s, Timeout: 5s).
Health State Transitions and Timing Formulas
Targets registered to an ALB transition through five distinct lifecycle states:
initial : Target is undergoing registration and the initial batch of health checks is in progress.
healthy : Target has met HealthyThresholdCount consecutive successful checks.
unhealthy : Target failed UnhealthyThresholdCount consecutive checks or timed out.
draining : Target has been deregistered or marked unhealthy, and in-flight requests are finishing.
unused : Target is not registered to an active target group or the target group is not bound to a listener rule.
The time required for an ALB to detect an outage and stop routing requests is determined by the formula: Detection Latency = HealthCheckIntervalSeconds × UnhealthyThresholdCount \text{Detection Latency} = \text{HealthCheckIntervalSeconds} \times \text{UnhealthyThresholdCount} Detection Latency = HealthCheckIntervalSeconds × UnhealthyThresholdCount
With default values ( 30 s × 2 30\text{s} \times 2 30 s × 2 ), an application outage takes up to 60 seconds to be detected per target, during which users receive HTTP 502/504 errors. In mission-critical environments, DevOps engineers optimize this by tuning the interval to 5 seconds and the unhealthy threshold to 2, reducing failure detection latency to 10 seconds .
Graceful Draining and Deregistration Delay
When a target is deregistered—whether through an automated blue/green deployment, an Auto Scaling scale-in event, or manual intervention—the ALB does not sever active TCP connections abruptly. Instead, it places the target into the draining state for the duration of the Deregistration Delay (also known as connection draining):
Default Value : 300 seconds (5 minutes). Configurable from 0 to 3600 seconds (1 hour).
Routing Behavior : The ALB immediately ceases sending new HTTP requests or TCP connections to the draining target. However, existing in-flight HTTP requests and active TCP sessions are permitted to continue until they complete cleanly.
Premature Completion : If all active in-flight requests finish before the deregistration delay expires, the ALB immediately closes remaining idle keep-alive connections and transitions the target to unused , without waiting for the full timer.
Tuning for CI/CD Pipelines : In containerized microservices (Amazon ECS / EKS) where API transactions execute in under 50 milliseconds, leaving deregistration delay at 300 seconds unnecessarily stalls rolling deployments for 5 minutes per batch. Lowering deregistration delay to 15 - 30 seconds dramatically accelerates deployment velocity while preventing dropped connections.
Shallow vs. Deep Health Checks: Preventing Cascading Failures
A critical design decision on the DOP-C02 exam is selecting the appropriate depth of the health check probe.
Attribute Shallow Health Check ( /healthz or /live ) Deep Health Check ( /healthz/deep or /ready )
Validation Scope Tests only local process liveness (web server process, memory, thread pool availability). Tests local process PLUS end-to-end downstream dependencies (database queries, Redis cache, 3rd-party APIs).
Response Mechanism Returns static 200 OK directly from memory without blocking I/O. Executes queries (e.g., SELECT 1 FROM orders ) and downstream HTTP ping requests.
Resource Cost Negligible CPU and memory overhead; sub-millisecond execution. Consumes database connections, worker threads, and network sockets per probe across all ALB nodes.
Primary Risk May keep target in service if downstream DB is down, but target itself is healthy. Cascading Failures : A transient DB spike causes all targets to fail checks simultaneously, killing the entire fleet.
Recommended Use ALB Target Group Health Checks and Auto Scaling group replacement triggers. Read-only diagnostic endpoints, deployment warmup verification, or Kubernetes readiness gates.
The Cascading Failure Anti-Pattern
Consider an architecture where 20 EC2 instances serve an e-commerce catalog behind an ALB. The target group is configured with a deep health check ( /health/full ) that executes a query against an Amazon Aurora MySQL database on every check. The health check interval is 10 seconds, and there are 3 ALB nodes in 3 Availability Zones: ALB Node 1 (AZ-a) ──┐
- ALB Node 2 (AZ-b) ──┼──> Probes 20 Instances every 10s = 60 DB queries / 10s = 6 QPS baseline
- ALB Node 3 (AZ-c) ──┘
If the Aurora database experiences a sudden query lock or CPU spike, query latency jumps from 5 ms to 6,000 ms:
The health check queries time out (exceeding the 5-second HealthCheckTimeoutSeconds ).
The ALB marks 5 instances as unhealthy .
The remaining 15 healthy instances must now absorb 100% of incoming user traffic.
Increased application load on the 15 instances drives even more connection pool contention and load to Aurora.
Within 30 seconds, the remaining 15 instances fail their deep health checks.
The ALB marks all 20 targets unhealthy and returns 503 Service Unavailable to 100% of end users.
Caution
Architectural Standard : Never configure an ALB target group health check to query downstream transactional databases or external synchronous dependencies directly. Implement a shallow liveness check for the ALB. To monitor downstream dependency health, use an asynchronous background thread that executes health pings periodically and sets a local circuit-breaker flag with exponential backoff and cached status.
Route 53 Health Checks & Multi-Region DNS Observability
Amazon Route 53 provides globally distributed DNS health checking to drive automated failover across multi-region and hybrid architectures. Route 53 operates outside the VPC from over 15 globally dispersed health checker locations.
Three Core Types of Route 53 Health Checks
Endpoint Health Checks : Probes an IP address or fully qualified domain name (FQDN) via HTTP, HTTPS, or TCP on a specified port (default 80 or 443).
Request Interval : Standard ( 30 seconds , default) or Fast ( 10 seconds , incurs higher cost).
Failure Threshold : Number of consecutive failed checks (range 1 - 10 , default 3 ) required before Route 53 marks the endpoint unhealthy.
String Matching : Inspects the first 5,120 bytes of the HTTP response body for a specific text string (e.g., "system_status": "OPERATIONAL" ).
CloudWatch Metric Health Checks : Monitors the status of an existing Amazon CloudWatch alarm. When the alarm enters ALARM state, the Route 53 health check transitions to unhealthy. Essential for alarming on application metrics (e.g., ALB 5xx error rate, SQS queue depth, or custom business KPIs) that cannot be probed via simple HTTP requests.
Calculated Health Checks : Aggregates up to 256 individual health checks into a single composite health decision using boolean logic ( AND , OR , NOT ) or threshold rules (e.g., health check passes if at least 3 of 5 regional endpoints are healthy). Useful for complex microservice dependency webs.
Route 53 Alias Records with EvaluateTargetHealth=true
When routing traffic to AWS resources—such as an Application Load Balancer, Network Load Balancer, or CloudFront distribution—using Route 53 Alias records, you should configure EvaluateTargetHealth=true : DNS Query: api.example.com
- │
- ▼
- Route 53 Alias Record (EvaluateTargetHealth = true)
- │
- ├─> Reads ALB Target Group Health directly from CloudWatch/ELB internal plane
- │ (No external probe charges, zero latency overhead)
- ▼
- ALB Target Group Health Status
- ├── All Targets Unhealthy? ──> Route 53 fails over to Secondary Region / S3 Error Page
- └── At least 1 Healthy? ──> Route 53 returns ALB DNS name / IP addresses
Zero Cost : Unlike standard Route 53 endpoint health checks, using EvaluateTargetHealth=true on Alias records pointing to an ALB incurs no additional health check fee .
No Security Group Ingress Needed : Standard endpoint health checks require opening your security group to hundreds of public Route 53 health checker IP addresses. In contrast, EvaluateTargetHealth queries the internal AWS control plane metrics of the load balancer directly, allowing your ALB security group to restrict ingress strictly to authorized client CIDRs or CloudFront managed prefix lists.
Target Group Sensitivity : If an ALB has multiple target groups bound to different listener rules, Route 53 evaluates the overall health: if any target group with registered targets has at least one healthy target, the ALB is considered healthy. If all targets across all target groups are unhealthy, Route 53 triggers DNS failover.
EC2 Status Checks & Automated Recovery
Amazon EC2 continuously monitors instances through two independent status check dimensions. Understanding the precise boundary between these checks is a frequent DOP-C02 question:
Check Dimension Underlying Causes Failure Indicator Automated Remediation
System Status Check ( StatusCheckFailed_System ) Infrastructure and AWS-managed hardware failures: physical host loss of power, hardware degradation, hypervisor software issues, or physical network switches. Underlying AWS hardware is degraded; host is unreachable from the AWS control plane. CloudWatch Alarm Action: EC2 Auto Recovery ( arn:aws:automate:<region>:ec2:recover ). Moves instance to a new physical host seamlessly.
Instance Status Check ( StatusCheckFailed_Instance ) Guest operating system and application failures: OS kernel panic, corrupted file systems, exhausted memory/threads, misconfigured network interfaces, or invalid driver configurations. AWS hardware is fully functional, but the guest OS is unresponsive to network traffic. CloudWatch Alarm Action: EC2 Reboot ( arn:aws:automate:<region>:ec2:reboot ) or Auto Scaling replacement.
EC2 Auto Recovery Architecture
When a CloudWatch alarm triggers on StatusCheckFailed_System :
The alarm executes the recovery action ( ec2:recover ).
AWS migrates the instance to a new, healthy physical server host within the same Availability Zone.
Preserved Attributes : The instance retains its original Instance ID , Private IPv4 address , Elastic IP address (public IPv4) , IPv6 addresses , and all attached EBS volumes and metadata.
Data In-Flight : Data in instance memory (RAM) is lost, exactly like a power cycle. Instance store (ephemeral) volume contents are lost. The instance must use an EBS-backed root volume to support Auto Recovery.
Auto Scaling Health Checks & Grace Periods
By default, an Amazon EC2 Auto Scaling group (ASG) evaluates only EC2 status checks . If an EC2 instance experiences an operating system freeze or hardware degradation, the ASG marks the instance unhealthy and initiates replacement.
The ELB Health Check Type Requirement Default ASG Behavior (HealthCheckType = EC2):
EC2 Hypervisor OK + Guest OS Network OK ──> ASG considers instance HEALTHY!
(Even if Web Server is crashed, returning 500 Internal Server Error, or ALB marks target UNHEALTHY)
Production ASG Behavior (HealthCheckType = ELB):
- EC2 Status Checks OK AND ALB Target Group reports HEALTHY ──> ASG considers instance HEALTHY
- If ALB marks target UNHEALTHY ──> ASG terminates and replaces instance automatically
Important
DOP-C02 Core Mandate : To enable Auto Scaling to automatically replace instances that fail application-layer health checks (e.g., returning 5xx status codes to the ALB), you must explicitly configure HealthCheckType: "ELB" on the Auto Scaling group. If set to EC2 , the ASG will keep instances running indefinitely even when the ALB has removed them from service due to failing HTTP health probes.
Health Check Grace Period ( HealthCheckGracePeriod )
When Auto Scaling launches a new EC2 instance, the application runtime (e.g., JVM startup, container image download, database connection pool establishment, local cache pre-warming) requires time before it can successfully respond to HTTP health probes with 200 OK .
The Health Check Grace Period defines the duration (default 300 seconds , range 0 - 7200 seconds ) that the ASG waits after an instance enters the InService state before checking its ELB health status.
The Premature Termination Death Loop : If the grace period is set too low (e.g., 30 seconds) for an enterprise Java application that takes 90 seconds to boot, the ALB probe will fail during startup. The ASG immediately flags the instance as unhealthy , terminates it, launches a replacement, and repeats the cycle perpetually, resulting in zero available capacity.
During the grace period, if the instance fails EC2 System Status Checks , the ASG replaces it immediately; the grace period applies strictly to application health checks (ELB).
DOP-C02 Exam Watchouts & Troubleshooting
Scenario / Symptom Root Cause Solution
ALB returns 502 Bad Gateway immediately after deploying new container tasks in ECS Containerized app takes 10s to start listening, but ALB health check timeout is 5s and unhealthy threshold is 2 Increase HealthCheckIntervalSeconds or add startPeriod in ECS container task definition to allow warmup before health check starts.
Auto Scaling instances continuously terminate and relaunch every 4 minutes HealthCheckType is ELB and application takes 180s to boot, but HealthCheckGracePeriod is configured to 60s Increase HealthCheckGracePeriod on the ASG to at least 240s to accommodate initialization and cold cache warming.
Route 53 fails over to disaster recovery region even though primary region EC2 instances are healthy Route 53 endpoint check uses deep health check that checks an external billing API, which suffered an outage Decouple endpoint health check from third-party APIs; use shallow health check or calculated check with threshold logic.
ALB marks all targets unhealthy simultaneously during a traffic surge, despite low target CPU Database connection pool exhaustion caused /healthz deep check queries to queue and exceed HealthCheckTimeoutSeconds Switch target group health check path to a lightweight, static /healthz shallow endpoint that does not query the database.
EC2 Auto Recovery alarm fires but returns Client.InvalidAction The instance has an instance-store root volume or belongs to an unsupported legacy instance family Ensure the instance uses an EBS-backed root volume and is on a modern nitro or supported virtualization family.