13.4 Root Cause Analysis for Auto Scaling & Container Failures

AWS Certified DevOps Engineer – Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CI/CD Pipeline Architecture & Source Integration

1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models

2 Chapter 2: Automated Testing, Test Gates & Quality Verification

2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates

3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies

3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda

4 Chapter 4: CloudFormation, CDK & Reusable IaC Components

4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance

5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower

5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation

6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration

6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig

7 Chapter 7: High Availability & Multi-Region Resiliency

7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR

8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation

8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery

9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation

9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray

10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps

11 Chapter 11: Automated Observability & Event-Driven Monitoring

11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries

12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines

12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling

13 Chapter 13: Automated Remediation & Deployment Troubleshooting

13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures

14 Chapter 14: Identity, Access Management & Secrets at Scale

14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager

15 Chapter 15: Security Automation, Data Protection & Compliance Auditing

15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config

13.4 Root Cause Analysis for Auto Scaling & Container Failures

Key Takeaways Explain This Topic AI

EC2 Auto Scaling launch failures are diagnosed via DescribeScalingActivities, typically stemming from service quotas, subnet IP exhaustion, invalid launch template configurations, or InsufficientInstanceCapacity.

Amazon ECS task terminations distinguish between Linux OOM killer termination (Exit Code 137, where container memory exceeds hard memory limits) and application crashes (Exit Code 1, uncaught exceptions).

The ECS Task Execution Role grants permissions to the ECS agent/Fargate infrastructure (pulling ECR images, writing CloudWatch logs, decrypting secrets), while the ECS Task Role grants permissions to the application code inside the container.

Amazon EKS workload failures require differentiating CrashLoopBackOff (application startup crash), OOMKilled (cgroup memory limit exceeded), and VPC CNI secondary IP exhaustion (mitigated via prefix delegation).

AWS Health events and CloudTrail Lake SQL queries correlate application service degradation with underlying AWS physical host retirements and mutating control-plane API calls.

Last updated: September 2026

Root Cause Analysis for EC2 Auto Scaling Launch Failures

When an Amazon EC2 Auto Scaling group (ASG) fails to launch replacement instances or scale out during high traffic, application availability quickly degrades. Rather than guessing, DevOps engineers must query the scaling activities via the AWS CLI or console: aws autoscaling describe-scaling-activities \

• -auto-scaling-group-name production-asg \
• -max-items 5
• Diagnostic Matrix for Auto Scaling Launch Failures

Error Message in DescribeScalingActivities Underlying Root Cause Architectural Resolution

You have reached your quota for number of instances for this instance type The AWS account has breached its regional EC2 On-Demand or Spot vCPU Service Quota for the target instance family. Request a service quota increase via AWS Service Quotas console; configure ASG Mixed Instances Policy across multiple instance families.

Cannot create instance because the subnet has no available IP addresses Subnet CIDR IP exhaustion. All available private IPv4 addresses in the target VPC subnets are consumed by existing instances, ENIs, or ALBs. Add secondary CIDR blocks to the VPC; expand subnets; attach additional subnets across alternative Availability Zones to the ASG.

The AMI ID 'ami-012345678' does not exist or architecture mismatch The AMI specified in the Launch Template has been deregistered, deleted, lacks cross-account sharing permissions, or architecture mismatch (x86_64 vs. ARM64 Graviton). Update Launch Template with valid, accessible AMI ID; verify instance type CPU architecture matches the compiled AMI architecture.

The security group 'sg-xyz' does not exist The Security Group referenced in the Launch Template was deleted, or belongs to a different VPC than the ASG subnets. Ensure Security Groups belong to the exact VPC associated with the ASG subnets; update Launch Template to latest revision.

InsufficientInstanceCapacity: We currently do not have sufficient capacity in the Availability Zone you requested AWS physical infrastructure in that specific AZ lacks available host capacity for the requested instance family and size. Implement an Auto Scaling Mixed Instances Policy with multiple instance families (e.g., m5.large , m5a.large , c5.large ); deploy across 3+ Availability Zones.

Amazon ECS Task & Service Failure Modes

Amazon Elastic Container Service (ECS) manages container lifecycle on EC2 container instances and AWS Fargate. When an ECS task stops unexpectedly, ECS records a Stopped Reason and Container Exit Code .

Loading diagram... Container and Compute Troubleshooting Decision Tree

Decoding Container Exit Codes

When a container marked essential: true exits, ECS immediately stops all other containers in the task:

Exit Code 137 ( SIGKILL - Out-Of-Memory Killer) :

Root Cause : The container's memory consumption exceeded its allocated limit ( memory parameter in task definition), or the sum of container memory exceeded the task-level memory limit. The Linux kernel OOM killer sent a SIGKILL (128 + 9 = 137) to terminate the container.

Resolution : Increase the container memory limit or task-level memory allocation in the task definition. Alternatively, tune application memory management (e.g., JVM heap limits -Xmx ).

Exit Code 1 (Application Runtime Error) :

Root Cause : The application process exited with a general error (e.g., unhandled exception, syntax error, missing mandatory environment variable, database connection failure on startup).

Resolution : Query Amazon CloudWatch Logs for the container log stream to inspect stdout / stderr stack traces.

Exit Code 143 ( SIGTERM ) :

Root Cause : Graceful termination (128 + 15 = 143). ECS sent a SIGTERM signal during a service deployment, Auto Scaling scale-in event, or task retirement. If the container does not exit within the stopTimeout (default 30 seconds), ECS sends SIGKILL (Exit Code 137).

Exit Code 125 : Docker daemon error (e.g., invalid docker run command parameters).

Exit Code 126 : The specified container entrypoint or command cannot be invoked (e.g., file permissions error: permission denied ).

Exit Code 127 : The specified command was not found in the container's PATH .

CannotPullContainerError

If the task stops before the application starts with CannotPullContainerError :

ECR Authentication : Verify the ECS Task Execution Role has permissions to call ecr:GetAuthorizationToken , ecr:BatchCheckLayerAvailability , and ecr:BatchGetImage .

Network Routing : For tasks running on Fargate or EC2 in private subnets, outbound traffic to ECR must be routed through a NAT Gateway or interface VPC Endpoints ( com.amazonaws.region.ecr.api , com.amazonaws.region.ecr.dkr , and an S3 Gateway Endpoint com.amazonaws.region.s3 to download image layers stored in S3).

Architectural Distinction: Task Role vs. Task Execution Role

Dimension ECS Task Execution Role ( executionRoleArn ) ECS Task Role ( taskRoleArn )

Who Uses It? The AWS ECS Container Agent and Fargate infrastructure plane The Application Code running inside the container

Primary Purpose Infrastructure provisioning and container startup Application runtime business operations

Required Permissions - Pulling images from private Amazon ECR; - Writing logs to Amazon CloudWatch Logs ( logs:CreateLogStream , logs:PutLogEvents ); - Decrypting secrets from AWS Secrets Manager / Parameter Store - Reading/writing to Amazon S3; - Querying Amazon DynamoDB; - Publishing messages to Amazon SQS / SNS; - Invoking other AWS microservices

• Trust Principal ecs-tasks.amazonaws.com ecs-tasks.amazonaws.com
• Amazon EKS Workload & Networking Failures

Amazon Elastic Kubernetes Service (EKS) introduces Kubernetes-native failure abstractions that DevOps engineers must diagnose using kubectl and AWS telemetry.

1. Pod Failure States & Root Causes

CrashLoopBackOff : The application container starts, fails, crashes, and is restarted repeatedly by the kubelet with exponential backoff. Diagnosis: Run kubectl logs <pod-name> --previous to inspect logs from the crashed container before the restart.

OOMKilled : The container exceeded its Kubernetes resource limit ( resources.limits.memory ). Diagnosis: Run kubectl describe pod <pod-name> , which reveals Last State: Terminated, Reason: OOMKilled, Exit Code: 137 .

ImagePullBackOff / ErrImagePull : The kubelet cannot pull the specified container image due to an invalid image tag, missing image pull secret, or unauthorized private registry.

Pending : The pod cannot be scheduled onto any worker node. Diagnosis: Check kubectl describe pod under Events: FailedScheduling . Causes include insufficient CPU/memory requests ( Insufficient cpu , Insufficient memory ), node taints without matching tolerations, or conflicting nodeSelector / nodeAffinity rules.

2. AWS VPC CNI Secondary IP Exhaustion

The AWS VPC Container Network Interface (CNI) plugin assigns native VPC private IP addresses directly to pods. Each EC2 worker node can only attach a fixed number of ENIs, and each ENI supports a limited number of secondary private IPv4 addresses based on the EC2 instance type.

The Failure : Pods remain in Pending or fail with FailedCreatePodSandBox: failed to assign an IP address to container .

The Resolution: Prefix Delegation : Rather than allocating individual secondary IP addresses per ENI slot, enable Prefix Delegation on the AWS VPC CNI: kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Prefix delegation assigns a /28 IPv4 subnet prefix (16 IP addresses) to each ENI slot instead of a single IP, dramatically increasing pod density per worker node without exhausting ENI limits.

Investigating Infrastructure Root Cause with AWS Health & CloudTrail

When multiple unrelated applications simultaneously experience intermittent connectivity, high latency, or sudden instance termination, the root cause often resides in the underlying AWS infrastructure rather than application code.

Correlating Telemetry

AWS Health API : Query DescribeEvents and DescribeAffectedEntities to verify if AWS has flagged degraded underlying hardware or scheduled retirement: aws health describe-affected-entities \

--filter "eventArns=['arn:aws:health:us-east-1::event/EC2/AWS_EC2_PERSISTENT_INSTANCE_RETIREMENT_SCHEDULED/...']"

CloudTrail Lake Forensics : When suspicious configuration drift or accidental resource termination occurs, use CloudTrail Lake to query multi-account, multi-region API logs with SQL: SELECT

• eventTime,
• eventName,
• userIdentity.arn AS actor,
• requestParameters
• FROM
• eds-secops-trail
• WHERE
• eventName IN ('TerminateInstances', 'DeleteSecurityGroup', 'UpdateSecurityGroupRuleDescriptionsIngress')
• AND eventTime > '2026-09-11 14:00:00'
• ORDER BY
• eventTime DESC;