13.3 Troubleshooting CI/CD Pipeline & Deployment Failures

AWS Certified DevOps Engineer – Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CI/CD Pipeline Architecture & Source Integration

1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models

2 Chapter 2: Automated Testing, Test Gates & Quality Verification

2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates

3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies

3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda

4 Chapter 4: CloudFormation, CDK & Reusable IaC Components

4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance

5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower

5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation

6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration

6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig

7 Chapter 7: High Availability & Multi-Region Resiliency

7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR

8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation

8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery

9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation

9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray

10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps

11 Chapter 11: Automated Observability & Event-Driven Monitoring

11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries

12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines

12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling

13 Chapter 13: Automated Remediation & Deployment Troubleshooting

13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures

14 Chapter 14: Identity, Access Management & Secrets at Scale

14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager

15 Chapter 15: Security Automation, Data Protection & Compliance Auditing

15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config

13.3 Troubleshooting CI/CD Pipeline & Deployment Failures

Key Takeaways Explain This Topic AI

Cross-account CodePipeline deployments fail when artifact S3 buckets use the default aws/s3 KMS key; an AWS KMS Customer Managed Key (CMK) with cross-account key policy permissions is mandatory.

CodeBuild projects running inside a custom VPC fail outbound internet calls unless deployed in private subnets routed through a NAT Gateway or supported VPC Endpoints; public subnets lack auto-assigned public IPs.

CodeBuild Docker image builds require enabling the privileged mode flag (privileged: true); otherwise, the Docker daemon cannot initialize inside the build container.

CodeDeploy in-place deployments fail at the ApplicationStop lifecycle hook when previous code or scripts are corrupted; setting --ignore-application-stop-failures or authoring idempotent scripts resolves this block.

AWS CloudFormation Rollback Triggers monitor CloudWatch alarms during stack operations, while setting OnFailure / DisableRollback (DO_NOTHING) preserves failed resources for root-cause diagnosis.

Last updated: September 2026

AWS CodePipeline Failure Analysis & Cross-Account Permissions

AWS CodePipeline orchestrates continuous delivery workflows. When pipeline executions stall or fail, DevOps engineers must rapidly pinpoint whether the root cause stems from stage transitions, IAM role trust relationships, or cross-account artifact encryption.

Common CodePipeline Execution Failure Modes

Disabled Stage Transitions : A stage transition can be deliberately or programmatically disabled, preventing pipeline executions from advancing. The pipeline execution displays a Transition disabled indicator with a recorded reason.

Pipeline Concurrency Behavior : By default, CodePipeline uses Superseded mode: if a newer pipeline execution reaches a stage while an older execution is running, the newer execution overtakes the older execution, causing the older execution to be canceled.

Artifact Bucket Policy & Ownership Mismatches : When CodePipeline writes build or test output to an Amazon S3 artifact bucket, S3 bucket policies enforcing bucket owner full control ( s3:PutObjectAcl with bucket-owner-full-control ) will reject uploads if the writing role does not explicitly attach the required canned ACL.

Cross-Account Deployment Permissions & The KMS CMK Requirement

Important

Critical DOP-C02 Concept : In cross-account CodePipeline architectures (e.g., Tooling Account A deploying to Production Account B), the default AWS-managed KMS key ( aws/s3 ) CANNOT be used. AWS-managed keys do not permit cross-account key policies. An AWS KMS Customer Managed Key (CMK) is mandatory. [ Tooling Account A (Pipeline Hub) ]

• ├── CodePipeline Service Role
• └── S3 Artifact Bucket (Encrypted with KMS Customer Managed Key)
• │
• ┌─────────────────────┴─────────────────────┐
• ▼ ▼
• [ KMS Key Policy (Account A) ] [ S3 Bucket Policy (Account A) ]
• Grants kms:Decrypt & GenerateDataKey Grants s3:GetObject* & PutObject*
• to Account B Cross-Account Role to Account B Cross-Account Role
• ▲ ▲
• └─────────────────────┬─────────────────────┘
• │ (STS AssumeRole)
• [ Production Account B (Deploy Target) ]
• └── CodeDeploy / CloudFormation Deployer Role
• Triad of Cross-Account Requirements

KMS Key Policy (in Account A) : Must explicitly grant kms:Decrypt , kms:DescribeKey , and kms:GenerateDataKey* to the deployment IAM role in Account B.

S3 Bucket Policy (in Account A) : Must grant s3:GetObject* , s3:PutObject* , and s3:ListBucket to the deployment IAM role in Account B.

Deployment IAM Role (in Account B) : Must possess IAM permissions allowing kms:Decrypt against the Account A KMS Key ARN, and must have a trust policy allowing CodePipeline in Account A ( codepipeline.amazonaws.com ) or the Account A Pipeline Role to assume it.

Loading diagram... CodeDeploy Lifecycle Hook Execution Order and Rollback Decision Tree

AWS CodeBuild Failure Modes & Diagnostics

AWS CodeBuild provisions isolated Docker containers to execute build specifications ( buildspec.yml ). When builds fail, DevOps engineers must systematically evaluate container networking, caching, credentials, and build phases.

1. VPC Subnet Routing & Missing Outbound Connectivity

A classic scenario tested on the DOP-C02 exam involves running CodeBuild inside a custom VPC (e.g., to access internal Amazon RDS databases, private code repositories, or internal microservices):

The Failure : CodeBuild fails during the pre_build phase with timeout errors when pulling public Docker images from Docker Hub, downloading npm/pip packages, or connecting to AWS public endpoints (S3, ECR, Secrets Manager).

The Cause : The CodeBuild project was configured to attach to public subnets with an Internet Gateway (IGW). CodeBuild provisions Elastic Network Interfaces (ENIs) inside the chosen subnets. However, CodeBuild ENIs do not receive public IPv4 addresses . Even if the subnet's route table points 0.0.0.0/0 to an Internet Gateway, traffic cannot egress because the ENIs have only private IPs and no public NAT mapping.

The Resolution : CodeBuild ENIs MUST be placed in private subnets whose route tables route 0.0.0.0/0 through a NAT Gateway , or VPC Endpoints must be deployed for all required AWS services (ECR, S3, CloudWatch Logs, Secrets Manager).

2. Docker Daemon Initialization & Privileged Mode

When building Docker images inside CodeBuild, the buildspec.yml typically executes docker build or docker-compose . If the build fails with: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Root Cause : The CodeBuild project environment lacks root/daemon capabilities. Docker-in-Docker operations require enabling the privileged: true flag in the CodeBuild project settings ( Environment.PrivilegedMode in CloudFormation).

3. ECR Authentication & Layer Caching

ECR Login Failures : Running aws ecr get-login-password fails with AccessDeniedException if the CodeBuild service role lacks ecr:GetAuthorizationToken or repository permissions ( ecr:BatchCheckLayerAvailability , ecr:BatchGetImage , ecr:PutImage ).

Missing Docker Layer Caching : Long build durations indicate Docker layer caching is inactive. To resolve, configure CodeBuild Local Docker Layer Cache (stores layers directly on the build host for repeated builds on same compute) or S3 Cache (persists layers across builds into an S3 bucket).

4. Buildspec Phase Exit Codes & Environment Variable Decryption

CodeBuild executes commands across sequential phases: install , pre_build , build , and post_build . By default, if any command in a phase returns a non-zero exit code , CodeBuild immediately terminates the build.

Parameter Store / Secrets Manager Decryption Failures : If buildspec.yml references secrets in env.parameter-store or env.secrets-manager , CodeBuild attempts to resolve them before executing phases. If the CodeBuild service role lacks ssm:GetParameters , secretsmanager:GetSecretValue , or kms:Decrypt permissions, the build fails immediately during container initialization before any phase commands run.

AWS CodeDeploy Failure Modes & Resolution Strategies

AWS CodeDeploy automates application deployments across Amazon EC2, AWS Lambda, and Amazon ECS. Each deployment targets lifecycle event hooks defined in the appspec.yml .

Diagnostic Matrix for CodeDeploy Failures

Failure Scenario Error / Symptom Root Cause Resolution Strategy

Agent Offline The overall deployment failed because too many instances were in an unhealthy state The CodeDeploy agent daemon on EC2 is stopped, uninstalled, or lacks IAM permissions ( codedeploy:PutLifecycleEventHookExecutionStatus ) Start/restart daemon ( systemctl restart codedeploy-agent ); verify instance profile has AWSCodeDeployRole policy.

ApplicationStop Hook Failure LifecycleEvent - ApplicationStop failed CodeDeploy executes the ApplicationStop script from the previous successful deployment , not the new revision. If the old script is missing or bugs were introduced, it fails. In the deployment request, enable --ignore-application-stop-failures ; author all hook scripts to be idempotent ( set +e , exit 0 ).

ValidateService Timeout LifecycleEvent - ValidateService script timed out The application takes longer to start than the hook timeout, or health check URL returns HTTP 5xx / connection refused Increase timeout in appspec.yml (e.g., from 30s to 300s); add exponential sleep loop polling the localhost endpoint.

ALB Health Check Failure The deployment failed because targets in the target group never reached healthy status In Blue/Green traffic shifting, the replacement instance fails target group health checks during AfterAllowTestTraffic Check security groups allowing ALB ingress on application port; verify health check path returns matching HTTP status code.

Permission Boundary Failure AccessDenied: User is not authorized to perform: iam:PassRole / elb:RegisterTargets CodeDeploy service role is restricted by an IAM Permission Boundary or Service Control Policy (SCP) Update IAM permission boundary to allow ELBv2 and Auto Scaling target manipulation.

AWS CloudFormation Rollback Triggers & Debugging

When a CloudFormation stack update fails, CloudFormation automatically initiates a rollback to the last known stable state. While this protects production from broken states, it destroys the failed resources and logs, complicating root-cause analysis.

Rollback Triggers & Monitoring Time

Rollback Triggers allow CloudFormation to monitor Amazon CloudWatch Alarms during stack creation or updates. If any configured alarm triggers during the deployment or during the specified monitoring time (e.g., MonitoringTimeInMinutes: 30 after resource creation completes), CloudFormation rolls back the entire deployment.

Preserving Failed Stacks for Root-Cause Analysis

To prevent CloudFormation from rolling back failed resources during troubleshooting:

CLI / Pipeline Setting : Use --disable-rollback (or in CloudFormation console, set Stack failure options to Preserve successfully provisioned resources / OnFailure: DO_NOTHING ).

Forensic Capability : The failed resource remains in CREATE_FAILED status. Engineers can connect to the underlying EC2 instance via AWS Systems Manager Session Manager, inspect application logs in /var/log/ , review cfn-init.log and cfn-wire.log , and diagnose configuration issues before recreating the stack.

Recovering from UPDATE_ROLLBACK_FAILED

If a resource cannot be rolled back (e.g., an S3 bucket cannot be deleted during rollback because objects were added during deployment, or an elastic IP was manually released), the stack enters UPDATE_ROLLBACK_FAILED .

Resolution : Use the ContinueUpdateRollback API and specify the non-rollbackable resource in ResourcesToSkip . CloudFormation skips the specified resource and restores the stack to an UPDATE_ROLLBACK_COMPLETE operational state.