Study guide
6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics
Key Takeaways Explain This Topic AI
Pod startup transitions through distinct phases (Pending, ContainerCreating, Running, CrashLoopBackOff, OOMKilled, Error, ImagePullBackOff); each failure state points to a specific layer in the container lifecycle.
Container exit codes reveal the root cause of crashes: Exit Code 0 (clean exit, missing long-running foreground process), Exit Code 1 (application runtime exception), Exit Code 137 (SIGKILL / OOMKilled by cgroup or kernel), Exit Code 139 (Segmentation fault), Exit Code 143 (SIGTERM graceful termination).
Differentiating probe failures is critical: startupProbe failures prevent containers from ever reaching Ready, livenessProbe failures trigger recurring container restarts, and readinessProbe failures remove the pod from Service Endpoints without restarting the container.
InitContainers execute sequentially to completion before any app container starts; if an initContainer fails or loops, application containers remain in 'Init:CrashLoopBackOff' or 'Init:0/1'.
Advanced pod debugging leverages 'kubectl logs <pod> --previous' for crashed instances, ephemeral debug containers ('kubectl debug -it <pod> --image=busybox --target=<container>'), and host kernel ring buffers ('dmesg -T | grep -i oom').
Last updated: August 2026
6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics
Pods are the atomic scheduling units of Kubernetes. When a pod fails to reach or maintain the Running and Ready state, applications suffer downtime. Because pods encapsulate container runtime processes, Linux namespaces, cgroups, network plumbing, storage volume mounts, and health probes, diagnosing pod failures requires a structured, step-by-step approach.
1. The Pod Lifecycle State Machine & Failure Taxonomy +-----------------------------------------------------------------------------------------+
- | POD LIFECYCLE & FAILURE TAXONOMY |
- | |
- | [SUBMITTED] |
- | | |
- | v |
- | [PENDING] -------------> Failure: FailedScheduling (Insufficient resources, Taints) |
- | | |
- | v |
- | [CONTAINERCREATING] ---> Failure: ImagePullBackOff / ErrImagePull (Registry/Tag typo) |
- | | ---> Failure: FailedMount / MountVolume.SetUp (Secret/PVC missing) |
- | | ---> Failure: Init:CrashLoopBackOff (InitContainer failed) |
- | v |
- | [RUNNING] |
- | | |
- | +---> [CRASHED / EXITED] ---> Exit Code 0 (Completed / No foreground process) |
- | | ---> Exit Code 1 (App error / bad config / unhandled exc)|
- | | ---> Exit Code 137 (OOMKilled - cgroup limit exceeded) |
- | | ---> Exit Code 139 (SIGSEGV - Segmentation Fault) |
- | | ---> Exit Code 143 (SIGTERM - Graceful stop timeout) |
- | v |
- | [CRASHLOOPBACKOFF] -----> Kubelet applies exponential backoff restart delay: |
- | 10s -> 20s -> 40s -> 80s -> 160s -> max 300s (5 mins) |
- | |
- | [UNHEALTHY PROBES] -----> Liveness Failure: Container restarted by kubelet |
- | -----> Readiness Failure: Pod removed from Endpoints (No restart) |
- +-----------------------------------------------------------------------------------------+
Detailed Breakdown of Pod Failure States:
ImagePullBackOff / ErrImagePull :
Symptoms: Container cannot be created because the container image cannot be retrieved.
Root Causes: Typo in image repository or tag (e.g., nginx:1.999 ), non-existent private repository, network failure communicating with registry, or missing/malformed imagePullSecrets .
Diagnostic: kubectl describe pod <name> -> Inspect Events .
CrashLoopBackOff :
Symptoms: The container starts, executes for a brief period, crashes/terminates, and is repeatedly restarted by the kubelet with an exponential backoff penalty (10s, 20s, 40s, ..., up to 300s).
Root Causes: Missing environment variables, missing configuration files, application panics, failed database connection on startup, or running a base OS container (like ubuntu or busybox ) without a long-running foreground command.
Diagnostic: kubectl logs <name> --previous and kubectl describe pod <name> .
OOMKilled (Out Of Memory):
Symptoms: Pod state displays OOMKilled or CrashLoopBackOff with Exit Code: 137 and Reason: OOMKilled .
Root Causes: The container's memory usage exceeded the strict .spec.containers[].resources.limits.memory defined in its manifest, triggering the Linux cgroup memory controller to send SIGKILL (signal 9) to the process ( 128 + 9 = 137 128 + 9 = 137 128 + 9 = 137 ).
Diagnostic: kubectl describe pod <name> -> Last State: Terminated , Reason: OOMKilled , Exit Code: 137 .
2. Container Exit Code Reference Table
When a container terminates, the Linux kernel and container runtime record an integer exit status. Interpreting this exit code provides immediate insight into why the process stopped:
Exit Code Signal / Type Technical Meaning & Common Causes
0 Success Process completed cleanly. For long-running server pods, this indicates the container entrypoint finished (e.g., a batch script exited or daemon went to background).
1 Application Error General application runtime exception, syntax error in interpreted scripts (Python, Node), or unhandled fatal error.
126 Cannot Execute Container entrypoint or command found, but lacks executable permissions ( chmod +x ).
127 Command Not Found Specified command or args binary does not exist inside the container filesystem (e.g., calling /bin/bash in an Alpine container that only has /bin/sh ).
137 SIGKILL (128 + 9) Process immediately terminated by external signal 9. Most commonly OOMKilled by cgroup limit, or terminated forcefully by kubectl delete --force .
139 SIGSEGV (128 + 11) Segmentation fault; application attempted to access unallocated memory or corrupted pointer in compiled binaries (C/C++, Go, Rust).
143 SIGTERM (128 + 15) Graceful termination signal sent by Kubernetes during pod deletion or probe failure; container did not shut down before grace period expired.
3. Diagnosing Health Probe Failures
Kubernetes provides three distinct probe mechanisms. Misunderstanding the differences leads to severe misdiagnoses: +-----------------------------------------------------------------------------------------+
- | HEALTH PROBE MATRIX |
- | |
- | +-------------------+ +------------------------------------------------------------+ |
- | | STARTUP PROBE | | Protects slow-starting containers (e.g., legacy Java/JVM). | |
- | | | | All other probes are DISABLED until startupProbe succeeds. | |
- | | | | Failure Action: Container is KILLED and RESTARTED. | |
- | +-------------------+ +------------------------------------------------------------+ |
- | +-------------------+ +------------------------------------------------------------+ |
- | | LIVENESS PROBE | | Detects runtime deadlocks or frozen processes. | |
- | | | | Failure Action: Container is KILLED and RESTARTED. | |
- | +-------------------+ +------------------------------------------------------------+ |
- | +-------------------+ +------------------------------------------------------------+ |
- | | READINESS PROBE | | Detects if container is ready to serve network traffic. | |
- | | | | Failure Action: Pod IP REMOVED from Service Endpoints. | |
- | | | | (CONTAINER IS NOT RESTARTED!). | |
- | +-------------------+ +------------------------------------------------------------+ |
- +-----------------------------------------------------------------------------------------+
Common Probe Configuration Pitfalls:
Liveness Probe Too Aggressive: Setting initialDelaySeconds: 0 on a heavy application causes the liveness probe to fail before the app boots, entering an endless restart loop.
Readiness vs. Liveness Confusion: If an external database is down, the pod's readiness probe should fail (taking it out of service routing), but its liveness probe should NOT fail (restarting the pod won't fix an external database). # Example of properly tuned probes
spec:
containers:
- name: web
- image: app:v1
startupProbe:
httpGet:
- path: /healthz
- port: 8080
- failureThreshold: 30
- periodSeconds: 10 # Allows up to 300s (5 mins) for initial startup
livenessProbe:
httpGet:
- path: /healthz
- port: 8080
- periodSeconds: 10
- failureThreshold: 3
readinessProbe:
httpGet:
- path: /ready
- port: 8080
- periodSeconds: 5
- failureThreshold: 2
- 4. InitContainer Failures
initContainers execute sequentially before application containers start. If an initContainer fails:
Pod status displays Init:0/1 , Init:Error , or Init:CrashLoopBackOff .
- Application containers never start. # Check status of initContainers
- kubectl describe pod <pod-name>
- # Stream logs from the specific initContainer
- kubectl logs <pod-name> -c <init-container-name>
- 5. Ephemeral Debug Containers & Kernel OOM Verification
- Ephemeral Containers ( kubectl debug )
In modern Kubernetes, distroless and minimal container images lack debugging utilities like curl , sh , or netstat . Ephemeral debug containers attach to an existing pod's namespaces: # Attach an interactive debug container sharing the target pod's process namespace
kubectl debug -it target-pod --image=busybox:1.36 --target=app-container
- # Create a copy of a broken pod with modified command or environment variables
- kubectl debug target-pod -it --copy-to=debug-pod --container=app-container -- /bin/sh
- Verifying Host-Level Kernel OOM Events
If Kubernetes events do not explicitly say OOMKilled , check the host Linux kernel ring buffer: # SSH to worker node where pod was running
ssh node01
- # Inspect dmesg for Linux OOM Killer invocations
- dmesg -T | grep -i -E "oom-killer|killed process"