Study guide
12.2 Troubleshooting Compute, Container & Storage Failures
CompTIA Cloud+ Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Cloud Architecture Fundamentals
1.1 Cloud Service Models & Shared Responsibility 1.2 Cloud Deployment Models & Multi-Cloud Strategies 1.3 High Availability, Resiliency & Redundancy 1.4 Disaster Recovery & Business Continuity 1.5 Evolving Cloud Technologies: AI/ML Services, IoT & Edge Computing
2 Chapter 2: Cloud Compute & Storage Architecture
2.1 Virtualization & Hypervisor Architectures 2.2 Containerization & Microservices Architecture 2.3 Container Orchestration & Kubernetes Architecture 2.4 Cloud Storage Architectures & Performance
3 Chapter 3: Cloud Networking Architecture
3.1 Virtual Private Clouds & Subnetting Architecture 3.2 Hybrid & Multi-Cloud Connectivity 3.3 DNS, Load Balancing & Traffic Routing 3.4 Cloud Network Security Architecture
4 Chapter 4: Workload Assessment & Migration Planning
4.1 Cloud Migration Strategies (The 6/7 Rs) 4.2 Pre-Migration Workload Assessment & Sizing 4.3 Data & Database Migration Methods 4.4 Migration Execution, Cutover & Post-Migration Validation
5 Chapter 5: Infrastructure as Code & Provisioning
5.1 Infrastructure as Code Fundamentals & State Management 5.2 IaC Tooling & Template Authoring 5.3 Provisioning Automation & Image Management 5.4 Deployment Strategies & Release Patterns
6 Chapter 6: Cloud Operations & Lifecycle Management
6.1 Resource Lifecycle & Patch Management 6.2 Cloud Scaling Strategies & Elasticity 6.3 Cloud Cost Management & Optimization (FinOps) 6.4 Capacity Planning & Service Quotas
7 Chapter 7: Cloud Backup, Recovery & Business Continuity Operations
7.1 Backup Strategies, Storage Media & Retention Policies 7.2 Immutable Backups, Vault Locking & Ransomware Defense 7.3 Point-in-Time Recovery & Data Integrity Validation 7.4 High Availability Failover Drills & Chaos Engineering
8 Chapter 8: Cloud Security Controls & Identity and Access Management
8.1 Identity & Access Management, RBAC & ABAC 8.2 Identity Federation, SSO & Directory Integration 8.3 Multi-Factor Authentication, PAM & Machine Identities 8.4 Secrets Management & Cryptographic Credential Hygiene
9 Chapter 9: Data Protection, Vulnerability & Compliance
9.1 Data Encryption, Key Management & Confidential Computing 9.2 Vulnerability Management, Posture Assessment & CWPP 9.3 Container, Workload & Runtime Security 9.4 Cloud Compliance Frameworks & Governance Auditing
10 Chapter 10: DevOps Fundamentals & CI/CD Pipelines
10.1 Version Control Systems & Branching Strategies 10.2 Continuous Integration & Automated Testing Pipelines 10.3 Continuous Delivery, Deployment Automation & GitOps 10.4 Configuration Management & Systems Integration
11 Chapter 11: Cloud Observability & Monitoring
11.1 The Three Pillars of Observability: Metrics, Logs & Traces 11.2 Cloud Monitoring Architecture, Alerting & SLOs 11.3 Centralized Log Aggregation, SIEM & Log Analytics
12 Chapter 12: Troubleshooting Cloud Architecture & Deployments
12.1 Structured Cloud Troubleshooting Methodology 12.2 Troubleshooting Compute, Container & Storage Failures 12.3 Troubleshooting Provisioning, IaC & Automation Failures
13 Chapter 13: Troubleshooting Cloud Networking & Security
13.1 Troubleshooting Cloud Network Connectivity & Routing 13.2 Troubleshooting IAM, Permissions & Encryption 13.3 Troubleshooting Disaster Recovery & Performance Degradation
12.2 Troubleshooting Compute, Container & Storage Failures
Key Takeaways Explain This Topic AI
High CPU utilization reflects active compute processing, whereas high system load average with low CPU utilization indicates threads bottlenecked in uninterruptible I/O wait (%wa) or shared hypervisor CPU steal (%st).
Linux Out-Of-Memory (OOM) killer terminates the process with the highest oom_score (typically the main application daemon, throwing exit code 137) when physical RAM and swap space are exhausted.
Kubernetes pod failures require precise symptom mapping: CrashLoopBackOff indicates runtime application crashes, ImagePullBackOff points to registry auth or URI typos, OOMKilled signifies container memory limit breaches, and Pending stems from insufficient cluster resources or node taint/affinity mismatches.
Cloud block storage performance degrades when burst credit buckets are depleted on burstable volumes (e.g., AWS EBS gp2/gp3 or Azure Standard SSD), requiring migration to Provisioned IOPS or volume resizing.
Disk space exhaustion occurs not only from data block capacity (df -h) but also from file system inode exhaustion (df -i), which prevents new file creation even when gigabytes of disk capacity remain free.
Last updated: August 2026
Troubleshooting Compute, Container & Storage Failures
Compute and storage infrastructure forms the foundational execution layer of every cloud workload. When compute instances thrash, containers enter continuous crash loops, or block storage volumes throttle I/O operations, cloud engineers must rapidly interpret operating system metrics, hypervisor telemetry, and container runtime states to restore operational health.
1. Cloud Compute Diagnostics & Performance Anomalies
Diagnosing compute performance requires understanding the vital distinction between CPU utilization, system load average, memory pressure, and hypervisor resource contention. +---------------------------------------------------------------------------------------------------+
- | COMPUTE PERFORMANCE DIAGNOSTIC MATRIX |
- | |
- | Metric / Symptom Root Cause Analysis Primary Diagnostic Command |
- | +------------------------------+---------------------------------+-----------------------------+ |
- | | High CPU %us (User Space) | Unoptimized code, infinite loop | top / htop / pidstat -u 1 | |
- | | High CPU %sy (Kernel Space) | Excessive context switching/syscalls | vmstat 1 / strace -c | |
- | | High CPU %st (Steal Time) | Noisy neighbor / Overcommitted host | top (%st > 5%) / mpstat 1| |
- | | High Load Average + Low CPU | Disk I/O wait bottleneck (%wa) | iostat -xz 1 / dmesg | |
- | | Memory Exhaustion (Exit 137) | Linux OOM Killer invoked | dmesg -T | grep -i oom | |
- | +------------------------------+---------------------------------+-----------------------------+ |
- +---------------------------------------------------------------------------------------------------+
- High CPU Utilization vs. High Load Average
High CPU Utilization ( %us / %sy ): Indicates that virtual CPU cores are actively processing instructions. %us (user space) points to application code, while %sy (system/kernel space) points to operating system overhead such as excessive context switching or interrupt handling.
High System Load Average with Low CPU Utilization: Load average represents the average number of processes that are either in a runnable state (using or waiting for CPU) or in an uninterruptible sleep state (waiting for disk I/O, network sockets, or lock acquisition, represented as state D in ps ). If a 4-vCPU instance has a load average of 28.0 but CPU utilization is only 15%, the system is starved for disk or network I/O, not compute power. Look at the %wa (I/O wait) metric in top .
CPU Steal Time ( %st )
In multi-tenant public cloud environments, virtual machines run on physical hypervisor hosts shared with other tenants. CPU Steal ( %st ) measures the percentage of time a virtual CPU was ready to execute instructions but was forced to wait because the physical hypervisor processor was busy servicing other virtual machines (the noisy neighbor problem ).
Diagnosis: If %st consistently exceeds 3% to 5% in top or mpstat , the cloud instance is experiencing hypervisor-level CPU starvation.
Remediation: Stop and start the cloud VM to force the hypervisor scheduler to instantiate the instance on a different physical host, upgrade to a larger compute-optimized instance family (e.g., AWS c6i / Azure Fsv2 ), or migrate to Dedicated Hosts / Dedicated Instances . # Inspect real-time CPU breakdown including %us, %sy, %wa, and %st
top -b -n 1 | head -n 5
- # Check process states; look for processes in 'D' state (uninterruptible I/O sleep)
- ps aux | awk '{if ($8 ~ /D/) print $0}'
- Memory Leaks & The Linux OOM Killer
When an application continuously allocates memory without freeing unreferenced objects (a memory leak ), available physical RAM and swap space dwindle to zero. When the kernel cannot fulfill an allocation request, the Out-Of-Memory (OOM) Killer is invoked.
Mechanism: The kernel calculates an oom_score for each running process based on memory consumption and oom_score_adj . It terminates the highest-scoring process using a SIGKILL (signal 9) to protect kernel stability.
Linux Exit Code 137: When a process or container is terminated by the OOM killer, it exits with code 137 ( 128 + signal 9 (SIGKILL) ).
Triage: Inspect kernel ring buffer logs using dmesg -T | grep -i oom or journalctl -k to identify which process was killed and its memory footprint at termination.
2. Container & Kubernetes Workload Troubleshooting
Container orchestration platforms abstract compute nodes into a unified cluster, but introduce specialized workload lifecycle states that indicate specific configuration or resource failures. +---------------------------------------------------------------------------------------------------+
- | KUBERNETES POD FAILURE TAXONOMY |
- | |
- | Pod Status Common Underlying Root Causes Resolution Workflow |
- | +----------------------+-----------------------------------------+----------------------------+ |
- | | CrashLoopBackOff | Application runtime bug, missing env | kubectl logs --previous | |
- | | | variables, failed health/liveness probe | kubectl describe pod <name>| |
- | | | | | |
- | | ImagePullBackOff / | Typo in image repository/tag, missing | Verify secret in manifest; | |
- | | ErrImagePull | imagePullSecrets, registry rate limits | check docker auth token | |
- | | | | | |
- | | OOMKilled (Exit 137) | Container exceeded resources.limits.mem | Increase memory limit in | |
- | | | or host node reached memory pressure | deployment spec; fix leak | |
- | | | | | |
- | | Pending | Insufficient CPU/RAM requests on nodes, | Add cluster nodes; remove | |
- | | | unmatched nodeSelectors, taints/affinity| un-tolerated node taints | |
- | +----------------------+-----------------------------------------+----------------------------+ |
- +---------------------------------------------------------------------------------------------------+
- CrashLoopBackOff
A pod in CrashLoopBackOff has been scheduled to a node, but the application container repeatedly starts, fails, and crashes. Kubernetes enforces an exponential backoff delay (10s, 20s, 40s... up to 5m) before restarting the container.
Diagnostic Steps:
Run kubectl describe pod <pod-name> and examine the Last State and Exit Code fields.
Run kubectl logs <pod-name> --previous to read stdout/stderr output from the crashed container instance before its restart.
Verify whether Liveness Probes or Startup Probes are failing due to aggressive initial delay timeouts before the application has finished warming up.
ImagePullBackOff & ErrImagePull
Indicates the kubelet on the worker node cannot fetch the container image from the container registry (e.g., Amazon ECR, Azure Container Registry, Docker Hub, Google Artifact Registry).
Root Causes:
Typographical error in the image URL or tag (e.g., app:v1.2 instead of app:v1.2.0 ).
Missing or misconfigured imagePullSecrets required to authenticate against a private registry.
Worker node subnet lacks a NAT Gateway or VPC Endpoint to route traffic to the registry.
Anonymous pull rate limit exceeded (e.g., Docker Hub HTTP 429 Too Many Requests).
OOMKilled vs. Node Memory Pressure
Container OOMKilled: If a container exceeds the memory threshold defined in resources.limits.memory in its pod specification, the container runtime (containerd/CRI-O) sends a SIGKILL specifically to that container. The pod restarts, and kubectl describe pod displays Reason: OOMKilled (Exit Code: 137) .
Node Eviction (NodeMemoryPressure): If the worker node itself runs out of physical memory, the kubelet begins evicting pods according to their Quality of Service (QoS) Class (BestEffort pods are evicted first, followed by Burstable, and finally Guaranteed pods).
Pending Pods
A pod remains in Pending state when the Kubernetes scheduler cannot find any worker node that satisfies the pod's scheduling requirements.
Root Causes:
Resource Starvation: Total cluster resources.requests.cpu or resources.requests.memory are fully allocated.
Taints and Tolerations: Worker nodes have taints (e.g., node.kubernetes.io/unreachable or dedicated GPU taints sku=gpu:NoSchedule ) that the pod does not have matching tolerations for.
Node Affinity / NodeSelector: The pod requires a node label (e.g., topology.kubernetes.io/zone: us-east-1a ) that matches zero available nodes.
Unbound PersistentVolumeClaim (PVC): The pod requires a storage volume that has not yet been provisioned or is in a different Availability Zone. # Inspect Kubernetes scheduling failures and event log
kubectl describe pod billing-service-78f9d-xkz42 | grep -A 10 Events:
- # Check resource capacity and allocation across all cluster nodes
- kubectl describe nodes | grep -A 5 "Allocated resources:"
- 3. Cloud Storage Troubleshooting & Bottlenecks
Cloud storage issues manifest across three primary failure domains: throughput/IOPS throttling, volume attachment deadlocks, and file system capacity exhaustion. +---------------------------------------------------------------------------------------------------+
- | CLOUD STORAGE FAILURE DOMAINS |
- | |
- | [ 1. IOPS & Throughput Throttling ] |
- | ├── Burstable storage burst credit depletion (AWS gp2/gp3 burst balance = 0%) |
- | ├── Symptoms: High average queue length (aqu-sz), massive write latency, database timeouts |
- | └── Fix: Resize volume for higher baseline IOPS, convert to Provisioned IOPS (io2/Ultra Disk) |
- | |
- | [ 2. Storage Capacity & Inode Exhaustion ] |
- | ├── Block Exhaustion: df -h shows 100% disk usage (large logs, core dumps) |
- | ├── Inode Exhaustion: df -i shows 100% inode usage (millions of zero-byte session/temp files) |
- | └── Unlinked Open Files: lsof +L1 shows deleted files held open by running processes |
- | |
- | [ 3. Volume Attachment & Multi-Attach Deadlocks ] |
- | ├── EBS volume stuck in 'attaching' or 'detaching' state (hypervisor lock / host crash) |
- | ├── Multi-attach file system corruption: mounting standard ext4/XFS read-write on multiple VMs |
- | └── NFS Stale File Handle (ESTALE): File deleted on NFS server while client holds open handle |
- +---------------------------------------------------------------------------------------------------+
- IOPS and Throughput Throttling (Burst Credit Depletion)
Many cloud block storage tiers (e.g., AWS EBS gp2 , Azure Standard SSD) use a token-bucket algorithm for I/O bursting. When I/O demand exceeds the baseline allowance, the volume consumes burst credits. When burst credits are exhausted:
The volume is throttled down to its baseline performance (e.g., 3 IOPS per GB on older gp2 volumes).
Operating system disk queue depth ( aqu-sz in iostat ) spikes dramatically.
Write latency increases from sub-millisecond levels to hundreds of milliseconds, triggering application database connection pool exhaustion.
Resolution: Migrate volumes to Provisioned IOPS (e.g., AWS EBS io2 Block Express , Azure Ultra Disk) or decouple write-intensive operations to ephemeral NVMe instance store volumes. # Check extended I/O statistics; look for high await (>20ms) and aqu-sz (queue depth > 2)
- iostat -xz 1 5
- Disk Space Exhaustion: Block Storage vs. Inodes
A frequent CompTIA exam trap involves understanding that a file system can run out of space in two distinct ways:
Data Block Exhaustion ( df -h ): The physical or virtual gigabytes are 100% full. Common culprits include unrotated application log files, database transaction logs, and core dumps.
Inode Exhaustion ( df -i ): Inodes store file metadata (permissions, owner, block pointers). Every file and directory consumes exactly one inode regardless of file size. If an application generates millions of tiny 1-byte session files or temporary lock files, 100% of available inodes can be consumed even while df -h shows 80% free disk space . When inodes are exhausted, the operating system returns No space left on device upon any new file creation attempt.
Deleted Open File Descriptors ( lsof +L1 ): When a log file is deleted with rm while a running daemon (e.g., Nginx or MySQL) still holds an open file descriptor to it, the disk blocks are not released back to the OS until the daemon is restarted or reloaded. # Check block capacity
df -h
- # Check inode capacity
- df -i
- # Find processes holding onto deleted unlinked files that are consuming disk space
- lsof +L1
- Volume Attachment & Multi-Attach Constraints
Stuck Volume Attachments: If a hypervisor crashes while detaching an elastic block volume, the volume may remain locked in an attaching or detaching state. Cloud engineers must use cloud CLI force-detach APIs or stop the parent virtual machine.
Availability Zone Boundaries: Cloud block storage volumes (AWS EBS, Azure Managed Disks) are physical constructs bound to a single Availability Zone . A volume created in us-east-1a cannot be attached to a virtual machine in us-east-1b .
Multi-Attach Limitations: Standard cloud block volumes do not support simultaneous read-write attachment to multiple virtual machines using standard file systems like ext4 , XFS , or NTFS . Doing so causes instant file system corruption because standard file systems lack distributed cache-coherency lock managers. Multi-attach block volumes require specialized clustered file systems (e.g., GFS2, OCFS2). For standard multi-node shared storage, use managed Network Attached Storage ( AWS EFS , Azure Files , Google Cloud Filestore ) over NFS/SMB.
NFS Stale File Handle ( ESTALE ): Occurs on shared file systems when a client attempts to read/write a file using an open file handle, but the file was deleted or replaced on the underlying NFS server by another client or automated batch job. The client must unmount and remount the export or restart the client process.