Study guide
6.1 Systematic Kubernetes Troubleshooting Methodology
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
6.1 Systematic Kubernetes Troubleshooting Methodology
Key Takeaways Explain This Topic AI
Kubernetes troubleshooting demands a top-down and bottom-up 4-tier triage model: Infrastructure & Node Layer, Control Plane & Core Daemons, Workload & Pod Lifecycle, and Service/Ingress Networking & Storage.
Diagnostic velocity relies on an established CLI tool hierarchy: triage with 'kubectl get/describe', drill down with 'kubectl logs' and 'kubectl exec', and drop into host-level 'crictl' and 'journalctl' when the control plane is unreachable.
Cluster event streams ('kubectl get events -A --sort-by=.metadata.creationTimestamp') reveal transient failures, scheduling rejections, image pull timeouts, and probe failures before they escalate.
Effective root-cause isolation requires separating the Control Plane (API server, etcd, scheduler, controllers) from the Data Plane (kubelet, container runtime, CNI, kube-proxy).
During time-constrained performance exams like the CKA, strict adherence to timeboxing (5–7 minutes per task) and systematic verification prevents compounding configuration errors.
Last updated: August 2026
6.1 Systematic Kubernetes Troubleshooting Methodology
Troubleshooting in Kubernetes is often perceived as daunting due to the distributed, multi-layered architecture of the platform. A single symptom—such as a web application returning an HTTP 502 Bad Gateway—can stem from a failure in physical networking, a corrupted CoreDNS configuration, an unhealthy Pod readiness probe, a saturated cgroup memory limit, an iptables forwarding bug in kube-proxy, or a control plane certificate expiration.
Without a structured diagnostic methodology, administrators waste critical time guessing at solutions. For the Certified Kubernetes Administrator (CKA) examination and enterprise operations, mastering a systematic, deterministic troubleshooting hierarchy is essential.
1. The 4-Tier Kubernetes Diagnostic Model
To rapidly isolate the root cause of an outage, divide the cluster into four distinct operational layers and evaluate them hierarchically: +-----------------------------------------------------------------------------------------+
- | THE 4-TIER KUBERNETES DIAGNOSTIC MODEL |
- | |
- | [TIER 4: APPLICATION & WORKLOAD LAYER] |
- | - Pod Phase (Pending, Running, CrashLoopBackOff, OOMKilled, Completed) |
- | - Container Exit Codes (0, 1, 137, 139, 143), Logs (stdout/stderr), Probe Failures |
- | - Manifest Syntax, Environment Variables, ConfigMaps, Secrets, Volume Mounts |
- | ^ |
- | | |
- | [TIER 3: NETWORKING & STORAGE SERVICES LAYER] |
- | - ClusterIP, NodePort, LoadBalancer Services & Endpoints / EndpointSlices |
- | - Ingress Controllers & Routing Rules, CoreDNS Resolution (/etc/resolv.conf) |
- | - PersistentVolumes, PersistentVolumeClaims, StorageClasses, CSI Attachments |
- | ^ |
- | | |
- | [TIER 2: NODE AGENT & DATA PLANE RUNTIME] |
- | - Kubelet Systemd Daemon & Configuration (/var/lib/kubelet/config.yaml) |
- | - Container Runtime (containerd, CRI-O, crictl ps/logs), CNI Plugins (/etc/cni/net.d) |
- | - Host Resources (DiskPressure, MemoryPressure, PIDPressure, Kernel OOM Reaper) |
- | ^ |
- | | |
- | [TIER 1: CONTROL PLANE & CLUSTER INFRASTRUCTURE] |
- | - kube-apiserver, etcd Database & Quorum, kube-controller-manager, kube-scheduler |
- | - Static Pod Manifests (/etc/kubernetes/manifests), TLS PKI Certs (/etc/kubernetes/pki)|
- | - Administrative Kubeconfig Files (/etc/kubernetes/admin.conf, ~/.kube/config) |
- +-----------------------------------------------------------------------------------------+
Triage Flow Rules:
Bottom-Up for Infrastructure Failures: If kubectl commands fail entirely ( The connection to the server <host>:6443 was refused ), start immediately at Tier 1 (Control Plane & Infrastructure).
Top-Down for Workload Failures: If kubectl works but a deployment is not serving traffic, start at Tier 4 (Workload), verify Tier 3 (Service/DNS), and drop down to Tier 2 (Node/Kubelet) only if container-level inspection indicates host-level resource exhaustion or network isolation.
2. The Diagnostic Toolchain Hierarchy
Effective administrators transition seamlessly between different abstraction levels depending on where the fault resides:
Level Primary Tools Typical Commands When to Use
Level 1: Cluster & API Scope kubectl kubectl get nodes -o wide ; kubectl get events -A --sort-by=.metadata.creationTimestamp ; kubectl cluster-info Initial situational awareness; identifying failing nodes, pending pods, and recent cluster-wide warning events.
Level 2: Resource Inspection kubectl kubectl describe <resource> <name> ; kubectl get <resource> <name> -o yaml Deep inspection of resource state, status conditions, controller events, selector mismatches, and spec definitions.
Level 3: Container Log & Shell kubectl kubectl logs <pod> -c <container> --previous ; kubectl exec -it <pod> -- /bin/sh ; kubectl debug -it <pod> Inspecting application runtime output, checking environment variables, testing local loopback networking, and examining filesystems.
Level 4: Host Runtime (Node SSH) crictl , nerdctl crictl pods ; crictl ps -a ; crictl logs <container-id> ; crictl inspect <container-id> Direct container runtime debugging when kubelet cannot report to the API server or when static pods are failing.
Level 5: Host Systemd & OS systemctl , journalctl , ip , ss systemctl status kubelet containerd ; journalctl -u kubelet -e --no-pager ; dmesg -T | grep -i oom ; ss -tulpn Low-level OS, daemon failure, kernel cgroup kills, memory/swap pressure, port conflicts, and disk exhaustion diagnostics.
3. High-Yield Cluster Event Analysis
The Kubernetes Event subsystem records state transitions, errors, and controller decisions across the cluster. Because events are namespaced and have a default retention period of only 1 hour, querying them chronologically is one of the fastest diagnostic shortcuts: # Stream all cluster-wide events sorted chronologically
kubectl get events -A --sort-by='.metadata.creationTimestamp'
- # Filter specifically for Warning and Error events across all namespaces
- kubectl get events -A --field-selector type=Warning --sort-by='.metadata.creationTimestamp'
- # Inspect events for a specific namespace in real time
- kubectl get events -n production --watch
Common Event Reason Codes & Root Causes:
Event Reason Message Pattern Root Cause / Immediate Action
FailedScheduling 0/3 nodes available: 3 Insufficient memory Total requested memory exceeds node allocatable capacity; adjust requests or add nodes.
FailedScheduling 0/3 nodes available: 3 node(s) had untolerated taint Workload lacks required toleration for master or custom tainted worker nodes.
FailedMount MountVolume.SetUp failed: secret 'db-creds' not found Referenced Secret or ConfigMap does not exist in the workload's namespace.
FailedAttachVolume Multi-Attach error for volume "pvc-xxxx" ReadWriteOnce volume is still attached to a previous node that crashed or has not released the lock.
BackOff Back-off restarting failed container Application crashed on startup; inspect kubectl logs <pod> --previous .
Unhealthy Liveness probe failed: HTTP probe failed with statuscode: 500 Application internal healthcheck failed; container will be restarted by kubelet.
4. Failure Domain Isolation Heuristic
When presented with a broken cluster or workload, follow this 5-step elimination algorithm: [STEP 1: Check API Server Reachability]
$ kubectl get nodes
├── SUCCESS: Control plane is alive -> Proceed to Step 2.
└── FAILURE: Error connection refused / unauthorized
└── Action: SSH to Control Plane Node -> Check kubelet, static pod manifests, and PKI certs.
- [STEP 2: Check Node Statuses]
- $ kubectl get nodes -o wide
├── All Ready -> Proceed to Step 3.
└── Node NotReady
└── Action: $ kubectl describe node <node> -> SSH to node -> systemctl status kubelet containerd.
- [STEP 3: Check Core Add-on Health]
- $ kubectl get pods -n kube-system
├── CoreDNS / CNI (Calico/Flannel) / kube-proxy Running -> Proceed to Step 4.
└── Core Add-ons Crashing / Pending
└── Action: Inspect CNI logs, CoreDNS ConfigMap, and network routing.
- [STEP 4: Check Target Workload Status]
- $ kubectl get pods -n <ns> -o wide
├── Pending -> Check scheduler events (Taints, Affinities, Requests, PVs).
├── ContainerCreating / CrashLoopBackOff / OOMKilled -> Check describe, logs --previous, probes.
└── Running (but traffic failing) -> Proceed to Step 5.
- [STEP 5: Check Networking & Endpoint Plumbing]
- $ kubectl get svc,ep,endpointslices -n <ns>
└── Endpoints <none> -> Label selector mismatch or Readiness probe failing.
└── Endpoints populated -> Check targetPort, Ingress routing, and NetworkPolicies.
Important
CKA Exam Triage Rules:
Always execute kubectl config use-context <context-name> before running any commands.
Check the namespace! If a resource is not found, verify if it was created in default instead of the requested namespace.
If you edit a manifest and kubectl apply errors due to immutable field restrictions (e.g., changing pod selectors on an existing Deployment), delete the resource with kubectl delete -f file.yaml --force --grace-period=0 and recreate it.