ExamFinch.com My Study

Study guide

10.2 Cluster & Node-Level Troubleshooting

Practice this exam

Kubernetes and Cloud Native Associate KCNA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Introduction & Exam Overview

1.1 About the KCNA Exam & Certification Path 1.2 How to Use This Study Guide

2 Chapter 2: Kubernetes Fundamentals - Cluster Architecture & the API

2.1 Kubernetes Architecture & Control Plane Components 2.2 Worker Node Architecture & Node Components 2.3 Kubernetes API, Declarative Management & kubectl 2.4 Kubernetes Releases, API Versioning & Deprecation Policy

3 Chapter 3: Kubernetes Fundamentals - Workloads & Configuration

3.1 Core Workload Objects (Pods, Deployments & ReplicaSets) 3.2 Advanced Controllers (StatefulSets, DaemonSets, Jobs & CronJobs) 3.3 Namespaces, Labels, Selectors & Annotations 3.4 ConfigMaps, Secrets & Application Configuration 3.5 Pod & Container Lifecycle, Restart Policies & Graceful Termination

4 Chapter 4: Kubernetes Fundamentals - Scheduling & Autoscaling

4.1 Scheduling, Resource Management & Node Selection 4.2 Advanced Scheduling: Affinity, Topology Spread, Priority & Disruption Budgets 4.3 Autoscaling Workloads & Clusters (HPA, VPA, Cluster Autoscaler, KEDA)

5 Chapter 5: Kubernetes Fundamentals - Cluster Administration & Extensibility

5.1 Kubernetes Distributions, Installation Tools & Managed Services 5.2 Cluster Administration: Node Maintenance, Upgrades & etcd Backup 5.3 Extending Kubernetes: CRDs, Operators & the Controller Pattern

6 Chapter 6: Kubernetes Fundamentals - Containerization

6.1 Container Fundamentals, Runtimes & Registries 6.2 Container Image Anatomy: Layers, Tags, Digests & Pull Policy

7 Chapter 7: Container Orchestration - Networking & Service Discovery

7.1 Kubernetes Networking, Services & Ingress 7.2 Cluster DNS & Service Discovery with CoreDNS 7.3 NetworkPolicies, Zero-Trust Segmentation & the Gateway API

8 Chapter 8: Container Orchestration - Storage

8.1 Storage Concepts (PVs, PVCs, StorageClasses & CSI) 8.2 Volume Types, Snapshots & Stateful Storage Operations

9 Chapter 9: Container Orchestration - Security

9.1 Kubernetes Security & Access Control (RBAC & ServiceAccounts) 9.2 The 4C's of Cloud Native Security & Supply-Chain Trust 9.3 Service Mesh, mTLS & Zero-Trust Service Identity

10 Chapter 10: Container Orchestration - Troubleshooting

10.1 Container Troubleshooting & Health Probes 10.2 Cluster & Node-Level Troubleshooting

11 Chapter 11: Cloud Native Application Delivery

11.1 Cloud Native CI/CD Pipelines & Build Tooling 11.2 GitOps & Package Management (Helm, Kustomize, Argo CD & Flux) 11.3 Deployment Strategies & Release Management 11.4 Progressive Delivery Automation: Argo Rollouts, Flagger & Feature Flags 11.5 Application Debugging & Developer Workflows in Kubernetes 11.6 Platform Engineering & Developer Self-Service

12 Chapter 12: Cloud Native Architecture - Observability & Principles

12.1 Cloud Native Observability (Logging, Metrics & Tracing) 12.2 Cloud Native Principles & Architecture Patterns 12.3 Serverless, FaaS & Event-Driven Architecture

13 Chapter 13: Cloud Native Architecture - CNCF Ecosystem & Community

13.1 The CNCF Landscape, Governance & Project Lifecycle 13.2 Cloud Native Community, Collaboration & Contribution

10.2 Cluster & Node-Level Troubleshooting

Key Takeaways Explain This Topic AI

Triage top-down: confirm the control plane answers, then node health, then networking and DNS, then the workload — most reported Pod problems are actually node or cluster problems.

A node reports NotReady when the kubelet stops sending heartbeats, and after a grace period the node controller marks its Pods for eviction.

MemoryPressure and DiskPressure trigger kubelet eviction, which removes BestEffort Pods first, then Burstable, then Guaranteed.

Events are the highest-value diagnostic surface in Kubernetes but expire after roughly one hour by default, so they must be shipped somewhere durable.

A Pending Pod is a scheduling problem, ImagePullBackOff is a registry or credential problem, and CrashLoopBackOff is an application problem — the phase names the layer.

Last updated: August 2026

10.2 Cluster & Node-Level Troubleshooting

Quick Answer: Troubleshoot top-down . Confirm the control plane answers, then node health , then networking and DNS , then the workload . A large share of incidents reported as "my Pod is broken" are in fact a NotReady node, an exhausted disk, or a CNI failure. The state a Pod is stuck in names the layer at fault: Pending is scheduling, ImagePullBackOff is registry or credentials, CrashLoopBackOff is the application.

Section 10.1 covered probes and container-level debugging. This section covers everything above the Pod.

  • 1. A Triage Order That Works 1. CONTROL PLANE kubectl get --raw='/readyz?verbose'
  • kubectl get componentstatuses (legacy but quick)

↓ answers?

  • 2. NODES kubectl get nodes -o wide
  • kubectl describe node <n> ← conditions + events + allocated resources

↓ all Ready?

3. CLUSTER ADD-ONS kubectl get pods -n kube-system ← CoreDNS, CNI, kube-proxy

↓ all Running?

  • 4. NAMESPACE kubectl get events -n <ns> --sort-by=.lastTimestamp
  • kubectl get pods -n <ns> -o wide

↓ narrowed to one workload?

5. WORKLOAD kubectl describe pod / logs --previous / exec (section 10.1)

Running step 5 first is the classic mistake: you read an application stack trace for twenty minutes before noticing the node it sits on has a full disk.

2. Node Conditions and What They Mean

kubectl describe node reports a condition block that is the fastest read in the whole system:

Condition True means Consequence

Ready The kubelet is healthy and accepting Pods False / Unknown ⇒ node controller begins evicting after a grace period (~5 minutes by default)

MemoryPressure Available memory has fallen below the eviction threshold Kubelet evicts Pods and taints the node node.kubernetes.io/memory-pressure

DiskPressure Root or image filesystem is critically low Kubelet garbage-collects images and evicts Pods; new image pulls fail

  • PIDPressure Host process IDs are near exhaustion New processes and containers fail to start
  • NetworkUnavailable The node's network route is not correctly configured Usually a CNI plugin fault
  • NotReady — What Actually Happened

Ready: Unknown means the kubelet stopped renewing its Lease in kube-node-lease . Common causes, in rough order of frequency:

The kubelet process crashed or is wedged — systemctl status kubelet , journalctl -u kubelet .

The container runtime (containerd/CRI-O) is down — the kubelet cannot report container status.

The CNI plugin is failing — the node cannot satisfy the Kubernetes networking model.

Network partition between node and control plane.

Resource exhaustion on the node making the kubelet unresponsive.

Expired certificates — the kubelet's client certificate lapsed and it can no longer authenticate.

When the node controller declares a node lost, Pods managed by controllers are recreated elsewhere; naked Pods and StatefulSet Pods with attached volumes are the slow cases , because a volume must be detached before it can be attached elsewhere.

3. Eviction Under Pressure

When a node runs low on memory or disk, the kubelet reclaims resources before the kernel OOM killer does something less discriminating. The eviction order follows QoS class , which is where section 4.1's material becomes operational: Evicted first ─────────────────────────────────────► Evicted last

  • BestEffort Burstable (over its request) Guaranteed
  • (no requests) (requests < limits) (requests == limits)

Distinguish two lookalike failures:

Signal Meaning

Pod status Evicted , node event NodeHasDiskPressure / NodeHasInsufficientMemory Node-level eviction — the node ran out, not the container

Container reason OOMKilled , exit code 137 Container-level — that container exceeded its own memory limit

The fix differs completely: eviction means the node is under-provisioned or workloads lack requests; OOMKilled means one container's limit is too low or it leaks.

  • 4. Events Are the Highest-Value Surface kubectl get events -A --sort-by=.lastTimestamp | tail -40
  • kubectl get events -n production --field-selector type=Warning

Events explain why the control plane did what it did — FailedScheduling with the exact predicate that failed, FailedMount , Unhealthy probe failures, NodeNotReady , Preempted .

The trap: Events are stored in etcd with a default TTL of about one hour . An incident investigated the next morning has no events left. Shipping events to a durable store (an events exporter into Loki or Elasticsearch) is standard practice in any cluster you have to operate.

5. Reading a Pending Pod

Pending always means "not yet placed or not yet started", and kubectl describe pod names the reason precisely:

Event message Cause Fix

Insufficient cpu / Insufficient memory No node has enough allocatable capacity for the requests Lower requests, add nodes, or enable the Cluster Autoscaler

node(s) had untolerated taint … Every candidate node is tainted Add a toleration or untaint

node(s) didn't match Pod's node affinity/selector No node carries the required labels Fix the selector or label a node

node(s) didn't find available persistent volumes to bind No PV matches the PVC Check the StorageClass, capacity, and access mode

node(s) had volume node affinity conflict The volume is in a different zone from the candidate nodes Use WaitForFirstConsumer binding

n node(s) didn't match pod topology spread constraints The placement would breach maxSkew Add capacity in the under-filled domain

Allocatable, not capacity. A node's schedulable resources are allocatable — capacity minus kube-reserved, system-reserved, and eviction thresholds. A "16-core" node does not offer 16 cores to workloads, which is why a Pod requesting 16 CPUs never schedules on it.

6. Control Plane and Add-On Failures

Symptom Likely component

Every kubectl command times out or is refused kube-apiserver down, load balancer misrouting, or expired client certificate

  • API responds but nothing is ever scheduled kube-scheduler not running or has lost leader election
  • Deleted objects linger; ReplicaSets do not reconcile kube-controller-manager down
  • Writes fail, reads work, etcdserver: request timed out etcd lost quorum — check member health and disk latency
  • Nothing resolves anywhere CoreDNS Pods not Ready, or CNI failure beneath them
  • Services have endpoints but traffic never arrives kube-proxy failing on that node
  • LoadBalancer Service stuck <pending> cloud-controller-manager down, or no cloud LB integration

On a kubeadm cluster the control plane runs as static Pods , so it can be inspected with kubectl -n kube-system logs kube-apiserver-<node> — or, when the API is unreachable, directly on the node with crictl ps and crictl logs , which talk to the container runtime without going through Kubernetes at all.

  • 7. Node-Level Commands
  • When kubectl cannot tell you enough, the answers are on the node: systemctl status kubelet
  • journalctl -u kubelet -n 200 --no-pager
  • crictl ps -a # containers, independent of the API server
  • crictl logs <container-id>
  • df -h /var/lib/containerd /var/log # DiskPressure is nearly always here
  • free -h

crictl is the CRI-level equivalent of docker ps , and it is the tool that still works when the API server is the thing that is broken.