Study guide
6.2 Node Failure, Kubelet Diagnostics & Systemd Services
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
6.2 Node Failure, Kubelet Diagnostics & Systemd Services
Key Takeaways Explain This Topic AI
Nodes report 'NotReady' when the kubelet stops updating its NodeLease in the kube-node-lease namespace, typically caused by a crashed kubelet systemd service, container runtime failure, or network partition.
Kubelet operates as a native systemd service; diagnosing node failures requires inspecting 'systemctl status kubelet' and reading service journal logs via 'journalctl -u kubelet -e --no-pager'.
Common kubelet failure modes include runtime socket errors, default swap rejection when swap was not configured, invalid kubelet YAML, expired or failed client-certificate rotation, and missing CNI configuration or binaries.
Under node pressure, kubelet ranks Pods by whether usage exceeds requests, then Pod Priority, then usage relative to requests; QoS class alone is not a strict eviction order.
Safe node maintenance requires a strict sequence: cordoning ('kubectl cordon <node>') to prevent new scheduling, draining ('kubectl drain <node> --ignore-daemonsets --delete-emptydir-data') to evict workloads safely, performing maintenance, and uncordoning ('kubectl uncordon <node>').
Last updated: August 2026
6.2 Node Failure, Kubelet Diagnostics & Systemd Services
Worker nodes provide the compute, memory, storage, and networking capacity for Kubernetes workloads. When a node transitions to the NotReady state or begins evicting pods due to resource pressure, the cluster's capacity degrades and workloads risk downtime.
Because the kubelet runs directly on the Linux operating system as a native systemd daemon rather than a containerized workload, troubleshooting node failures requires fluency in Linux system administration, systemd service management, journal log inspection, and host-level container runtime diagnostics.
1. Node Status & Health Condition Lifecycle
The Kubernetes control plane monitors worker node health through the Node Lifecycle Controller inside kube-controller-manager . The node's kubelet regularly renews a Lease object in the kube-node-lease namespace (by default every 10 seconds). +-----------------------------------------------------------------------------------------+
- | NODE HEARTBEAT & LEASE RENEWAL TIMELINE |
- | |
- | Kubelet (Node) ---> Renew Lease (kube-node-lease/node01) ---> kube-apiserver / etcd |
- | [Every 10 Seconds: node-status-update-frequency] |
- | |
- | IF HEARTBEAT FAILS: |
- | - T = 0s: Heartbeat stops (Kubelet crash / host network partition / power failure) |
- | - T = 40s: Grace period expires (node-monitor-grace-period) |
- | Controller marks Node condition: Ready = Unknown / NotReady |
- | NodeController applies taints: |
- | - node.kubernetes.io/unreachable:NoSchedule |
- | - node.kubernetes.io/unreachable:NoExecute |
- | - T = 300s: default not-ready/unreachable NoExecute toleration expires |
- | Pods on failed node transitioned to 'Terminating' |
- | ReplicaSet controller spawns replacement pods on surviving healthy nodes |
- +-----------------------------------------------------------------------------------------+
Node Status Conditions:
Execute kubectl describe node <node-name> and inspect the Conditions block:
Condition Healthy Value Failure Indicator / Root Cause
Ready True False (Kubelet unhealthy) or Unknown (Kubelet stopped posting heartbeats).
MemoryPressure False True (Node available memory dropped below eviction threshold, e.g., < 100Mi ).
DiskPressure False True (Node root filesystem or container image filesystem free space < 10% ).
PIDPressure False True (Available Linux Process IDs on host dropped below threshold, e.g., fork-bomb).
NetworkUnavailable False True (CNI plugin route is not configured or network bridge is down).
2. Kubelet Diagnostic & Troubleshooting Runbook
When a node reports NotReady , SSH directly into the affected host to perform targeted systemd diagnostics. # 1. SSH into the failing worker node
- ssh node01
- sudo -i
- # 2. Check the systemd service status
- systemctl status kubelet
- # 3. If kubelet is inactive (dead) or activating (auto-restart loop), inspect logs
- journalctl -u kubelet -e --no-pager -n 100
Common Kubelet Failure Modes and Solutions:
Failure Mode A: Swap is Enabled on the Host
By default, kubelet refuses to start if swap memory is active on the Linux host (unless kubelet swap support is explicitly configured with failSwapOn: false and an appropriate memorySwap.swapBehavior ): Error: failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false
- Remediation: # Temporarily disable swap immediately
- swapoff -a
- # Permanently disable swap in /etc/fstab by commenting out the swap partition
- cp /etc/fstab /etc/fstab.pre-kubelet
- sed -i '/[[:space:]]swap[[:space:]]/ s/^/#/' /etc/fstab
- # Restart kubelet
- systemctl restart kubelet
- Failure Mode B: Container Runtime Socket Path Misconfigured or Runtime Stopped
If containerd or CRI-O is inactive or crashed, kubelet cannot establish its gRPC connection over the UNIX domain socket: Error: failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unavailable desc = connection error
- Remediation: # Check container runtime daemon status
- systemctl status containerd
- # If inactive, restart containerd and verify the socket is present
- systemctl daemon-reload
- systemctl restart containerd
- systemctl enable containerd
- ls -la /run/containerd/containerd.sock
- # Restart kubelet
- systemctl restart kubelet
- Failure Mode C: Syntax or Parameter Error in Kubelet Configuration
The kubelet daemon reads its configuration parameters from /var/lib/kubelet/config.yaml . Invalid YAML indentation or deprecated flags will cause immediate startup failure: # Validate YAML syntax
cat /var/lib/kubelet/config.yaml
- # Check systemd drop-in configuration overrides
- cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
- Failure Mode D: Expired Kubelet Client Certificates
Worker node kubelets authenticate to the API server using client certificates located at /var/lib/kubelet/pki/kubelet-client-current.pem . If automatic certificate rotation fails or certificates expire: # Check certificate expiration date
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates
- # Inspect rotation and pending CSRs before changing credentials
- readlink -f /var/lib/kubelet/pki/kubelet-client-current.pem
- journalctl -u kubelet -n 100 --no-pager
- kubectl get csr
# Do not delete client keys blindly. Back them up, then use the cluster's
# documented kubeadm TLS-bootstrap or node rejoin procedure with a valid token.
Failure Mode E: CNI Network Plugin Directory or Configuration Missing
If /etc/cni/net.d/ is empty or CNI binary plugins are missing from /opt/cni/bin/ , the node reports NetworkUnavailable=True and Ready=False with the message cni plugin not initialized . # Check CNI config directory
ls -la /etc/cni/net.d/
- # Check CNI binaries
- ls -la /opt/cni/bin/
- 3. Node-Pressure Signals and Eviction Ranking
When a node crosses a configured memory, filesystem, inode, or PID threshold, kubelet first attempts node-level reclamation such as removing dead containers or unused images where applicable. If the signal remains under pressure, kubelet evicts Pods.
For the starved resource, kubelet ranks candidates by:
- whether the Pod's usage exceeds its request;
- Pod Priority; and
usage relative to the request.
QoS is useful context because BestEffort Pods have zero requests and Guaranteed Pods have requests equal to limits, but kubelet does not simply sort all Pods by the label BestEffort, Burstable, Guaranteed. For example, among Pods over their requests, a lower-priority Burstable Pod can be evicted before a higher-priority BestEffort Pod. Inspect kubectl describe node , Pod events, PriorityClasses, requests, and kubelet logs.
Common Linux hard thresholds include memory.available<100Mi , nodefs.available<10% , imagefs.available<15% , and nodefs.inodesFree<5% ; node configuration can override them. Soft thresholds add a grace period.
- 4. Safe Node Lifecycle---
- 4. Safe Node Lifecycle & Maintenance Procedures
Performing kernel patches, operating system upgrades, or hardware maintenance on a worker node requires a strict cordon-drain-uncordon workflow to prevent workload disruption. +-----------------------------------------------------------------------------------------+
- | SAFE NODE MAINTENANCE EXECUTION LIFECYCLE |
- | |
- | 1. CORDON NODE |
- | $ kubectl cordon node01 |
- | - Marks node Unschedulable (applies node.kubernetes.io/unschedulable taint) |
- | - Existing running pods remain untouched; no new pods can be scheduled. |
- | | |
- | v |
- | 2. DRAIN NODE |
- | $ kubectl drain node01 --ignore-daemonsets --delete-emptydir-data --force |
- | - Evicts all standalone and controller-managed pods gracefully. |
- | - DaemonSet pods are ignored (they cannot be evicted to other nodes). |
- | - Warning: Pods using emptyDir will lose local ephemeral data. |
- | | |
- | v |
- | 3. PERFORM MAINTENANCE (SSH to node01) |
- | - Upgrade packages, patch kernel, reboot: $ sudo reboot |
- | - Verify daemons after boot: $ systemctl status containerd kubelet |
- | | |
- | v |
- | 4. UNCORDON NODE |
- | $ kubectl uncordon node01 |
- | - Removes Unschedulable flag; node returns to active scheduling pool. |
- +-----------------------------------------------------------------------------------------+
Caution
Critical Drain Flags:
--ignore-daemonsets : Mandatory if any DaemonSet (e.g., kube-proxy , calico-node , prom-node-exporter ) runs on the node; otherwise kubectl drain will immediately abort.
--delete-emptydir-data : Required if any pod on the node mounts an emptyDir volume; otherwise drain aborts to prevent unacknowledged data loss.
--force : Bypasses standalone pods that are not managed by a Deployment, StatefulSet, or ReplicaSet (Note: standalone pods will be permanently deleted and not rescheduled).