ExamFinch.com My Study

Study guide

6.3 Control Plane Failures, Static Pods & Component Logs

Practice this exam

Certified Kubernetes Administrator CKA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CKA Exam Environment & Cluster Architecture

1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures

2 Chapter 2: Cluster Extensions, Packaging & RBAC

2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers

3 Chapter 3: Workloads, Lifecycle & Pod Scheduling

3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics

4 Chapter 4: Services, Networking & Ingress Architecture

4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB

5 Chapter 5: Storage Architecture & Persistent Volumes

5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance

6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics

6.3 Control Plane Failures, Static Pods & Component Logs

Key Takeaways Explain This Topic AI

In a kubeadm-bootstrapped cluster, control plane components (kube-apiserver, etcd, kube-controller-manager, kube-scheduler) run as Static Pods managed directly by the local kubelet via manifests in /etc/kubernetes/manifests/.

Static pods do not depend on the API server or kube-scheduler for lifecycle management; if a manifest contains syntax errors or invalid flag arguments, the kubelet cannot start the container, and no Mirror Pod will appear in 'kubectl get pods -n kube-system'.

When the control plane is down and kubectl commands fail, diagnostics must be conducted directly on the host using 'crictl ps -a', 'crictl logs <container-id>', and inspection of /var/log/pods and /var/log/containers.

Control plane component TLS certificates in /etc/kubernetes/pki expire after 1 year; administrators inspect validity with 'kubeadm certs check-expiration' and rotate them using 'kubeadm certs renew all'.

Kubelet automatically reconciles static-Pod manifest changes. Keep every backup outside the watched directory; to force a clean recreation, move the manifest out, wait for termination, then move it back.

Last updated: August 2026

6.3 Control Plane Failures, Static Pods & Component Logs

The Kubernetes control plane is the brains of the cluster. When a control plane component fails—whether due to a corrupted configuration file, expired TLS certificates, an invalid command-line argument, or etcd unavailability—the entire cluster's declarative reconciliation engine breaks down. In severe cases, kubectl stops functioning entirely.

On the CKA examination and in enterprise disaster recovery scenarios, administrators must be capable of recovering control plane components directly from the host operating system without relying on high-level Kubernetes APIs.

1. Static Pod Mechanics & Manifest Lifecycle

In clusters provisioned with kubeadm , the four core control plane components operate as Static Pods :

  • kube-apiserver ( /etc/kubernetes/manifests/kube-apiserver.yaml )
  • etcd ( /etc/kubernetes/manifests/etcd.yaml )
  • kube-controller-manager ( /etc/kubernetes/manifests/kube-controller-manager.yaml )

kube-scheduler ( /etc/kubernetes/manifests/kube-scheduler.yaml ) +-----------------------------------------------------------------------------------------+

  • | STATIC POD RECONCILIATION ARCHITECTURE |
  • | |
  • | Filesystem: /etc/kubernetes/manifests/ |
  • | +--------------------+ +---------------+ +---------------------+ +---------------+ |
  • | | kube-apiserver.yaml| | etcd.yaml | |kube-controller-mgr | |kube-scheduler | |
  • | +---------+----------+ +-------+-------+ +----------+----------+ +-------+-------+ |
  • | | | | | |
  • | v v v v |
  • | [Kubelet File Watcher] (monitors directory specified by staticPodPath in config.yaml) |
  • | | |
  • | v |
  • | [CRI gRPC Socket] -> containerd creates / restarts containers |
  • | | |
  • | v |
  • | [Mirror Pod Creator] -> Creates read-only Mirror Pod in kube-apiserver (when API is up)|
  • +-----------------------------------------------------------------------------------------+

Critical Characteristics of Static Pods:

Managed Solely by Kubelet: The local kubelet daemon directly watches /etc/kubernetes/manifests/ . It creates, restarts, or deletes the corresponding containers when files are added, modified, or removed.

No Scheduler Dependency: Static pods bypass kube-scheduler completely and are bound to the local node.

Mirror Pods: Once kube-apiserver is running, the kubelet creates a read-only Mirror Pod in the kube-system namespace so status is visible via kubectl get pods -n kube-system . However, deleting a mirror pod with kubectl delete pod will not stop the static pod; the kubelet will immediately recreate the mirror pod.

Caution

Static Pod Directory Traps: If you create a backup file inside /etc/kubernetes/manifests/ (e.g., cp kube-apiserver.yaml kube-apiserver.yaml.bak ), the kubelet will attempt to parse .bak as a valid static pod manifest, leading to container port collisions (e.g., multiple instances attempting to bind to port 6443). Always store backup manifests outside the static pod directory , such as in /tmp or /root/manifests_backup/ .

2. Low-Level Control Plane Diagnostics with crictl and Host Logs

When kube-apiserver is down, running kubectl commands yields error messages such as: The connection to the server <host>:6443 was refused or Client.Timeout exceeded while awaiting headers .

In this state, you must SSH into the control plane host and utilize container runtime tools: # 1. SSH into the Control Plane node and elevate to root

  • ssh master-node
  • sudo -i
  • # 2. Configure crictl endpoint if not already configured
  • cat <<EOF > /etc/crictl.yaml
  • runtime-endpoint: unix:///run/containerd/containerd.sock
  • image-endpoint: unix:///run/containerd/containerd.sock
  • timeout: 10
  • debug: false
  • EOF
  • # 3. List all containers (including exited/crashed ones)
  • crictl ps -a
  • # 4. Filter for crashing control plane containers
  • crictl ps -a --name kube-apiserver
  • crictl ps -a --name etcd
  • # 5. Extract logs from a failed container instance
  • crictl logs <container-id>

Direct Log File Paths on Host Filesystem:

If crictl is unresponsive, inspect raw log files written to disk:

  • Container Log Directory: /var/log/pods/kube-system_kube-apiserver-<node>_<uid>/
  • Active symlinks: /var/log/containers/kube-apiserver-*.log
  • Kubelet daemon systemd log: journalctl -u kubelet -e --no-pager -n 100
  • 3. High-Frequency Control Plane Failure Modes & Fixes
  • Scenario 1: Manifest Syntax Corruption & Path Typos

An administrator modified /etc/kubernetes/manifests/kube-apiserver.yaml and introduced a typo in a command-line flag (e.g., --etcd-server instead of --etcd-servers ) or invalid YAML indentation.

  • Diagnosis: # Inspect kubelet logs for manifest processing errors
  • journalctl -u kubelet -e --no-pager | grep -i "error parsing manifest"
  • # Or inspect container termination reasons
  • crictl ps -a --name kube-apiserver
  • crictl logs <exited-container-id>
  • Output: unknown flag: --etcd-server

Remediation: Correct the flag in /etc/kubernetes/manifests/kube-apiserver.yaml . The kubelet detects file modification in inotify and immediately respawns the container.

Scenario 2: etcd Communication Failure (Port or Certificate Mismatch)

kube-apiserver cannot start because it cannot establish mutual TLS with etcd .

Diagnosis: crictl logs <apiserver-container-id> displays: Error: context deadline exceeded

Failed to create client: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

Root Cause Checklist:

Check /etc/kubernetes/manifests/kube-apiserver.yaml parameters:

  • -etcd-servers=https://127.0.0.1:2379 (Verify IP and port)
  • -etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
  • -etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
  • -etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
  • Verify the etcd container is alive: crictl ps --name etcd
  • Verify etcd port 2379 is listening: ss -tulpn | grep 2379
  • Scenario 3: Expired Control Plane TLS Certificates

Control plane certificates in /etc/kubernetes/pki/ have a 1-year lifespan. When they expire, all inter-component TLS handshakes fail. +-----------------------------------------------------------------------------------------+

  • | CONTROL PLANE CERTIFICATE RENEWAL PIPELINE |
  • | |
  • | 1. CHECK EXPIRATION |
  • | $ kubeadm certs check-expiration |
  • | | |
  • | v |
  • | 2. RENEW ALL CERTIFICATES |
  • | $ kubeadm certs renew all |
  • | - Updates certificates in /etc/kubernetes/pki/ |
  • | - Updates embedded client certs in /etc/kubernetes/*.conf |
  • | | |
  • | v |
  • | 3. UPDATE ADMIN KUBECONFIG |
  • | $ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config |
  • | $ chown $(id -u):$(id -g) $HOME/.kube/config |
  • | | |
  • | v |
  • | 4. RESTART STATIC PODS |
  • | - Move manifests temporarily: mv /etc/kubernetes/manifests/*.yaml /tmp/ |
  • | - Wait 10s -> Move back: mv /tmp/*.yaml /etc/kubernetes/manifests/ |
  • | - Or restart kubelet: systemctl restart kubelet |
  • +-----------------------------------------------------------------------------------------+
  • 4. kube-controller-manager & kube-scheduler Failures

When kube-apiserver is healthy but controllers fail:

Symptoms: Pods remain in Pending indefinitely ( kube-scheduler down); Deployments do not scale when updated ( kube-controller-manager down).

Diagnostics: Check their dedicated kubeconfigs located at /etc/kubernetes/controller-manager.conf and /etc/kubernetes/scheduler.conf . # Verify scheduler static pod health

kubectl get pods -n kube-system -l component=kube-scheduler

  • # If crashing, check logs directly
  • kubectl logs -n kube-system kube-scheduler-<master-node>
  • Component Dedicated Kubeconfig File Core Responsibility
  • kube-apiserver Direct local/etcd access API endpoint, authentication, authorization, storage

kube-controller-manager /etc/kubernetes/controller-manager.conf ReplicaSet, NodeLifecycle, Endpoints reconciliation

  • kube-scheduler /etc/kubernetes/scheduler.conf Node predicate filtering and priority scoring
  • cluster-admin /etc/kubernetes/admin.conf Administrative CLI access via kubectl

Loading diagram... Control Plane Crash Investigation & Recovery Flowchart