Study guide

6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation

Certified Kubernetes Administrator CKA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CKA Exam Environment & Cluster Architecture

1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures

2 Chapter 2: Cluster Extensions, Packaging & RBAC

2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers

3 Chapter 3: Workloads, Lifecycle & Pod Scheduling

3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics

4 Chapter 4: Services, Networking & Ingress Architecture

4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB

5 Chapter 5: Storage Architecture & Persistent Volumes

5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance

6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics

6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation

Key Takeaways Explain This Topic AI

etcd is a distributed, consistent key-value store utilizing the Raft consensus algorithm; maintaining a functioning cluster requires a strict majority quorum of Q = floor(n/2) + 1 members.

When quorum is lost, etcd cannot commit writes or reliably serve quorum-dependent linearizable operations; API requests may time out or fail until a majority is restored.

The default etcd backend quota is 2 GiB; 8 GiB is a suggested normal-environment maximum rather than a hard configurable ceiling.

etcdctl commands require specifying the v3 API ('ETCDCTL_API=3') along with mutual TLS flags: '--endpoints', '--cacert', '--cert', and '--key'.

Save a live snapshot with etcdctl, but use etcdutl for current offline snapshot status and restore operations, then update the stacked etcd static Pod hostPath to the restored directory.

Last updated: August 2026

6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation

etcd is the authoritative state store for the entire Kubernetes cluster. Every resource definition, namespace, secret, configmap, pod status, and node lease is stored within etcd. Because kube-apiserver is stateless, the health, performance, and durability of the entire cluster depend directly on etcd.

Understanding Raft consensus mechanics, quorum requirements, database maintenance routines (compaction, defragmentation, alarm disarming), and snapshot backup/restoration is a core competency tested heavily on the CKA exam.

1. Raft Consensus & Quorum Dynamics

etcd uses the Raft consensus algorithm to ensure consistent data replication across a cluster of nodes. To commit a transaction or elect a new leader, etcd requires agreement from a strict majority (quorum) of active members. Quorum = ⌊ N 2 ⌋ + 1 \text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1 Quorum = ⌊ 2 N ​ ⌋ + 1

Where N N N is the total number of members in the etcd cluster.

  • Cluster Size ( N N N ) Quorum Needed Maximum Tolerable Node Failures
  • 1 (Single Master) 1 0 (Any failure results in total outage)
  • 3 (Standard HA) 2 1 (1 node can fail; 2 surviving nodes maintain quorum)
  • 5 (Large Enterprise HA) 3 2 (2 nodes can fail; 3 surviving nodes maintain quorum)
  • 7 (Ultra HA) 4 3 (3 nodes can fail; 4 surviving nodes maintain quorum)

Important

Odd Number of Members: etcd clusters should always contain an odd number of members (1, 3, or 5). Adding an even member (e.g., moving from 3 to 4 nodes) increases the quorum requirement from 2 to 3 without increasing fault tolerance (both 3-node and 4-node clusters can only survive 1 failure), while introducing additional network overhead.

Quorum Loss Symptoms:

If a 3-node etcd cluster loses 2 nodes, quorum is lost:

etcdctl reports: Error: context deadline exceeded or rafthttp: failed to find member on leader .

kube-apiserver returns: HTTP 500 Internal Server Error: etcdserver: no leader .

Do not describe this as a supported read-only mode. Quorum-dependent etcd operations cannot complete, so Kubernetes API reads or writes may time out or fail depending on the request and cache path.

2. Setting Up the etcdctl Environment

In standard kubeadm deployments, etcd is configured with mutual TLS authentication. Running etcdctl without proper credentials will result in connection rejection. # 1. Export ETCDCTL_API version 3

export ETCDCTL_API=3

  • # 2. Define standard PKI asset paths
  • export ETCD_CA="/etc/kubernetes/pki/etcd/ca.crt"
  • export ETCD_CERT="/etc/kubernetes/pki/etcd/server.crt"
  • export ETCD_KEY="/etc/kubernetes/pki/etcd/server.key"

# 3. Create a reusable shell alias for the CKA exam

alias etcd-cmd="etcdctl --cacert=$ETCD_CA --cert=$ETCD_CERT --key=$ETCD_KEY --endpoints=https://127.0.0.1:2379"

  • Essential Health & Cluster Inspection Commands: # Check health of local or remote endpoints
  • etcd-cmd endpoint health
  • # Output: https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.15ms
  • # Inspect detailed status (Database Size, Leader ID, Raft Index)
  • etcd-cmd endpoint status --write-out=table
  • # List all members of the etcd cluster
  • etcd-cmd member list --write-out=table
  • 3. Database Maintenance: Space Quota, Compaction & Defragmentation

By default, etcd uses a 2 GiB backend quota. The etcd documentation suggests 8 GiB as the maximum for a normal environment; it is guidance, not a hard parser limit on --quota-backend-bytes . When keys are frequently created, updated, or deleted (e.g., high-frequency node leases and events), etcd retains historical revisions for MVCC (Multi-Version Concurrency Control). Over time, the database file ( member/snap/db ) grows. +-----------------------------------------------------------------------------------------+

  • | ETCD DATABASE SPACE EXHAUSTION & RECOVERY |
  • | |
  • | 1. QUOTA EXCEEDED -> etcd triggers 'NOSPACE' Alarm -> API Server writes rejected |
  • | | |
  • | v |
  • | 2. CHECK ALARMS |
  • | $ etcd-cmd alarm list |
  • | Output: memberID:1029384756 alarm:NOSPACE |
  • | | |
  • | v |
  • | 3. COMPACT HISTORICAL REVISIONS |
  • | # Get current revision number from endpoint status |
  • | REV=$(etcd-cmd endpoint status --write-out=json | jq .[0].Status.header.revision) |
  • | $ etcd-cmd compact $REV |
  • | | |
  • | v |
  • | 4. DEFRAGMENT DATABASE (Reclaims free disk space from fragmented pages) |
  • | $ etcd-cmd defrag |
  • | | |
  • | v |
  • | 5. DISARM ALARM (Restores write operations) |
  • | $ etcd-cmd alarm disarm |
  • +-----------------------------------------------------------------------------------------+
  • Step-by-Step Defragmentation Runbook: # Step 1: Check active alarms
  • etcd-cmd alarm list

# Step 2: Compact database up to current revision

# Retrieve the current revision:

  • CURRENT_REV=$(ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  • -cacert=/etc/kubernetes/pki/etcd/ca.crt \
  • -cert=/etc/kubernetes/pki/etcd/server.crt \
  • -key=/etc/kubernetes/pki/etcd/server.key \
  • endpoint status -w json | grep -o '"revision":[0-9]*' | cut -d: -f2)
  • ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  • -cacert=/etc/kubernetes/pki/etcd/ca.crt \
  • -cert=/etc/kubernetes/pki/etcd/server.crt \
  • -key=/etc/kubernetes/pki/etcd/server.key \
  • compact $CURRENT_REV
  • # Step 3: Defragment the database to shrink the physical on-disk file
  • ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  • -cacert=/etc/kubernetes/pki/etcd/ca.crt \
  • -cert=/etc/kubernetes/pki/etcd/server.crt \
  • -key=/etc/kubernetes/pki/etcd/server.key \
  • defrag
  • # Step 4: Disarm the NOSPACE alarm to enable writes
  • ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  • -cacert=/etc/kubernetes/pki/etcd/ca.crt \
  • -cert=/etc/kubernetes/pki/etcd/server.crt \
  • -key=/etc/kubernetes/pki/etcd/server.key \
  • alarm disarm
  • 4. Disaster Recovery: Current Tool Split

Use etcdctl with endpoint and mTLS flags for live server operations, including snapshot save , endpoint health, compaction, defragmentation, and alarms. Use the offline etcdutl utility for snapshot inspection and restore: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \

  • -cacert=/etc/kubernetes/pki/etcd/ca.crt \
  • -cert=/etc/kubernetes/pki/etcd/server.crt \
  • -key=/etc/kubernetes/pki/etcd/server.key \
  • snapshot save /var/lib/etcd-backup.db
  • etcdutl --write-out=table snapshot status /var/lib/etcd-backup.db
  • etcdutl snapshot restore /var/lib/etcd-backup.db \
  • -data-dir=/var/lib/etcd-restored

Before activating a restore on a stacked kubeadm control plane, move the API server and etcd manifests out of /etc/kubernetes/manifests and wait for their containers to stop. Restore to a new empty host directory. In the backed-up etcd.yaml , change the etcd-data hostPath to /var/lib/etcd-restored , while leaving the container mount path and internal --data-dir=/var/lib/etcd consistent. Restore the etcd manifest, inspect it with crictl , restore the API server manifest, and verify endpoint health plus Kubernetes objects. Multi-member recovery additionally requires unique member names and peer settings.

Loading diagram... etcd Maintenance and Current Snapshot Tooling