Study guide
1.3 etcd Architecture, Clustering & Raft Consensus
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
1.3 etcd Architecture, Clustering & Raft Consensus
Key Takeaways Explain This Topic AI
etcd is a strongly consistent, distributed key-value store implementing the Raft consensus algorithm, serving as Kubernetes' exclusive datastore for all cluster metadata and resource objects.
Raft requires a strict quorum majority ( Q = ⌊ N / 2 ⌋ + 1 Q = \lfloor N/2 \rfloor + 1 Q = ⌊ N /2 ⌋ + 1 ) to elect a Leader and commit write transactions; a cluster of N N N nodes can tolerate at most ⌊ ( N − 1 ) / 2 ⌋ \lfloor (N-1)/2 \rfloor ⌊( N − 1 ) /2 ⌋ simultaneous node failures.
Odd etcd member counts are recommended because adding an even member increases quorum cost without improving failure tolerance; choose member count from the required fault tolerance and latency budget.
etcd utilizes two distinct network ports secured via mutual TLS (mTLS): Port 2379 for client API traffic (apiserver-to-etcd) and Port 2380 for peer-to-peer Raft synchronization.
etcd maintenance involves monitoring database fragmentation, key space revisions, executing periodic defragmentation (etcdctl defrag), and compaction (etcdctl compact) to prevent database full alarms.
Last updated: August 2026
1.3 etcd Architecture, Clustering & Raft Consensus
At the foundation of every Kubernetes cluster sits etcd , an open-source, distributed, strongly consistent key-value store created by CoreOS and maintained by the CNCF. In Kubernetes, etcd holds the complete single source of truth: every Namespace, Pod, ConfigMap, Secret, Deployment, CRD, and event object is serialized into Protocol Buffers and persisted into etcd's bbolt database file ( member/snap/db ).
Because all state mutations pass through etcd, understanding its consensus mechanics, clustering constraints, failure domains, and CLI tooling ( etcdctl ) is one of the highest-yield subjects on the CKA examination.
1. Raft Consensus Mechanics & State Replication
etcd implements the Raft consensus algorithm to ensure that all members in a distributed cluster maintain an identical, ordered log of key-value operations despite node crashes or network partitions. +-----------------------------------------------------------------------------+
- | RAFT CONSENSUS STATE MACHINE |
- | |
- | Client Request (Write) |
- | | |
- | v |
- | +---------------+ |
- | | RAFT LEADER | |
- | | (Node 1:2379) | |
- | +---------------+ |
- | | |
- | +-------------------+ AppendEntries RPC (Port 2380) |
- | | | |
- | v v |
- | +---------------+ +---------------+ |
- | | RAFT FOLLOWER | | RAFT FOLLOWER | |
- | | (Node 2) | | (Node 3) | |
- | +---------------+ +---------------+ |
- | | | |
- | +-----> [QUORUM: 2 of 3 Acknowledge Write] <-----+ |
- | | |
- | v |
- | [TRANSACTION COMMITTED TO DISK] |
- +-----------------------------------------------------------------------------+
Raft Operational Roles:
Leader: Accepts all write proposals from client applications ( kube-apiserver ). The leader assigns monotonically increasing index numbers to entries, writes them to its local Write-Ahead Log (WAL), and replicates entries to all Follower nodes via AppendEntries RPCs.
Follower: Completely passive. Forwards write requests to the Leader. Accepts heartbeats and log replication RPCs from the leader. If heartbeats cease within an election timeout window, the follower transitions to Candidate.
Candidate: Initiates a new election term, increments term counter, votes for itself, and broadcasts RequestVote RPCs to peer nodes.
Write Lifecycle & Quorum Commit:
Kube-apiserver issues a PUT request to the Leader on client port 2379.
Leader writes entry to local Write-Ahead Log (WAL).
Leader broadcasts AppendEntries RPCs to Followers across peer port 2380.
When a Quorum Majority of nodes confirm writing the log entry to their disk, the Leader commits the transaction.
The Leader applies the transaction to its key-value state machine (bbolt DB) and returns success to kube-apiserver .
Followers apply the committed entry to their local state machines upon receiving the Leader's next heartbeat.
2. Quorum Formulation & Fault Tolerance Analysis
Raft guarantees consistency by requiring that any state modification or leader election receive votes from a strict majority of the cluster , known as Quorum . Quorum = ⌊ N 2 ⌋ + 1 \text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1 Quorum = ⌊ 2 N ⌋ + 1
Where N N N is the total active member count of the etcd cluster. The maximum number of simultaneous failure tolerance ( F F F ) is calculated as: F = ⌊ N − 1 2 ⌋ F = \left\lfloor \frac{N - 1}{2} \right\rfloor F = ⌊ 2 N − 1 ⌋
Quorum & Fault Tolerance Matrix:
Total Members ( N N N ) Quorum Needed Maximum Tolerated Failures ( F F F ) Architectural Viability
1 1 0 Dev / Test clusters only. Zero fault tolerance.
2 2 0 POOR DESIGN: 2 nodes require 2 votes for quorum; 1 failure breaks quorum. Worse than 1 node!
3 2 1 Standard Production Baseline: Survives loss of 1 node.
4 3 1 POOR DESIGN: Requires 3 votes for quorum. Tolerates only 1 failure (same as 3 nodes) with added network overhead.
5 3 2 Enterprise High Availability: Survives loss of 2 nodes simultaneously.
7 4 3 Large-scale clusters. Additional members increase cross-node RPC latency.
Important
The Odd Number Rule: Adding an even member (e.g., going from 3 to 4 nodes) does not increase fault tolerance (both tolerate only 1 failure), but it increases the number of votes required for quorum from 2 to 3. The fourth member adds replication work without increasing tolerated failures. Prefer an odd member count such as 3 or 5 unless a documented transition or topology requirement justifies a temporary even count; quorum loss makes consensus-dependent operations unavailable, not a supported read-only cluster mode.
3. etcd Communication Ports & Mutual TLS (mTLS)
etcd isolates administrative client traffic from inter-node synchronization by binding to two distinct TCP ports:
Port 2379 (Client API): Listens for incoming queries from kube-apiserver and etcdctl .
Port 2380 (Peer-to-Peer): Listens for internal Raft consensus heartbeats, leader election ballots, and log streaming between etcd cluster members.
Kubeadm configures etcd client and peer communication with mutual TLS (mTLS) . Other etcd deployments are configurable, so verify the actual flags and certificates. To execute administrative CLI commands, you must supply the trusted CA cert, client certificate, and private key. /etc/kubernetes/pki/etcd/
- ├── ca.crt # etcd Certificate Authority
- ├── ca.key # etcd CA Private Key
- ├── server.crt # Server certificate (Port 2379)
- ├── server.key # Server private key
- ├── peer.crt # Peer-to-peer cert (Port 2380)
- ├── peer.key # Peer private key
- ├── healthcheck-client.crt # Used by the local etcd health probe
- └── healthcheck-client.key
- 4. etcdctl v3 Command Operations & Health Diagnostics
On Kubernetes nodes, the etcdctl binary communicates using the v3 API. Ensure ETCDCTL_API=3 is set in your environment.
Essential Diagnostic Commands: # Export API version:
export ETCDCTL_API=3
# Common TLS flag alias for exam speed:
- alias etcd-cmd="etcdctl \
- -cacert=/etc/kubernetes/pki/etcd/ca.crt \
- -cert=/etc/kubernetes/pki/etcd/server.crt \
- -key=/etc/kubernetes/pki/etcd/server.key \
- -endpoints=https://127.0.0.1:2379"
# 1. Check Cluster Member List:
etcd-cmd member list -w table
# 2. Check Endpoint Health:
etcd-cmd endpoint health -w table
# 3. Check Endpoint Status (DB Size, Leader ID, Raft Revision):
etcd-cmd endpoint status -w table
Output Interpretation of endpoint status :
Column Technical Meaning Exam Significance
ENDPOINT IP:Port of the etcd listener Validates correct local or remote target.
ID 64-bit Hex Member ID Identifies member in Raft cluster.
VERSION etcd binary version (e.g., 3.5.15) Verifies version parity across nodes.
DB SIZE Total allocated disk size of bbolt DB If size nears quota limit (default 2GB), alarms trigger.
IS LEADER Boolean ( true / false ) Identifies which node is currently Raft leader.
RAFT REVISION Total sequential counter of all mutations Revisions must be near-identical across healthy peers.
5. Compaction, Defragmentation & Space Management
etcd is a Multi-Version Concurrency Control (MVCC) datastore. When a Kubernetes object is updated or deleted, etcd does not overwrite the old data immediately; it appends a new revision so clients can watch historical changes. Over time, old revisions consume disk space. # 1. Compact old revisions up to revision 35000:
etcd-cmd compact 35000
# 2. Defragment backend storage to release freed pages back to the filesystem:
etcd-cmd defrag --endpoints=https://127.0.0.1:2379
# 3. Disarm space quota alarm if triggered: