Study guide

1.5 Highly Available (HA) Control Plane Topologies

Certified Kubernetes Administrator CKA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CKA Exam Environment & Cluster Architecture

1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures

2 Chapter 2: Cluster Extensions, Packaging & RBAC

2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers

3 Chapter 3: Workloads, Lifecycle & Pod Scheduling

3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics

4 Chapter 4: Services, Networking & Ingress Architecture

4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB

5 Chapter 5: Storage Architecture & Persistent Volumes

5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance

6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics

1.5 Highly Available (HA) Control Plane Topologies

Key Takeaways Explain This Topic AI

High Availability (HA) in Kubernetes eliminates Single Points of Failure (SPOF) by running multiple redundant control plane nodes fronted by a Layer 4 Load Balancer.

Stacked etcd co-locates etcd static pods on each control plane node, requiring fewer servers but coupling API server compute with etcd disk I/O.

External etcd isolates etcd members onto dedicated standalone instances, maximizing performance and blast-radius protection at the cost of infrastructure overhead.

Bootstrapping HA clusters with kubeadm requires specifying --control-plane-endpoint (LB DNS/VIP) and --upload-certs to automatically distribute PKI secrets across masters.

Additional control plane nodes join using kubeadm join ... --control-plane --certificate-key <key>, which automatically adds the new member to the existing etcd Raft cluster.

Last updated: August 2026

1.5 Highly Available (HA) Control Plane Topologies

In enterprise production environments, a single-master Kubernetes deployment constitutes an unacceptable Single Point of Failure (SPOF) . If the control plane host suffers hardware failure, kernel panics, or unrecoverable disk corruption, all cluster administration stops: scheduling halts, failed pods cannot be replaced, autoscaling fails, and the Kubernetes API becomes unresponsive (even though existing data plane pods may temporarily continue running).

To achieve High Availability (HA) , Kubernetes deploys multiple redundant control plane nodes synchronized across independent physical fault domains. This section dissects the two primary HA control plane topologies, load-balancing mechanisms, certificate distribution, and multi-master consensus management.

1. Architectural Topologies: Stacked vs. External etcd

When designing an HA control plane, Kubernetes architects must choose between two distinct structural models for state persistence: +---------------------------------------------------------------------------------------+

  • | TOPOLOGY COMPARISON |
  • | |
  • | [TOPOLOGY 1: STACKED ETCD TOPOLOGY] |
  • | |
  • | +--------------------------+ +--------------------------+ +---------------------+ |
  • | | CONTROL PLANE NODE 1 | | CONTROL PLANE NODE 2 | | CONTROL PLANE NODE 3| |
  • | | - kube-apiserver | | - kube-apiserver | | - kube-apiserver | |
  • | | - kube-controller-mgr | | - kube-controller-mgr | | - kube-controller-m| |
  • | | - kube-scheduler | | - kube-scheduler | | - kube-scheduler | |
  • | | - etcd (local static pod| | - etcd (local static pod| | - etcd (local pod) | |
  • | +--------------------------+ +--------------------------+ +---------------------+ |
  • | \ | / |
  • | +---------------------------+---------------------------+ |
  • | | |
  • | [Raft Peer Sync: Port 2380] |
  • | |
  • | --------------------------------------------------------------------------------- |
  • | |
  • | [TOPOLOGY 2: EXTERNAL ETCD TOPOLOGY] |
  • | |
  • | +--------------------+ +--------------------+ +--------------------+ |
  • | | CONTROL PLANE CP1 | | CONTROL PLANE CP2 | | CONTROL PLANE CP3 | |
  • | | (apiserver/cm/sch) | | (apiserver/cm/sch) | | (apiserver/cm/sch) | |
  • | +--------------------+ +--------------------+ +--------------------+ |
  • | \ | / |
  • | +-------------------------+-------------------------+ |
  • | | (HTTPS Port 2379) |
  • | +-------------------------+-------------------------+ |
  • | / | \ |
  • | +--------------------+ +--------------------+ +--------------------+ |
  • | | DEDICATED ETCD ET1 |<===>| DEDICATED ETCD ET2 |<===>| DEDICATED ETCD ET3 | |
  • | +--------------------+ +--------------------+ +--------------------+ |
  • | (Port 2380 Peer Sync) |
  • +---------------------------------------------------------------------------------------+

Detailed Trade-Off Matrix:

Architectural Attribute Stacked etcd Topology External etcd Topology

Infrastructure Footprint Minimal: Requires 3 nodes minimum (Control Plane + etcd co-located). High: Requires 6 nodes minimum (3 Control Plane nodes + 3 dedicated etcd nodes).

Resource Isolation Coupled: etcd competes with API server and OS for memory and disk I/O. Dedicated: etcd has isolated CPU, RAM, and dedicated fast NVMe SSD storage.

Blast Radius High: Losing a control plane node simultaneously destroys an etcd cluster member. Isolated: Control plane node crashes do not impact etcd quorum or storage stability.

Operational Complexity Low: Fully automated deployment, member join, and cert lifecycle via kubeadm . High: Requires independent clustering scripts, manual PKI distribution, and maintenance.

Scaling Model Tied 1:1. Adding an API server node requires adding an etcd member. Decoupled. Scale API servers horizontally (e.g., 5 nodes) while keeping etcd at 3 nodes.

Recommended Use Case Standard on-premise and cloud production clusters up to 1,000 nodes. Ultra-high-scale enterprise clusters (1,000+ nodes) with extreme write throughput.

2. Load Balancing the Control Plane: HAProxy & Keepalived

Because multiple kube-apiserver instances run actively across control plane nodes, worker nodes and human administrators must not point to individual node IP addresses. Instead, all traffic is routed through a single Control Plane Endpoint (Virtual IP / DNS Name) backed by a Layer 4 Load Balancer. +-----------------------------------------------------------------------------+

  • | CONTROL PLANE LOAD BALANCING |
  • | |
  • | [Worker Nodes / kubectl CLI] |
  • | | |
  • | v |
  • | [Virtual IP (VIP): 192.168.1.100:6443 (Managed by Keepalived VRRP)] |
  • | | |
  • | v |
  • | [HAProxy TCP Load Balancer (Round Robin / Leastconn)] |
  • | | |
  • | +-------+-------+----------------------------------+ |
  • | | (TCP 6443) | (TCP 6443) | (TCP 6443) |
  • | v v v |
  • | +-----------+ +-----------+ +-----------+ |
  • | | Control | | Control | | Control | |
  • | | Plane CP1 | | Plane CP2 | | Plane CP3 | |
  • | +-----------+ +-----------+ +-----------+ |
  • +-----------------------------------------------------------------------------+
  • Sample /etc/haproxy/haproxy.cfg Configuration: frontend kubernetes-frontend
  • bind 192.168.1.100:6443
  • mode tcp
  • option tcplog
  • default_backend kubernetes-backend
  • backend kubernetes-backend
  • mode tcp
  • option tcp-check
  • balance roundrobin
  • server cp1 192.168.1.11:6443 check fall 3 rise 2
  • server cp2 192.168.1.12:6443 check fall 3 rise 2
  • server cp3 192.168.1.13:6443 check fall 3 rise 2

Note

Layer 4 Passthrough: The load balancer must operate in Layer 4 TCP passthrough mode (not Layer 7 HTTP termination) so that mTLS client certificates passed by kubectl and kubelet terminate directly on kube-apiserver .

3. Multi-Master Bootstrapping with Kubeadm ( --upload-certs )

In older Kubernetes releases, joining secondary control plane nodes required manual SCP copying of PKI certificates ( ca.crt , ca.key , sa.key , etcd/ca.crt , etcd/ca.key ). Modern kubeadm automates this using the Automated Certificate Upload Mechanism . +-----------------------------------------------------------------------------+

  • | KUBEADM HA CERTIFICATE DISTRIBUTION FLOW |
  • | |
  • | [PRIMARY MASTER: CP1] |
  • | $ kubeadm init --control-plane-endpoint "lb.k8s.lan:6443" --upload-certs |
  • | |
  • | 1. Generates PKI CA, etcd, and SA keys. |
  • | 2. Generates random 32-byte AES decryption key (Certificate Key). |
  • | 3. Encrypts certificates with Key -> Saves to Secret in kube-system |
  • | (Secret: kubeadm-certs, TTL: 2 Hours). |
  • | 4. Prints Join Command with --control-plane and --certificate-key. |
  • | |
  • | [SECONDARY MASTER: CP2 / CP3] |
  • | $ kubeadm join lb.k8s.lan:6443 --token <tok> --discovery-token-ca-cert-hash \
  • | sha256:<hash> --control-plane --certificate-key <key> |
  • | |
  • | 1. Fetches encrypted Secret from CP1 via bootstrap token. |
  • | 2. Decrypts certs locally using --certificate-key. |
  • | 3. Adds local etcd member to Raft cluster via peer port 2380. |
  • | 4. Spawns local apiserver, controller-manager, scheduler static pods. |
  • +-----------------------------------------------------------------------------+

Step-by-Step Multi-Master Initialization Workflow:

  • Step 1: Initialize Primary Master (CP1) sudo kubeadm init \
  • -control-plane-endpoint "k8s-lb.production.lan:6443" \
  • -upload-certs \
  • -pod-network-cidr=10.244.0.0/16

The output displays two distinct join commands:

One for joining additional control plane nodes (contains --control-plane and --certificate-key ).

One for joining worker nodes (standard token and hash only).

  • Step 2: Joining Secondary Control Plane Nodes (CP2 and CP3)
  • Execute on CP2 and CP3: sudo kubeadm join k8s-lb.production.lan:6443 \
  • -token 9v7q8z.1234567890abcdef \
  • -discovery-token-ca-cert-hash sha256:7f8e9d0a... \
  • -control-plane \
  • -certificate-key e4a1b2c3d4e5f60718293a4b5c6d7e8f90123456789abcdef0123456789a

Handling Expired Certificate Keys (Past 2-Hour Window):

If more than 2 hours have elapsed, the uploaded Secret is automatically garbage-collected. Generate a fresh key and re-upload certificates with: sudo kubeadm init phase upload-certs --upload-certs

4. Multi-Master Consensus & Leader Election

While all kube-apiserver instances run Active-Active behind the load balancer, components with single-writer reconciliation logic ( kube-controller-manager and kube-scheduler ) run in Active-Passive mode using Lease Objects ( coordination.k8s.io ). # Inspecting active leader leases:

  • kubectl get leases -n kube-system NAME HOLDER AGE
  • kube-controller-manager cp-node-1.lan_a8b2... 14m
  • kube-scheduler cp-node-2.lan_c9d4... 14m

If cp-node-1 fails, its lease renewal heartbeat fails. Within 15 seconds ( --leader-elect-lease-duration ), standby controller managers on cp-node-2 or cp-node-3 acquire the lease and seamlessly resume reconciliation loops.

Loading diagram... High Availability Multi-Master Control Plane with Stacked etcd