Study guide

5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

Certified Kubernetes Administrator CKA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CKA Exam Environment & Cluster Architecture

1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures

2 Chapter 2: Cluster Extensions, Packaging & RBAC

2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers

3 Chapter 3: Workloads, Lifecycle & Pod Scheduling

3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics

4 Chapter 4: Services, Networking & Ingress Architecture

4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB

5 Chapter 5: Storage Architecture & Persistent Volumes

5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance

6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics

5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

Key Takeaways Explain This Topic AI

StatefulSets provide stable, unique pod identities (pod-0, pod-1) and dedicated persistent storage per replica using volumeClaimTemplates, unlike Deployments where all replicas share the same volume.

The dynamic PVC generation formula strictly follows '<volumeClaimTemplate-name>-<statefulset-name>-<ordinal-index>', creating an independent, zero-indexed PVC for every replica.

When a StatefulSet Pod terminates or is rescheduled to a different node, the replacement Pod retains its exact ordinal identity and automatically reattaches to its dedicated existing PVC and PV.

persistentVolumeClaimRetentionPolicy gives granular control over PVC lifecycle when a StatefulSet is deleted (whenDeleted: Retain/Delete) or scaled down (whenScaled: Retain/Delete).

Partitioned rolling updates (updateStrategy.rollingUpdate.partition) enable phased canary deployments and controlled database schema migrations across ordinal stateful replicas.

Last updated: August 2026

5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

Deploying distributed, stateful clustering systems—such as Apache Kafka, Elasticsearch, Cassandra, ZooKeeper, and replicated MySQL/PostgreSQL databases—presents challenges that standard Kubernetes Deployments and ReplicaSets cannot resolve. Stateless Deployments treat pods as anonymous, interchangeable entities: all replicas share the exact same PodSpec , scale up and down arbitrarily, and if configured with a PVC, attempt to mount the identical shared volume (requiring ReadWriteMany support).

Clustered stateful applications, however, demand three architectural invariants:

Unique, persistent network identity that remains invariant across pod restarts.

Dedicated, isolated persistent storage per replica that is never shared with peer replicas.

Deterministic, ordered provisioning and graceful decommissioning to maintain distributed quorum.

Kubernetes fulfills these requirements through the StatefulSet workload controller.

1. StatefulSet vs. Deployment Storage Architecture =============================== DEPLOYMENT STORAGE ===============================

  • Deployment (replicas: 3) ---> References single PVC: shared-pvc
  • +--------------+ +--------------+ +--------------+
  • | web-pod-7x8a | | web-pod-9b2c | | web-pod-1z4e |
  • +-------+------+ +-------+------+ +-------+------+
  • | | |
  • +-----------------+-----------------+
  • |
  • v
  • +-------------------+
  • | shared-pvc | (Requires RWX; all pods share same disk)
  • +-------------------+
  • ============================== STATEFULSET STORAGE ===============================
  • StatefulSet: kafka (replicas: 3) with volumeClaimTemplate: data
  • +---------------+ +---------------+ +---------------+
  • | kafka-0 | | kafka-1 | | kafka-2 |
  • +-------+-------+ +-------+-------+ +-------+-------+
  • | | |
  • v v v
  • +---------------+ +---------------+ +---------------+
  • | data-kafka-0 | | data-kafka-1 | | data-kafka-2 |
  • | (PVC / PV) | | (PVC / PV) | | (PVC / PV) |
  • +---------------+ +---------------+ +---------------+
  • (Dedicated 1-to-1 persistent disk per ordinal replica; preserves state across failover)

Architectural Dimension Deployment Storage Model StatefulSet Storage Model

Storage Declaration References pre-existing PVC in spec.template.spec.volumes Dynamically generates PVCs via spec.volumeClaimTemplates

Volume Cardinality All replicas bind to the same single PVC Each replica receives its own dedicated, isolated PVC

Pod Identity Random hash suffix (e.g., api-6d4b9f7c-x8k2j ) Zero-indexed ordinal (e.g., kafka-0 , kafka-1 , kafka-2 )

Failover Reattachment Replacement pod binds to whatever volume is declared Replacement kafka-1 binds strictly to existing data-kafka-1

  • DNS Hostname Ephemeral Pod IP behind Service Predictable DNS via Headless Service
  • 2. Dynamic PVC Generation Formula

When a StatefulSet with replicas: N is applied, the StatefulSet controller dynamically synthesizes and creates N N N individual PersistentVolumeClaim resources. The PVC names are generated using a deterministic formula:

<volumeClaimTemplate-name>-<statefulset-name>-<ordinal-index>

Formula Evaluation Matrix:

  • StatefulSet Name volumeClaimTemplates.metadata.name Replica Count Generated PVC Names
  • cassandra data 3 data-cassandra-0 , data-cassandra-1 , data-cassandra-2

zk-cluster zookeeper-log 3 zookeeper-log-zk-cluster-0 , zookeeper-log-zk-cluster-1 , zookeeper-log-zk-cluster-2

  • elastic storage 2 storage-elastic-0 , storage-elastic-1
  • 3. Pod Rescheduling & Persistent Storage Binding

The fundamental design guarantee of StatefulSets is that storage is bound to the ordinal identity, not the physical worker node . +-----------------------------------------------------------------------------------------+

  • | STATEFULSET FAILOVER & REATTACHMENT |
  • | |
  • | [INITIAL HEALTHY STATE] |
  • | Worker Node 1: kafka-0 <=======> Mounts: data-kafka-0 (PV-0) |
  • | Worker Node 2: kafka-1 <=======> Mounts: data-kafka-1 (PV-1) |
  • | |
  • | [CRASH EVENT: Worker Node 2 Hardware Failure] |
  • | 1. Node 2 becomes NotReady -> Kubelet stops heartbeating. |
  • | 2. Node controller evicts kafka-1. |
  • | 3. StatefulSet controller detects missing replica with ordinal 1. |
  • | 4. Scheduler places new kafka-1 on Worker Node 3. |
  • | |
  • | [FAILOVER RECOVERY: Worker Node 3] |
  • | 5. Kubelet on Node 3 watches Pod kafka-1. |
  • | 6. CSI attacher detaches PV-1 from Node 2 and attaches PV-1 to Node 3. |
  • | 7. kafka-1 starts with 100% of its previous persistent data intact! |
  • +-----------------------------------------------------------------------------------------+
  • 4. persistentVolumeClaimRetentionPolicy (StatefulSet Scaling & Deletion)

Historically, deleting or scaling down a StatefulSet never deleted any associated PVCs. While this protected against data loss, it caused orphaned PVCs and persistent cloud billing. Modern Kubernetes (stable since v1.32 and available in v1.35) provides the persistentVolumeClaimRetentionPolicy API. spec:

persistentVolumeClaimRetentionPolicy:

  • whenDeleted: Retain # Options: Retain, Delete
  • whenScaled: Retain # Options: Retain, Delete

Retention Policies Explained:

whenDeleted : Dictates what occurs to associated PVCs when the parent StatefulSet resource is deleted ( kubectl delete statefulset <name> ):

Retain (Default): PVCs and underlying PVs are preserved in the namespace.

Delete : All PVCs created by volumeClaimTemplates are automatically deleted along with the StatefulSet.

whenScaled : Dictates what occurs to PVCs when the StatefulSet replica count is reduced (e.g., scaling from 5 replicas down to 2):

Retain (Default): PVCs for scaled-down pods ( data-app-4 , data-app-3 , data-app-2 ) remain intact. If scaled back up, the pods reattach to them.

Delete : The PVCs belonging to the terminated replicas are permanently deleted, freeing cloud storage assets.

  • 5. Pod Management Policies & Headless Services
  • 1. podManagementPolicy ( OrderedReady vs. Parallel )

OrderedReady (Default): Replicas are created sequentially in strict ascending order ( 0 → 1 → 2 0 \to 1 \to 2 0 → 1 → 2 ). Pod N N N is not launched until Pod N − 1 N-1 N − 1 is in the Running and Ready state. Teardown occurs in reverse order ( 2 → 1 → 0 2 \to 1 \to 0 2 → 1 → 0 ). Essential for clustering protocols (e.g., bootstrapping Raft leaders).

Parallel : Replicas are launched and terminated concurrently without waiting for peer readiness. Used when workloads need independent storage identities but rapid scaling.

2. Headless Service ( clusterIP: None )

A StatefulSet requires a companion Headless Service to establish stable network identities. CoreDNS automatically generates A/AAAA DNS records for each individual pod:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

Example: kafka-0.kafka-headless.production.svc.cluster.local .

6. Partitioned Rolling Updates & Canary Migrations

StatefulSets support staged rollouts using the partition directive under spec.updateStrategy.rollingUpdate . spec:

updateStrategy:

type: RollingUpdate

rollingUpdate:

  • partition: 2
  • Partition Mechanics & Canary Progression

When partition: 2 is specified on a 4-replica StatefulSet ( db-0 , db-1 , db-2 , db-3 ):

Only replicas with ordinal index ≥ 2 \ge 2 ≥ 2 ( db-2 and db-3 ) are updated to the new PodSpec.

Replicas with ordinal index < 2 < 2 < 2 ( db-0 and db-1 ) remain untouched on the legacy version.

Canary Database Migration Workflow:

Set partition: 3 -> Only db-3 is updated. Validate telemetry, database schema migrations, and replication lag.

If healthy, decrement partition: 1 -> Updates db-2 and db-1 .

Decrement partition: 0 -> Updates the final replica ( db-0 ), completing full cluster upgrade.

  • 7. Production-Grade StatefulSet Declarative Manifest apiVersion: v1
  • kind: Service

metadata:

  • name: postgres-ha
  • namespace: production

labels:

app: postgres

spec:

clusterIP: None

ports:

  • port: 5432
  • name: postgresql

selector:

  • app: postgres
  • --
  • apiVersion: apps/v1
  • kind: StatefulSet

metadata:

  • name: pg-cluster
  • namespace: production

spec:

  • serviceName: postgres-ha
  • replicas: 3
  • podManagementPolicy: OrderedReady

updateStrategy:

type: RollingUpdate

rollingUpdate:

partition: 0

persistentVolumeClaimRetentionPolicy:

  • whenDeleted: Retain
  • whenScaled: Retain

selector:

matchLabels:

app: postgres

template:

metadata:

labels:

app: postgres

spec:

containers:

  • name: postgres
  • image: postgres:16-alpine

ports:

  • containerPort: 5432
  • name: postgresql

volumeMounts:

  • name: pgdata
  • mountPath: /var/lib/postgresql/data
  • subPath: data

volumeClaimTemplates:

- metadata:

name: pgdata

spec:

  • accessModes: [ "ReadWriteOnce" ]
  • storageClassName: high-performance-ebs

resources:

requests:

  • storage: 100Gi
  • 8. StatefulSet Disaster Recovery & Storage Maintenance # 1. Scale down StatefulSet to isolate broken replica
  • kubectl scale statefulset pg-cluster -n production --replicas=2
  • # 2. Inspect orphaned PVCs left behind after scale-down
  • kubectl get pvc -n production -l app=postgres

# 3. Perform online volume expansion on StatefulSet PVCs

kubectl patch pvc pgdata-pg-cluster-0 -n production --type=merge -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

kubectl patch pvc pgdata-pg-cluster-1 -n production --type=merge -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

kubectl patch pvc pgdata-pg-cluster-2 -n production --type=merge -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

Loading diagram... StatefulSet Ordinal Identity, Headless Service DNS, and Dedicated Volume Architecture