Study guide

6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors

Certified Kubernetes Administrator CKA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CKA Exam Environment & Cluster Architecture

1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures

2 Chapter 2: Cluster Extensions, Packaging & RBAC

2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers

3 Chapter 3: Workloads, Lifecycle & Pod Scheduling

3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics

4 Chapter 4: Services, Networking & Ingress Architecture

4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB

5 Chapter 5: Storage Architecture & Persistent Volumes

5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance

6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics

6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors

Key Takeaways Explain This Topic AI

PersistentVolumeClaims (PVCs) remain in the 'Pending' phase due to: missing or non-default StorageClasses, insufficient available PersistentVolume (PV) capacity, accessModes mismatches (e.g., ReadWriteMany vs ReadWriteOnce), or label/selector conflicts.

When 'volumeBindingMode: WaitForFirstConsumer' is configured on a StorageClass, dynamic PV provisioning and PVC binding are intentionally delayed until a pod referencing the PVC is scheduled to a specific node.

Pods get stuck in 'ContainerCreating' with 'Multi-Attach error for volume' when a ReadWriteOnce (RWO) volume is still locked by a detached or crashed node that has not completed unmount/detach operations.

Volume expansion requires allowVolumeExpansion on the StorageClass; online filesystem growth depends on CSI and filesystem support, and FileSystemResizePending may require restarting a consuming Pod.

Before reusing a Retain-policy Released PV, secure or scrub its data for the intended claimant, then clear spec.claimRef and constrain the new binding deliberately.

Last updated: August 2026

6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors

Managing persistent state in a distributed container orchestration system involves coordinating the Kubernetes storage control loops ( PV controller , Attach/Detach controller , and Volume Manager ) with third-party Container Storage Interface (CSI) drivers, cloud storage APIs, and host filesystem mount points.

Storage misconfigurations cause pods to hang in Pending or ContainerCreating . Diagnosing storage problems requires understanding the binding lifecycle, CSI plugin communication, and reclaim policies.

1. Storage Architecture & Binding Lifecycle +-----------------------------------------------------------------------------------------+

  • | STORAGE BINDING & ATTACHMENT PIPELINE |
  • | |
  • | 1. DEVELOPER CREATES PVC |
  • | apiVersion: v1, kind: PersistentVolumeClaim (Requests 10Gi, ReadWriteOnce) |
  • | | |
  • | v |
  • | 2. PV CONTROLLER ATTEMPTS BINDING |
  • | - Static Provisioning: Matches existing PV with sufficient capacity, matching |
  • | accessModes, and matching storageClassName. |
  • | - Dynamic Provisioning: StorageClass invokes CSI External Provisioner to create PV. |
  • | | |
  • | v |
  • | 3. PVC STATUS: BOUND (PV status: Bound) |
  • | | |
  • | v |
  • | 4. POD REFERENCING PVC IS SCHEDULED |
  • | - Attach/Detach Controller calls CSI Driver -> Attaches block device to Node. |
  • | - Kubelet Volume Manager mounts filesystem to /var/lib/kubelet/pods/<uid>/volumes. |
  • | - Container launches with mounted volume. |
  • +-----------------------------------------------------------------------------------------+
  • 2. Troubleshooting PVC: Pending State

When a PersistentVolumeClaim remains in Pending , execute kubectl describe pvc <pvc-name> to inspect controller events: kubectl describe pvc my-claim

High-Frequency PVC Pending Causes & Fixes:

Failure Message in Events Root Cause Remediation

no volume plugin matched name or storageclass not found The PVC specifies a storageClassName that does not exist in the cluster ( kubectl get sc ). Create the missing StorageClass or fix the typo in pvc.spec.storageClassName .

waiting for a volume to be created, either by external provisioner or manually No StorageClass was specified and no default StorageClass is configured in the cluster. Mark a StorageClass as default: kubectl patch sc <sc-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' .

cannot satisfy accessModes [ReadWriteMany] Static PVs exist, but none provide the requested ReadWriteMany (RWX) access mode (most cloud block stores only support ReadWriteOnce ). Change PVC request to ReadWriteOnce or provision an NFS/Ceph/EFS PV supporting RWX.

insufficient capacity: requested 20Gi, available 10Gi Static PV exists but its capacity is smaller than the requested size in the PVC. Provision a larger PV or decrease the PVC request.

waiting for first consumer to be created before binding NORMAL BEHAVIOR: The StorageClass has volumeBindingMode: WaitForFirstConsumer . The PVC will remain Pending until a Pod referencing this PVC is created and scheduled. # Example of Default StorageClass Annotation

  • apiVersion: storage.k8s.io/v1
  • kind: StorageClass

metadata:

name: standard-ssd

annotations:

  • storageclass.kubernetes.io/is-default-class: "true"
  • provisioner: kubernetes.io/no-provisioner
  • volumeBindingMode: WaitForFirstConsumer
  • 3. Pod Stuck in ContainerCreating : Mount & Multi-Attach Errors

When a PVC is Bound , but the referencing pod remains stuck in ContainerCreating , inspect kubectl describe pod <pod-name> :

Failure Mode 1: Multi-Attach error for volume Events:

Warning FailedAttachVolume 15s attachdetach-controller Multi-Attach error for volume "pvc-9a8b7c": Volume is already exclusively attached to one node and cannot be attached to another

Root Cause: A ReadWriteOnce (RWO) cloud block volume (e.g., AWS EBS, GPD) can only be attached to a single virtual machine node at a time. If the previous node crashed or was abruptly partitioned without completing the detach operation, the cloud provider's control plane locks the volume to the dead node while the scheduler attempts to run the replacement pod on a new node.

Remediation:

Verify the old node status. If dead, delete the old pod with kubectl delete pod <old-pod> --force --grace-period=0 .

Inspect VolumeAttachment objects: kubectl get volumeattachments .

If the old node is conclusively gone, follow the CSI driver and storage-provider force-detach procedure. Deleting a VolumeAttachment object is not a generic cloud detach and can risk concurrent attachment or corruption; do it only when the provider runbook or task explicitly calls for it.

Failure Mode 2: CSI Node Driver Pod Unhealthy Events:

Warning FailedMount 10s kubelet MountVolume.SetUp failed for volume "pvc-xxxx" : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing /var/lib/kubelet/plugins_registry/csi.sock: connect: no such file or directory"

Root Cause: The CSI node plugin DaemonSet pod on that specific worker node has crashed or restarted, breaking the local gRPC UNIX domain socket.

Remediation: Inspect CSI DaemonSet health: kubectl get pods -n kube-system -l app=csi-driver and check logs.

4. Volume Expansion Failures ( FileSystemResizePending )

To increase the size of an existing PVC:

  • Ensure the underlying StorageClass has allowVolumeExpansion: true : apiVersion: storage.k8s.io/v1
  • kind: StorageClass

metadata:

  • name: expandable-sc
  • allowVolumeExpansion: true
  • provisioner: ebs.csi.aws.com

Edit the PVC: kubectl edit pvc data-claim and increase .spec.resources.requests.storage from 10Gi to 20Gi .

Symptom: The PVC status changes to FileSystemResizePending .

Explanation: The controller-side volume may already be larger while node-side filesystem expansion is incomplete. Many CSI/filesystem combinations support expansion while the Pod is running; if this combination requires offline expansion, restart the consuming Pod and inspect CSI and kubelet events.

5. Recovering Data from Released PersistentVolumes

When a PVC is deleted, the bound PV's fate is governed by its persistentVolumeReclaimPolicy :

Delete : Kubernetes removes the PV and asks the provisioner to delete the backing asset; completion is asynchronous and provider-dependent.

Retain : The PV is preserved, transitioning to status Released . +-----------------------------------------------------------------------------------------+

  • | RECLAIMING A 'RELEASED' PERSISTENTVOLUME |
  • | |
  • | 1. PVC DELETED ---> PV status transitions to 'Released' |
  • | (PV contains data, but cannot be bound by any new PVC because |
  • | it still holds a pointer in .spec.claimRef to the deleted PVC). |
  • | | |
  • | v |
  • | 2. SECURE OR SCRUB RETAINED DATA FOR THE INTENDED NEW CLAIMANT
  • | | |
  • | v |
  • | 3. STRIP CLAIMREF POINTER |
  • | $ kubectl patch pv pv-data -p '{"spec":{"claimRef": null}}' |
  • | | |
  • | v |
  • | 4. PV STATUS TRANSITIONS TO 'Available' |
  • | New PVCs can now successfully bind to this PV and access retained data. |
  • +-----------------------------------------------------------------------------------------+

Loading diagram... Persistent Storage Troubleshooting & Reclaim Lifecycle