Study guide
6.6 Workload Scheduling Failures & Resource Constraint Resolution
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
6.6 Workload Scheduling Failures & Resource Constraint Resolution
Key Takeaways Explain This Topic AI
Pods remain stuck in the 'Pending' phase with a 'FailedScheduling' event when the kube-scheduler cannot find a node satisfying all filtering predicates (Filter phase).
Common scheduling predicate failures include: 'Insufficient cpu' or 'Insufficient memory' (resource requests exceed node allocatable capacity), 'node(s) didn't match Pod's node affinity/selector', and 'node(s) had untolerated taint'.
Taints and Tolerations enforce node-level exclusion: 'NoSchedule' prevents new pods from scheduling, 'PreferNoSchedule' tries to avoid placement, and 'NoExecute' evicts existing non-tolerating pods immediately.
PersistentVolume scheduling constraints (e.g., 'VolumeZoneConflict' or 'unbound immediate PersistentVolumeClaims') prevent pods from binding to nodes outside the storage volume's availability zone.
Administrators can bypass the kube-scheduler entirely during emergency maintenance by explicitly setting 'spec.nodeName: <target-node>' in the pod manifest.
Last updated: August 2026
6.6 Workload Scheduling Failures & Resource Constraint Resolution
The kube-scheduler is responsible for assigning unscheduled pods ( spec.nodeName == "" ) to the most appropriate worker node in the cluster. When scheduling constraints cannot be satisfied, pods remain stuck in the Pending state indefinitely.
Diagnosing scheduling failures requires understanding the scheduler's internal evaluation pipeline, interpreting predicate failure messages, and resolving conflicting resource requests, affinity rules, taints, tolerations, and storage topology constraints.
1. The Two-Phase Scheduling Algorithm
When a pod is submitted to the API server, the scheduler runs two sequential phases to select a target node: +-----------------------------------------------------------------------------------------+
- | KUBE-SCHEDULER TWO-PHASE PIPELINE |
- | |
- | [UNSCHEDULED POD] (spec.nodeName is empty) |
- | | |
- | v |
- | [PHASE 1: FILTERING (PREDICATES)] |
- | Evaluates hard requirements. Drops nodes that cannot host the pod: |
- | - NodeResourcesFit (Checks CPU, Memory, Inodes, Pod count against Allocatable) |
- | - NodeName / NodeSelector / NodeAffinity (Label matching) |
- | - PodToleratesNodeTaints (Taint / Toleration compatibility) |
- | - PodTopologySpread / InterPodAffinity (Anti-affinity rules) |
- | - VolumeBinding / VolumeZoneLimits (Storage availability zone matching) |
- | | |
- | +---> [0 NODES SURVIVED] ===> Emit 'FailedScheduling' Event -> Pod remains PENDING
- | | |
- | v [1+ NODES SURVIVED] |
- | [PHASE 2: SCORING (PRIORITIES)] |
- | Ranks surviving nodes on a 0–100 scale: |
- | - NodeResourcesBalancedAllocation (Spreads resource usage evenly) |
- | - ImageLocalityPriority (Prefers nodes with container image already cached) |
- | - PreferredDuringScheduling Node/Pod Affinities |
- | | |
- | v |
- | [BINDING] (Highest scoring node selected -> Pod.spec.nodeName committed to etcd) |
- +-----------------------------------------------------------------------------------------+
- 2. Diagnosing FailedScheduling Events
The primary diagnostic tool for scheduling failures is kubectl describe pod <pod-name> . Inspect the Events table at the bottom of the output: kubectl describe pod pending-app
Common Predicate Failure Patterns & Root Causes:
Pattern 1: Insufficient Compute Resources ( NodeResourcesFit ) Events:
- Type Reason Age From Message
- --- ------ ---- ---- -------
Warning FailedScheduling 12s default-scheduler 0/4 nodes available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 Insufficient memory.
Root Cause: The pod's .spec.containers[].resources.requests.memory exceeds the allocatable memory capacity of all eligible worker nodes.
Key Distinction: Scheduling evaluates requests , NOT limits . Even if a node has 16GB of actual idle RAM, if existing pods have already reserved 15GB in requests on a 16GB node, a new pod requesting 2GB will be rejected.
Pattern 2: NodeSelector & NodeAffinity Mismatches Warning FailedScheduling 8s default-scheduler 0/3 nodes available: 3 node(s) didn't match Pod's node affinity/selector.
Root Cause: The pod specifies a nodeSelector or requiredDuringSchedulingIgnoredDuringExecution node affinity requiring labels (e.g., disktype: ssd or topology.kubernetes.io/zone: us-east-1a ) that are not present on any worker node. # Check existing labels on all nodes
kubectl get nodes --show-labels
- # Add the missing label to a worker node to resolve the scheduling block
- kubectl label node node02 disktype=ssd
Pattern 3: Taints & Tolerations Rejections Warning FailedScheduling 5s default-scheduler 0/3 nodes available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {dedicated: gpu}.
Root Cause: Worker nodes have been tainted (e.g., dedicated=gpu:NoSchedule ), but the pod manifest lacks the corresponding tolerations block.
3. Taints, Tolerations & Node Cordoning Mechanics
Taints allow a node to repel a set of pods unless the pod explicitly tolerates the taint. +-----------------------------------------------------------------------------------------+
- | TAINT EFFECT MATRIX |
- | |
- | +-------------------+ +------------------------------------------------------------+ |
- | | NoSchedule | | New pods without matching toleration CANNOT be scheduled. | |
- | | | | Existing pods already running on the node remain untouched.| |
- | +-------------------+ +------------------------------------------------------------+ |
- | +-------------------+ +------------------------------------------------------------+ |
- | | PreferNoSchedule | | Scheduler PREFERS to avoid placing pods without toleration | |
- | | | | on this node, but will place them if no other nodes exist. | |
- | +-------------------+ +------------------------------------------------------------+ |
- | +-------------------+ +------------------------------------------------------------+ |
- | | NoExecute | | New pods without toleration CANNOT be scheduled. | |
- | | | | EXISTING pods without toleration are IMMEDIATELY EVICTED. | |
- | +-------------------+ +------------------------------------------------------------+ |
- +-----------------------------------------------------------------------------------------+
- Taint Management Commands: # Apply a taint to worker node
- kubectl taint nodes node01 dedicated=special:NoSchedule
- # Remove a taint (note the trailing minus sign)
- kubectl taint nodes node01 dedicated=special:NoSchedule-
- # Check taints on all nodes
- kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
Pod Manifest Toleration Example: spec:
tolerations:
- key: "dedicated"
- operator: "Equal"
- value: "special"
- effect: "NoSchedule"
- 4. Storage Zone Conflicts ( VolumeZoneConflict )
In multi-availability zone cloud environments (e.g., AWS EBS, GCP Persistent Disks), cloud block storage volumes are tied to a specific physical availability zone (e.g., us-east-1a ).
If a pod binds to a PersistentVolume in us-east-1a , but the scheduler attempts to place the pod on a node in us-east-1b , the scheduler fails with VolumeZoneConflict . # StorageClass using WaitForFirstConsumer prevents premature volume binding:
- apiVersion: storage.k8s.io/v1
- kind: StorageClass
metadata:
- name: topology-aware-sc
- provisioner: ebs.csi.aws.com
- volumeBindingMode: WaitForFirstConsumer # Delays PV creation until Pod is scheduled
- 5. Manual Pod Scheduling ( spec.nodeName )
When kube-scheduler is disabled or failing during an emergency, an administrator can bypass the scheduling pipeline entirely by manually binding a pod directly to a target node. apiVersion: v1
kind: Pod
metadata:
name: emergency-debug-pod
spec:
# Directly assigns pod to node01, bypassing kube-scheduler entirely:
nodeName: node01
containers:
- name: web
- image: nginx:alpine
Note
When spec.nodeName is set, the scheduler ignores the Pod, so nodeSelector , affinity, scoring, and NoSchedule taints are not evaluated by the scheduler. The named kubelet still tries to admit the Pod and it can fail for unavailable resources; a matching NoExecute taint can eject it unless tolerated.