Study guide
7.1 AI Workload Scheduling: Slurm & Kubernetes
NVIDIA-Certified Associate: AI Infrastructure and Operations NCA-AIIO
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview, AI Fundamentals & Adoption Drivers
1.1 Exam Structure, Logistics & Certification Roadmap 1.2 AI, Machine Learning & Deep Learning Workloads 1.3 Training vs. Inference Computational Profiles 1.4 Why AI Adoption Accelerated: Data, Compute & Algorithms 1.5 Key AI Use Cases Across Industries
2 Chapter 2: GPU Architecture & CUDA Fundamentals
2.1 GPU vs. CPU Architecture & Parallel Execution 2.2 CUDA Programming Model & Hierarchy 2.3 GPU Memory Hierarchy & Bandwidth
3 Chapter 3: NVIDIA AI Software Stack, Solution Portfolio & Lifecycle
3.1 CUDA-X Libraries & Accelerated Computing 3.2 NVIDIA AI Enterprise (NVAIE) & NeMo Framework 3.3 TensorRT, TensorRT-LLM & NIM Microservices 3.4 The NVIDIA Solution Portfolio: Domain Platforms & Blueprints 3.5 Software Components Across the AI Development & Deployment Lifecycle
4 Chapter 4: GPU Microarchitecture, Tensor Cores & Precision
4.1 Streaming Multiprocessors & Tensor Core Evolution 4.2 Numerical Precision Formats & Transformer Engine 4.3 High-Bandwidth Memory (HBM) & Memory Reliability
5 Chapter 5: Interconnects, NVLink & GPUDirect Technologies
5.1 PCIe, NVLink Generations & NVSwitch Architecture 5.2 GPUDirect Technologies: P2P, RDMA & Storage (GDS) 5.3 Distributed Training Interconnects & NCCL Collectives
6 Chapter 6: AI Systems, Networking, Facilities & Deployment Models
6.1 NVIDIA DGX Systems & HGX Architecture 6.2 AI Data Center Networking: InfiniBand & Spectrum-X 6.3 BlueField DPUs, Power, Cooling & SuperPOD Scale 6.4 On-Premises vs. Cloud AI Infrastructure & DGX Cloud 6.5 Data Center Facility Requirements & Site Readiness
7 Chapter 7: Workload Orchestration, Scheduling & Run:ai
7.1 AI Workload Scheduling: Slurm & Kubernetes 7.2 NVIDIA Run:ai Architecture & Dynamic Orchestration 7.3 NVIDIA Container Toolkit & Kubernetes GPU Operator
8 Chapter 8: GPU Virtualization, Partitioning & Multi-Instance GPU (MIG)
8.1 Multi-Instance GPU (MIG) Architecture & Profiles 8.2 NVIDIA vGPU Software & Virtualization Models 8.3 Multi-Tenant Workload Isolation & Resource Slicing
9 Chapter 9: AI Operations — Telemetry, DCGM, Triton Serving & Cluster Management
9.1 GPU Telemetry, nvidia-smi & DCGM Architecture 9.2 DCGM Health Checks, Diagnostics & Prometheus Monitoring 9.3 Triton Inference Server Architecture & Deployment 9.4 AI Data Center Management: Base Command Manager, BMC & Out-of-Band Networks
7.1 AI Workload Scheduling: Slurm & Kubernetes
Key Takeaways Explain This Topic AI
High-Performance Computing (HPC) batch scheduling (Slurm) and Cloud-Native container orchestration (Kubernetes) represent two fundamentally distinct paradigms for managing enterprise accelerated AI clusters.
Slurm (Simple Linux Utility for Resource Management) operates via a centralized controller (slurmctld) and per-node daemons (slurmd), managing GPU accelerators through Generic Resource Scheduling (GRES), multi-factor Fair-Share prioritization, backfill scheduling, and Quality of Service (QoS) tiers.
Common Slurm submission workflows utilize sbatch for asynchronous batch execution with declarative resource directives (#SBATCH --gres=gpu:h100:8) and srun for synchronous interactive execution and parallel process launching.
Native Kubernetes kube-scheduler was architected for long-running microservices with sequential, pod-by-pod scheduling, creating critical failure modes for distributed deep learning including partial allocation deadlocks (lack of gang scheduling), GPU fragmentation, and NUMA/NVLink topology ignorance.
Modern cloud-native batch schedulers such as Volcano and Kueue extend Kubernetes with all-or-nothing (gang) scheduling, hierarchical resource queues, dynamic cohort borrowing, and automated preemption tailored for distributed PyTorch and JAX training.
Last updated: August 2026
7.1 AI Workload Scheduling: Slurm & Kubernetes
Executive Summary: Deep learning training workloads differ fundamentally from traditional enterprise web microservices. A distributed large language model (LLM) training job is not a collection of independent, loosely coupled processes; it is a tightly synchronized, all-or-nothing distributed application where hundreds or thousands of GPUs must execute simultaneously across high-speed NVLink and InfiniBand fabrics. Managing these compute-intensive workloads requires sophisticated workload managers. Organizations generally deploy one of two primary architectural paradigms: traditional HPC batch schedulers (Slurm) or cloud-native container orchestrators (Kubernetes) augmented with batch scheduling extensions such as Volcano and Kueue .
1. The Scheduling Paradigm Shift: HPC vs. Cloud-Native
To understand modern AI infrastructure operations, infrastructure engineers must contrast the design principles of High-Performance Computing (HPC) batch schedulers with cloud-native container orchestration platforms. AI WORKLOAD SCHEDULING PARADIGMS
- ┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐
- │ HIGH-PERFORMANCE COMPUTING (HPC) │ │ CLOUD-NATIVE AI PLATFORM │
- │ (e.g., Slurm) │ │ (e.g., Kubernetes) │
- ├────────────────────────────────────────┤ ├────────────────────────────────────────┤
- │ • Static Batch Queueing │ │ • Dynamic Microservices & Jobs │
- │ • Rigid All-or-Nothing Gang Allocation │ │ • Pod-by-Pod Sequential Scheduling │
- │ • Direct Hardware / Bare-Metal Access │ │ • Containerized Abstraction Layers │
- │ • Native Topology & NUMA Awareness │ │ • Requires Add-on Batch Plugins (Kueue)│
- │ • Long-Running Multi-Day Training Runs │ │ • Elastic Inference & Rapid Prototyping│
- │ • POSIX Shared Filesystem Integration │ │ • Declarative YAML / GitOps Control │
- └────────────────────────────────────────┘ └────────────────────────────────────────┘
- Architectural Comparison Matrix
Architectural Dimension Slurm Workload Manager Native Kubernetes ( kube-scheduler ) Cloud-Native Batch (K8s + Volcano/Kueue)
Primary Workload Model Batch jobs, MPI/NCCL distributed applications Long-running stateless/stateful microservices Batch jobs, PyTorchJobs, RayClusters, JobSets
- Scheduling Unit Job / Job Array / Step Pod (individual container group) PodGroup / Workload object
- All-or-Nothing (Gang) Native and mandatory Not supported natively Native via PodGroup / Admission checks
Hardware Topology Deep hardware awareness (Sockets, Cores, NUMA, GRES) Basic resource counts (Requests/Limits) NUMA-aware and topology-aware scoring
Multi-Tenancy Model Accounts, Users, Partitions, QoS, Fair-Share trees Namespaces, ResourceQuotas, PriorityClasses Hierarchical Queues, Cohort borrowing
Interconnect Mapping Direct PCIe/NVLink mapping via gres.conf Generic device plugin advertisement ( nvidia.com/gpu ) CDI and Device Plugin with Topology Manager
Preemption Model QoS priority, Requeue, Suspend/Resume PriorityClass preemption (pod-level eviction) Workload-level preemption with grace periods
2. Slurm Workload Manager Architecture & Mechanics
Slurm (Simple Linux Utility for Resource Management) is the dominant open-source workload manager across supercomputing centers and dedicated AI supercomputers (including many large-scale NVIDIA DGX SuperPOD deployments). SLURM DAEMON ARCHITECTURE
- ┌────────────────────────────────────────────────────────────────────────┐
- │ slurmctld (Central Controller) │
- │ - Job Queues & Reservations - Fair-Share Priority Engine │
- │ - Node State & Health Monitoring - Backfill Scheduling Algorithm │
- └────────────────────▲───────────────────────────────▲───────────────────┘
- │ │
- RPC Management │ │ Accounting Records
- │ │
- ┌────────────────────▼──────────────┐ ┌─────────────▼───────────────────┐
- │ slurmd (Compute Nodes) │ │ slurmdbd (Database Daemon) │
- │ - Spawns tasks via slurmd/stepd │ │ - Historical resource tracking │
- │ - Monitors local CPU/GPU/GRES │ │ - Multi-cluster user accounts │
- │ - Enforces cgroups resource limits│ │ - Fair-Share usage decay data │
- └───────────────────────────────────┘ └─────────────────────────────────┘
- Core Slurm Daemons & Components
slurmctld (Central Management Daemon): The brain of the Slurm cluster. It maintains cluster state, monitors compute node availability, processes job submissions, computes job priorities using multi-factor algorithms, and executes the scheduling engine.
slurmd (Compute Node Daemon): Runs on every compute host. It monitors local physical resources, receives job allocation instructions from slurmctld , spawns per-job container/step daemons ( slurmstepd ), attaches hardware devices via Linux cgroups , and reports node status back to the controller.
slurmdbd (Database Daemon): Secure interface to a central relational database (typically MySQL/MariaDB). It stores user accounts, project hierarchies, Quality of Service (QoS) definitions, and historical job execution logs used for billing and Fair-Share calculations.
Client CLI Utilities:
sbatch : Submits asynchronous batch job scripts to the scheduling queue.
srun : Launches interactive jobs or parallel task steps in real time.
salloc : Allocates compute resources dynamically for interactive exploration.
squeue : Queries the state of active, pending, and running jobs in the queues.
sinfo : Reports compute node and partition availability.
scontrol : Administrative tool to view and modify cluster configuration, nodes, and running jobs.
Generic Resource Scheduling (GRES) for GPUs
Slurm manages accelerators through the Generic Resource (GRES) framework. In slurm.conf and gres.conf , system administrators explicitly define the physical topology of GPUs, mapping them to specific PCIe buses, NUMA memory domains, and NVLink fabrics. # Example /etc/slurm/gres.conf on a DGX H100 node:
NodeName=dgx-h100-[01-32] Name=gpu Type=h100 File=/dev/nvidia[0-7] Cores=0-111
When a researcher submits a job requesting GPUs, Slurm leverages GRES to isolate the specific GPU device files inside the container or process namespace via Linux cgroups: #!/bin/bash
- #SBATCH --job-name=llama3-70b-finetune
- #SBATCH --nodes=4
- #SBATCH --ntasks-per-node=8
- #SBATCH --gres=gpu:h100:8
- #SBATCH --cpus-per-task=14
- #SBATCH --partition=deeplearning
- #SBATCH --qos=high_priority
- #SBATCH --time=48:00:00
- #SBATCH --output=logs/%x_%j.out
- # Launch 32 PyTorch distributed processes across 4 DGX H100 nodes
- srun --export=ALL torchrun \
- -nproc_per_node=8 \
- -nnodes=4 \
- -rdzv_id=$SLURM_JOB_ID \
- -rdzv_backend=c10d \
- -rdzv_endpoint=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1):29500 \
- train.py --config config_70b.yaml
- Advanced Slurm Scheduling Mechanics
Multi-Factor Priority Algorithm: Slurm calculates the priority of queued jobs using a weighted formula combining multiple independent factors: Job Priority = W A ⋅ Age + W F S ⋅ Fair-Share + W J S ⋅ Job Size + W P ⋅ Partition + W Q ⋅ QoS \text{Job Priority} = W_A \cdot \text{Age} + W_{FS} \cdot \text{Fair-Share} + W_{JS} \cdot \text{Job Size} + W_P \cdot \text{Partition} + W_Q \cdot \text{QoS} Job Priority = W A ⋅ Age + W FS ⋅ Fair-Share + W J S ⋅ Job Size + W P ⋅ Partition + W Q ⋅ QoS
Fair-Share Multi-Tenant Algorithm: Tracks historical GPU consumption per user, research group, and department. As an organization consumes compute hours, its Fair-Share score decreases, elevating the priority of underserved teams. A half-life decay parameter gradually discounts past usage over time.
Backfill Scheduling: In traditional First-In, First-Out (FIFO) scheduling, a massive 64-node job waiting for resources would block all incoming work, leaving idle nodes underutilized. Slurm's backfill scheduler plans a future reservation for the large job, then searches the pending queue to identify smaller, shorter jobs that can execute on idle nodes and finish before the large job's reservation window begins, maximizing cluster utilization without delaying high-priority work.
Partitions & Quality of Service (QoS): Partitions act as virtual queues with specific node subsets and access rules. QoS overlays define priority boosts, maximum wall-clock limits, GPU concurrency caps, and preemption rules (e.g., allowing a production training job to preempt an exploratory interactive run).
3. Kubernetes Native Scheduling Limitations for Deep Learning
The native Kubernetes scheduler ( kube-scheduler ) was architected around the requirements of web microservices: stateless, independently scalable containers with asynchronous lifecycles. When applied directly to distributed AI training, kube-scheduler introduces fundamental operational limitations. THE DISTRIBUTED DEADLOCK SCENARIO (LACK OF GANG SCHEDULING)
- Cluster Capacity: 2 Nodes (16 Total GPUs)
- Job 1 requires 16 GPUs (2 Nodes) | Job 2 requires 16 GPUs (2 Nodes)
- ┌───────────────────────────────────┐ ┌───────────────────────────────────┐
- │ COMPUTE NODE 01 │ │ COMPUTE NODE 02 │
- │ (8x H100 SXM5 GPUs) │ │ (8x H100 SXM5 GPUs) │
- ├───────────────────────────────────┤ ├───────────────────────────────────┤
- │ [Allocated to Job 1 - Pod 0] │ │ [Allocated to Job 2 - Pod 0] │
- │ Status: WAITING FOR PEER PODS │ │ Status: WAITING FOR PEER PODS │
- │ (Consumes 8 GPUs, Idles) │ │ (Consumes 8 GPUs, Idles) │
- └───────────────────────────────────┘ └───────────────────────────────────┘
- ▲ ▲
- └────────────── MUTUAL RESOURCE DEADLOCK ──────────────┘
- Job 1 holds Node 1 and blocks waiting for Node 2.
- Job 2 holds Node 2 and blocks waiting for Node 1.
- Result: 16 enterprise GPUs sit 100% idle indefinitely.
Core Failure Modes of Native kube-scheduler in AI Clusters
Pod-by-Pod Scheduling & Distributed Deadlocks:
kube-scheduler evaluates pods individually and sequentially without holistic awareness of multi-pod distributed jobs.
In an 8-node PyTorchJob, if only 4 nodes are available, native Kubernetes schedules 4 pods and leaves the remaining 4 pending. Because distributed PyTorch utilizes synchronous collective initialization (such as torch.distributed.init_process_group with NCCL), the running pods stall indefinitely waiting for the missing ranks to join the communication barrier. If another distributed job acquires the remaining nodes, both jobs enter a mutual resource deadlock , consuming expensive GPU hours while delivering zero compute progress.
Lack of Hardware & Interconnect Topology Awareness:
In modern multi-GPU nodes, GPUs are arranged across specific PCIe root complexes, CPU sockets, and NVLink switch crossbars. Connecting two GPUs across distinct NUMA domains or misaligned PCIe switches degrades inter-GPU memory bandwidth significantly.
Native Kubernetes treats nvidia.com/gpu: 2 as fungible scalar integers. It may assign GPU 0 and GPU 7 to the same container—forcing communication across separate CPU sockets and host PCIe buses rather than adjacent NVLink pairs.
GPU Bin-Packing vs. Spreading Deficiencies:
Native scoring algorithms ( NodeResourcesFit ) can inadvertently scatter interactive single-GPU pods across every node in a cluster, fragmenting 8-GPU nodes so that no single node retains 8 contiguous GPUs for large-scale training jobs.
4. Cloud-Native Batch Schedulers: Volcano & Kueue
To overcome native Kubernetes limitations, the cloud-native ecosystem developed purpose-built batch scheduling frameworks that bring supercomputing-grade orchestration into standard Kubernetes clusters. CLOUD-NATIVE BATCH SCHEDULERS
- ┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐
- │ VOLCANO (CNCF) │ │ KUEUE (K8s SIG-SCHEDULING) │
- ├────────────────────────────────────────┤ ├────────────────────────────────────────┤
- │ • Full custom secondary scheduler │ │ • Non-intrusive Job Queue Controller │
- │ • PodGroup Custom Resource Definition │ │ • ClusterQueue & LocalQueue CRDs │
- │ • Native Gang Scheduling engine │ │ • Workload Admission & Cohort Borrowing│
- │ • Custom Plugins (NUMA, SSH, MPI) │ │ • Integrates with PyTorchJob, Ray, Jobs│
- │ • Replaces kube-scheduler for batch │ │ • Works with native & custom schedulers│
- └────────────────────────────────────────┘ └────────────────────────────────────────┘
- Volcano Architecture & Capabilities
Volcano is a Cloud Native Computing Foundation (CNCF) batch scheduling engine engineered specifically for high-performance AI, deep learning, and big data workloads:
PodGroup CRD & Gang Scheduling: Volcano introduces the PodGroup Custom Resource Definition with a minMember parameter. Volcano will not bind any pod in a job to a node until sufficient cluster resources exist to bind all minMember pods simultaneously. If resources are insufficient, the entire job remains pending in the queue, completely eliminating distributed deadlocks.
Queue Management & Dynamic Weights: Supports multi-tenant hierarchical queues with assigned weights, minimum resource guarantees, and maximum capacity caps.
Task-Topology & NUMA Awareness: Evaluates network distance and NUMA topology when placing multi-task jobs, placing cooperating ranks on the closest possible physical nodes.
Kueue Architecture & Capabilities
Kueue is a Kubernetes-native, lightweight batch queuing controller developed by Kubernetes Special Interest Group (SIG) Scheduling. Unlike Volcano, which replaces the core scheduler, Kueue operates as an admission and job lifecycle controller that governs when jobs are admitted to the cluster: apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: enterprise-ai-queue
spec:
- namespaceSelector: {}
- cohort: research-division
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
flavors:
- name: h100-sxm5-flavor
resources:
- name: "nvidia.com/gpu"
- nominalQuota: 64
- borrowingLimit: 32
Core Constructs:
ResourceFlavor : Associates physical node labels and taints with specific hardware types (e.g., H100 SXM5 vs. A100 PCIe).
ClusterQueue : Cluster-scoped resource pool defining nominal quotas, borrowing limits, and preemption policies.
LocalQueue : Namespaced queue submitted to by data scientists, mapping directly to an upstream ClusterQueue .
Workload : Represents the resource request of a batch application (e.g., PyTorchJob, RayCluster, JobSet, or standard Batch Job).
Cohort Sharing & Resource Borrowing: Multiple ClusterQueues can join a shared Cohort . When Department A is not using its nominal quota of 64 GPUs, Department B can dynamically borrow up to its borrowingLimit . When Department A submits new work, Kueue automatically preempts Department B's borrowed workloads to reclaim the guaranteed capacity.