Study guide

8.3 Multi-Tenant Workload Isolation & Resource Slicing

NVIDIA-Certified Associate: AI Infrastructure and Operations NCA-AIIO

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview, AI Fundamentals & Adoption Drivers

1.1 Exam Structure, Logistics & Certification Roadmap 1.2 AI, Machine Learning & Deep Learning Workloads 1.3 Training vs. Inference Computational Profiles 1.4 Why AI Adoption Accelerated: Data, Compute & Algorithms 1.5 Key AI Use Cases Across Industries

2 Chapter 2: GPU Architecture & CUDA Fundamentals

2.1 GPU vs. CPU Architecture & Parallel Execution 2.2 CUDA Programming Model & Hierarchy 2.3 GPU Memory Hierarchy & Bandwidth

3 Chapter 3: NVIDIA AI Software Stack, Solution Portfolio & Lifecycle

3.1 CUDA-X Libraries & Accelerated Computing 3.2 NVIDIA AI Enterprise (NVAIE) & NeMo Framework 3.3 TensorRT, TensorRT-LLM & NIM Microservices 3.4 The NVIDIA Solution Portfolio: Domain Platforms & Blueprints 3.5 Software Components Across the AI Development & Deployment Lifecycle

4 Chapter 4: GPU Microarchitecture, Tensor Cores & Precision

4.1 Streaming Multiprocessors & Tensor Core Evolution 4.2 Numerical Precision Formats & Transformer Engine 4.3 High-Bandwidth Memory (HBM) & Memory Reliability

5 Chapter 5: Interconnects, NVLink & GPUDirect Technologies

5.1 PCIe, NVLink Generations & NVSwitch Architecture 5.2 GPUDirect Technologies: P2P, RDMA & Storage (GDS) 5.3 Distributed Training Interconnects & NCCL Collectives

6 Chapter 6: AI Systems, Networking, Facilities & Deployment Models

6.1 NVIDIA DGX Systems & HGX Architecture 6.2 AI Data Center Networking: InfiniBand & Spectrum-X 6.3 BlueField DPUs, Power, Cooling & SuperPOD Scale 6.4 On-Premises vs. Cloud AI Infrastructure & DGX Cloud 6.5 Data Center Facility Requirements & Site Readiness

7 Chapter 7: Workload Orchestration, Scheduling & Run:ai

7.1 AI Workload Scheduling: Slurm & Kubernetes 7.2 NVIDIA Run:ai Architecture & Dynamic Orchestration 7.3 NVIDIA Container Toolkit & Kubernetes GPU Operator

8 Chapter 8: GPU Virtualization, Partitioning & Multi-Instance GPU (MIG)

8.1 Multi-Instance GPU (MIG) Architecture & Profiles 8.2 NVIDIA vGPU Software & Virtualization Models 8.3 Multi-Tenant Workload Isolation & Resource Slicing

9 Chapter 9: AI Operations — Telemetry, DCGM, Triton Serving & Cluster Management

9.1 GPU Telemetry, nvidia-smi & DCGM Architecture 9.2 DCGM Health Checks, Diagnostics & Prometheus Monitoring 9.3 Triton Inference Server Architecture & Deployment 9.4 AI Data Center Management: Base Command Manager, BMC & Out-of-Band Networks

8.3 Multi-Tenant Workload Isolation & Resource Slicing

Key Takeaways Explain This Topic AI

Modern GPU multi-tenancy encompasses five distinct sharing mechanisms: Bare-Metal Dedicated GPUs, Multi-Instance GPU (MIG), NVIDIA vGPU, CUDA Multi-Process Service (MPS), and Kubernetes Time-Slicing.

MIG provides hardware-level fault isolation: an out-of-memory (OOM) error, illegal memory dereference, or kernel crash in one MIG slice terminates only that instance, leaving adjacent slices unaffected.

CUDA Multi-Process Service (MPS) allows multiple host processes to multiplex onto a single GPU with reduced context-switching overhead, but shares address space and lacks hardware fault isolation.

Administrators configure MIG lifecycle via the nvidia-smi command-line utility using flags such as -mig 1 to enable MIG mode, -cgi to create GPU instances, and -cci to create compute instances.

In cloud-native Kubernetes environments, the NVIDIA GPU Operator and NVIDIA MIG Manager automate dynamic re-partitioning of GPUs via declarative ConfigMaps without requiring manual host-level intervention.

Last updated: August 2026

8.3 Multi-Tenant Workload Isolation & Resource Slicing

Architectural Mandate: As AI models proliferate across enterprise business units, infrastructure engineers must maximize hardware utilization across expensive accelerator fleets while enforcing strict security, performance, and fault isolation boundaries. Selecting the appropriate GPU partitioning mechanism requires balancing multi-tenant density , administrative complexity , and Quality of Service (QoS) guarantees .

1. Comprehensive Comparison Matrix of GPU Sharing Mechanisms

Enterprise AI infrastructure supports five distinct methodologies for allocating and sharing GPU compute and memory resources: SPECTRUM OF GPU MULTI-TENANCY & SHARING MECHANISMS

[Software Multiplexing] ────────────────────────────────────────► [Hardware Silicon Isolation]

  • K8s Time-Slicing CUDA MPS NVIDIA vGPU Multi-Instance GPU Bare-Metal
  • (Device Plugin) (Client/Server) (Hypervisor VM) (MIG) Dedicated GPU
  • ┌───────────────────┐┌─────────────────┐┌───────────────────┐┌───────────────────┐┌───────────────┐
  • │* Oversubscription ││* Shared Context ││* VM Isolation ││* Silicon Slicing ││* 100% Compute │
  • │* Zero Memory QoS ││* Active Threads ││* Time-Slice or MIG││* Dedicated SM/HBM ││* 100% Memory │
  • │* High Crash Risk ││* No Fault Isolat││* Enterprise Mgmt ││* Full Fault Isolat││* Full Scale-Up│
  • └───────────────────┘└─────────────────┘└───────────────────┘└───────────────────┘└───────────────┘
  • Detailed Technical Comparison Matrix

Technical Dimension Bare-Metal Dedicated GPU Multi-Instance GPU (MIG) NVIDIA vGPU (Time-Sliced) CUDA Multi-Process Service (MPS) Kubernetes Time-Slicing

Partitioning Layer Physical GPU (1:1) Silicon ASIC Hardware Hypervisor Kernel ( mdev ) CUDA Driver / IPC Server K8s Device Plugin

Compute Isolation 100% Dedicated Dedicated GPCs/SMs Time-Sliced SM Scheduling Shared SMs (Thread % Cap) Time-Sliced SM Scheduling

Memory Allocation Full Physical HBM Dedicated HBM Slices Fixed Virtual Framebuffer Shared HBM (Volta+ Limit) Shared Physical VRAM

Memory Bandwidth QoS Fully Dedicated Fully Dedicated (Crossbar) Contended / Shared Contended / Shared Contended / Shared

Fault Isolation Isolated to Node 100% Hardware Isolated Hypervisor VM Boundary None (Server Crash Risk) None (OOM Kills Node GPU)

Context Switch Overhead None None (Parallel Execution) Millisecond Time Slicing Microsecond IPC Multiplex Millisecond Time Slicing

Max Density per GPU 1 Workload Up to 7 Instances Up to 32–64 VMs Hundreds of Processes Up to 48+ Pods

Primary Use Case Large LLM Training / NCCL Multi-Tenant AI Inference Virtual Desktops & VMs MPI / HPC Batch Jobs Light Dev / CI-CD Pods

2. Security Boundaries, Fault Isolation & Memory Protection

Understanding the failure modes and security boundaries of each sharing mechanism is critical for designing multi-tenant enterprise platforms: +---------------------------------------------------------------------------------------------------+

  • | FAULT ISOLATION IN MULTI-TENANT SHARING |
  • | |
  • | SCENARIO: Tenant A executes an invalid pointer dereference or triggers an Out-Of-Memory (OOM) |
  • | |
  • | A. IN SOFTWARE SHARING (MPS / K8s Time-Slicing): |
  • | ┌──────────────┐ ┌──────────────┐ |
  • | │ TENANT A │ │ TENANT B │ |
  • | │ (OOM Crash!) │ │ (Inference) │ |
  • | └──────┬───────┘ └──────┬───────┘ |
  • | │ │ |
  • | ▼ ▼ |
  • | ══════════════════════════════════════════ |
  • | SHARED GPU CONTEXT / DRIVER MEMORY ENGINE ──► CUDA EXCEPTION THROWS (GPU RESET) |
  • | ══════════════════════════════════════════ |
  • | * Tenant B's active inference job CRASHES |
  • | * Entire GPU drops offline for recovery |
  • | |
  • | B. IN HARDWARE SILICON SHARING (MIG): |
  • | ┌──────────────┐ ┌──────────────┐ |
  • | │ MIG SLICE 0 │ │ MIG SLICE 1 │ |
  • | │ (Tenant A) │ │ (Tenant B) │ |
  • | │ (OOM Crash!) │ │ (Inference) │ |
  • | └──────┬───────┘ └──────┬───────┘ |
  • | │ (Isolated Fault) │ |
  • | ▼ ▼ |
  • | ┌──────────────┐ ┌──────────────┐ |
  • | │ MMU / Log 0 │ │ MMU / Log 1 │ |
  • | ├──────────────┤ ├──────────────┤ |
  • | │ 10GB HBM (0) │ │ 20GB HBM (1) │ |
  • | └──────────────┘ └──────────────┘ |
  • | * Slice 0 context resets safely * Slice 1 experiences ZERO interruption |
  • | * Hardware MMU traps invalid address * 100% Memory & Compute SLA maintained |
  • +---------------------------------------------------------------------------------------------------+
  • 1. Fault Isolation in Multi-Instance GPU (MIG)

Independent Hardware Contexts: Each MIG instance operates with dedicated memory management units, page directory pointers, interrupt lines, and hardware error logging registers.

Error Containment: If a tenant container encounters a segmentation fault, out-of-memory condition, or illegal memory access, the GPU hardware traps the fault strictly within that instance. The host driver resets the affected MIG instance without issuing a global device reset, allowing neighboring instances to continue running with zero downtime or latency degradation.

2. Failure Propagation in CUDA Multi-Process Service (MPS)

Shared Memory Address Space: In CUDA MPS, multiple client processes connect to a centralized MPS control daemon, sharing the same underlying CUDA driver context on the GPU.

Uncontained Failures: While MPS allows fine-grained SM allocation ( CUDA_MPS_ACTIVE_THREAD_PERCENTAGE ), an unhandled fatal exception in a single client process can crash the shared MPS server daemon, corrupting the execution state for all co-located clients.

3. Memory Protection in Kubernetes Time-Slicing

No Memory Quota Enforcement: Kubernetes time-slicing advertises multiple virtual GPUs from a single physical device. However, all containers access the same physical global memory. If Pod A experiences a memory leak, it can allocate 100% of the GPU's VRAM, causing Pod B to immediately fail with a CUDA out-of-memory error.

3. Operational Administration & CLI Workflows ( nvidia-smi mig )

System administrators configure, inspect, and manage the lifecycle of MIG instances using the nvidia-smi command-line utility.

Step 1: Query Current MIG State & Enable MIG Mode

By default, enterprise GPUs boot with MIG mode disabled. Enabling MIG mode requires administrative root privileges and requires that no active compute processes are running on the target GPU: # Check MIG mode status on all GPUs

nvidia-smi --query-gpu=index,name,mig.mode.current --format=csv

  • # Enable MIG mode on GPU 0 (requires root)
  • sudo nvidia-smi -i 0 -mig 1

Operational Tip: If active processes or display servers are attached to the GPU, nvidia-smi -mig 1 will fail. On modern Ampere/Hopper systems with dynamic driver reload, a reboot or GPU reset ( sudo nvidia-smi --gpu-reset -i 0 ) applies the setting.

Step 2: List Available GPU Instance Profiles

To determine the valid profile configurations and their associated Profile IDs for the target GPU: # List all supported GPU Instance Profiles on GPU 0

nvidia-smi mig -lgip -i 0

Example Output on NVIDIA A100-SXM4-80GB: +--------------------------------------------------------------------------+

  • | GPU instance profiles: |
  • | GPU ID Name Shared Memory (MB) Total Memory (MB) CE ME|
  • |==========================================================================|
  • | 0 0 7g.80gb 81920 81250 7 7|
  • | 0 5 4g.40gb 40960 40625 4 4|
  • | 0 9 3g.40gb 40960 40625 3 3|
  • | 0 14 2g.20gb 20480 20312 2 2|
  • | 0 19 1g.10gb 10240 10156 1 1|
  • +--------------------------------------------------------------------------+
  • Step 3: Create GPU Instances (GI) and Compute Instances (CI)

Administrators can create GPU Instances using Profile IDs or profile names. Passing the -C flag automatically provisions matching Compute Instances within each created GI: # Create a 3g.40gb instance (Profile ID 9) and automatically create CI (-C)

sudo nvidia-smi mig -cgi 9 -C -i 0

  • # Create two 2g.20gb instances (Profile ID 14) and one 1g.10gb instance (ID 19)
  • sudo nvidia-smi mig -cgi 14,14,19 -C -i 0
  • Step 4: List and Inspect Provisioned Instances
  • To view active MIG instances and their unique device UUIDs: # List created GPU instances on all GPUs
  • nvidia-smi mig -lgi
  • # Standard nvidia-smi display now shows partitioned MIG instances
  • nvidia-smi
  • Step 5: Target a Specific MIG Instance in Applications / Docker

To expose a specific MIG instance to an application, export its unique MIG device UUID via CUDA_VISIBLE_DEVICES : # Find the UUID from nvidia-smi -L

# Example: MIG-GPU-9b5f1234-abcd-ef01-2345-6789abcdef01/1/0

  • # Expose the instance to a local application
  • export CUDA_VISIBLE_DEVICES=MIG-GPU-9b5f1234-abcd-ef01-2345-6789abcdef01/1/0
  • python3 serve_model.py --model llama-3-8b-instruct
  • # Run a Docker container mapped strictly to the MIG instance
  • docker run --gpus '"device=MIG-GPU-9b5f1234-abcd-ef01-2345-6789abcdef01/1/0"' \
  • v /models:/models nvcr.io/nvidia/tritonserver:24.06-py3 tritonserver --model-repository=/models
  • Step 6: Teardown and Disabling MIG Mode
  • To destroy instances and return the GPU to monolithic mode: # Destroy all Compute Instances (CI) on GPU 0
  • sudo nvidia-smi mig -dci -i 0
  • # Destroy all GPU Instances (GI) on GPU 0
  • sudo nvidia-smi mig -dgi -i 0
  • # Disable MIG mode on GPU 0
  • sudo nvidia-smi -i 0 -mig 0
  • 4. Automated Dynamic Slicing with NVIDIA MIG Manager in Kubernetes

Manually executing nvidia-smi mig commands across hundreds of Kubernetes cluster nodes is operationally impractical. The NVIDIA MIG Manager component (integrated into the NVIDIA GPU Operator ) provides automated, declarative, dynamic MIG lifecycle management. KUBERNETES DYNAMIC MIG ORCHESTRATION

  • ┌────────────────────────────────────────────────────────────────────────┐
  • │ 1. Declarative Node Label Applied (e.g. mig.config=all-1g.10gb) │
  • └───────────────────────────────────┬────────────────────────────────────┘
  • │ Node Label Detected
  • ▼
  • ┌────────────────────────────────────────────────────────────────────────┐
  • │ 2. NVIDIA MIG Manager (mig-parted daemon on host) │
  • │ - Automatically cordons & drains active pods from node │
  • │ - Invokes NVML API to teardown existing MIG instances │
  • │ - Re-partitions GPU to match new ConfigMap specification │
  • └───────────────────────────────────┬────────────────────────────────────┘
  • │ Reconfiguration Complete
  • ▼
  • ┌────────────────────────────────────────────────────────────────────────┐
  • │ 3. NVIDIA Kubernetes Device Plugin & GPU Feature Discovery (GFD) │
  • │ - Discovers new MIG UUIDs │
  • │ - Advertises Extended Resources to Kubelet: │
  • │ nvidia.com/mig-1g.10gb: 7 │
  • └───────────────────────────────────┬────────────────────────────────────┘
  • │ Uncordon Node
  • ▼
  • ┌────────────────────────────────────────────────────────────────────────┐
  • │ 4. Kube-Scheduler Dispatches Pending AI Pods to Specific MIG Slices │
  • └────────────────────────────────────────────────────────────────────────┘
  • Declarative MIG Configuration via ConfigMap

The MIG Manager uses a Kubernetes ConfigMap defining standardized MIG geometry templates (e.g., all-1g.10gb , all-2g.20gb , all-3g.40gb , all-balanced ): apiVersion: v1

kind: ConfigMap

metadata:

  • name: default-mig-parted-config
  • namespace: gpu-operator

data:

  • config.yaml: |
  • version: v1

mig-configs:

all-1g.10gb:

  • devices: [all]
  • mig-enabled: true

mig-devices:

"1g.10gb": 7

all-2g.20gb:

  • devices: [all]
  • mig-enabled: true

mig-devices:

  • "2g.20gb": 3
  • "1g.10gb": 1

all-3g.40gb:

  • devices: [all]
  • mig-enabled: true

mig-devices:

  • "3g.40gb": 2
  • "1g.10gb": 1
  • Applying Dynamic Configurations to Cluster Nodes

To reconfigure an entire 8-GPU node to host 56x 1g.10gb inference microservices, an administrator simply applies a label to the Kubernetes node: # Trigger dynamic re-partitioning across all GPUs on worker node 'gpu-node-01'

  • kubectl label node gpu-node-01 nvidia.com/mig.config=all-1g.10gb --overwrite
  • Scheduling Pods onto MIG Slices via Extended Resources

When MIG mode is active, the NVIDIA Kubernetes Device Plugin registers MIG slices as extended resources with the Kubelet. Application developers request MIG slices directly in their Pod deployment manifests: apiVersion: v1

kind: Pod

metadata:

  • name: embedding-service
  • namespace: ai-inference

spec:

containers:

  • name: bge-embedding-worker
  • image: nvcr.io/nvidia/pytorch:24.06-py3
  • command: ["python3", "serve_embeddings.py"]

resources:

limits:

nvidia.com/mig-1g.10gb: 1 # Requests exactly one 1g.10gb hardware slice

requests:

  • nvidia.com/mig-1g.10gb: 1
  • cpu: "2"
  • memory: "8Gi"

Loading diagram... Multi-Tenant GPU Isolation & Failure Propagation Spectrum

GPU Sharing Mechanisms: Fault Isolation vs. Multi-Tenant Density