Study guide

1.3 Training vs. Inference Computational Profiles

NVIDIA-Certified Associate: AI Infrastructure and Operations NCA-AIIO

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview, AI Fundamentals & Adoption Drivers

1.1 Exam Structure, Logistics & Certification Roadmap 1.2 AI, Machine Learning & Deep Learning Workloads 1.3 Training vs. Inference Computational Profiles 1.4 Why AI Adoption Accelerated: Data, Compute & Algorithms 1.5 Key AI Use Cases Across Industries

2 Chapter 2: GPU Architecture & CUDA Fundamentals

2.1 GPU vs. CPU Architecture & Parallel Execution 2.2 CUDA Programming Model & Hierarchy 2.3 GPU Memory Hierarchy & Bandwidth

3 Chapter 3: NVIDIA AI Software Stack, Solution Portfolio & Lifecycle

3.1 CUDA-X Libraries & Accelerated Computing 3.2 NVIDIA AI Enterprise (NVAIE) & NeMo Framework 3.3 TensorRT, TensorRT-LLM & NIM Microservices 3.4 The NVIDIA Solution Portfolio: Domain Platforms & Blueprints 3.5 Software Components Across the AI Development & Deployment Lifecycle

4 Chapter 4: GPU Microarchitecture, Tensor Cores & Precision

4.1 Streaming Multiprocessors & Tensor Core Evolution 4.2 Numerical Precision Formats & Transformer Engine 4.3 High-Bandwidth Memory (HBM) & Memory Reliability

5 Chapter 5: Interconnects, NVLink & GPUDirect Technologies

5.1 PCIe, NVLink Generations & NVSwitch Architecture 5.2 GPUDirect Technologies: P2P, RDMA & Storage (GDS) 5.3 Distributed Training Interconnects & NCCL Collectives

6 Chapter 6: AI Systems, Networking, Facilities & Deployment Models

6.1 NVIDIA DGX Systems & HGX Architecture 6.2 AI Data Center Networking: InfiniBand & Spectrum-X 6.3 BlueField DPUs, Power, Cooling & SuperPOD Scale 6.4 On-Premises vs. Cloud AI Infrastructure & DGX Cloud 6.5 Data Center Facility Requirements & Site Readiness

7 Chapter 7: Workload Orchestration, Scheduling & Run:ai

7.1 AI Workload Scheduling: Slurm & Kubernetes 7.2 NVIDIA Run:ai Architecture & Dynamic Orchestration 7.3 NVIDIA Container Toolkit & Kubernetes GPU Operator

8 Chapter 8: GPU Virtualization, Partitioning & Multi-Instance GPU (MIG)

8.1 Multi-Instance GPU (MIG) Architecture & Profiles 8.2 NVIDIA vGPU Software & Virtualization Models 8.3 Multi-Tenant Workload Isolation & Resource Slicing

9 Chapter 9: AI Operations — Telemetry, DCGM, Triton Serving & Cluster Management

9.1 GPU Telemetry, nvidia-smi & DCGM Architecture 9.2 DCGM Health Checks, Diagnostics & Prometheus Monitoring 9.3 Triton Inference Server Architecture & Deployment 9.4 AI Data Center Management: Base Command Manager, BMC & Out-of-Band Networks

1.3 Training vs. Inference Computational Profiles

Key Takeaways Explain This Topic AI

Training is a throughput-bound workload prioritizing aggregate FLOP/s, large batch sizes, backward pass gradient computation, and multi-GPU collective synchronization.

Inference is a latency-sensitive, SLA-governed workload executing only the forward pass, evaluated on Time to First Token (TTFT), Inter-Token Latency (ITL), and concurrency.

Large Language Model inference is split into two distinct operational phases: the compute-bound prompt prefill phase and the memory-bandwidth-bound autoregressive decoding phase.

The Roofline Model establishes hardware performance ceilings by plotting Operational Intensity (FLOPs/byte) against peak memory bandwidth and peak compute throughput.

Mixed precision execution (FP16, BF16, FP8) accelerates compute throughput on Tensor Cores and drastically reduces HBM memory footprint while preserving training stability.

Last updated: August 2026

1.3 Training vs. Inference Computational Profiles

Architectural Overview: In enterprise AI infrastructure, training and inference represent fundamentally different computational, memory, and networking paradigms. Training is an offline, throughput-oriented workload designed to learn parameters across large distributed clusters over days or weeks. Inference is an online, latency-sensitive workload designed to evaluate queries within strict Service Level Agreements (SLAs).

1. Deep Comparison: Training vs. Inference

Designing optimal GPU infrastructure requires matching hardware capabilities to the specific computational requirements of training or inference workloads.

Architectural Dimension Model Training Profile Model Inference Profile

Primary Objective Maximize aggregate computational throughput (tokens/sec, samples/sec, TFLOPS utilization) Minimize end-to-end response latency (P90/P99 latency, TTFT, ITL) while maximizing concurrency

Computational Passes Full Forward Pass + Loss Evaluation + Full Backward Pass (Backprop) + Optimizer Update Forward Pass Only (Evaluation / Activation Mapping)

FLOP Ratio per Sample ~3× Forward Pass FLOPs (1× Forward + 2× Backward pass gradient computation) 1× Forward Pass FLOPs

Latency Tolerance High tolerance (batch jobs run asynchronously for hours, days, or months) Low tolerance (interactive SLAs often require <100ms response or <20ms per token)

Batch Sizing Large batch sizes (e.g., 32–4,096 per GPU) to saturate Tensor Cores and amortize memory access Small or dynamic batch sizes (e.g., 1–32) to meet strict interactive latency targets

Memory Residency Weights + Gradients + Optimizer States (e.g., AdamW) + Stored Activations for Backprop Weights + KV Cache (for LLMs) + Small dynamic activation working buffer

Distributed Scaling Data Parallelism (DDP/FSDP), Tensor Parallelism (TP), Pipeline Parallelism (PP), Megatron-LM Tensor Parallelism (TP for large models), Pipeline Parallelism (PP), Model Replication

Numerical Precision Mixed Precision: FP32 master weights, BF16/FP16 forward/backward, FP8 Transformer Engine Quantized / Reduced Precision: FP16, BF16, FP8, INT8, INT4 (via TensorRT / vLLM)

Interconnect Demand Massive, continuous All-Reduce / Reduce-Scatter gradient synchronization across GPUs Modest communication; TP All-Reduce within single node (NVLink) or light multi-node TP

2. Memory Footprint Breakdown: Training vs. Inference

The total memory footprint allocated inside GPU High Bandwidth Memory (HBM) differs dramatically between training and inference. +-----------------------------------------------------------------------------+

  • | GPU HBM MEMORY ALLOCATION PROFILES |
  • | |
  • | [ TRAINING MEMORY ALLOCATION ] [ INFERENCE MEMORY ALLOCATION ] |
  • | +-----------------------------------+ +--------------------------------+ |
  • | | Optimizer States (AdamW: 12-16B/p)| | Model Weights (FP16/FP8: 1-2B/p)| |
  • | +-----------------------------------+ +--------------------------------+ |
  • | | Gradients (FP16/BF16: 2-4B/param) | | Dynamic KV Cache | |
  • | +-----------------------------------+ | (Grows with Context & Concur.) | |
  • | | Model Parameters (2-4B/param) | +--------------------------------+ |
  • | +-----------------------------------+ | Ephemeral Activation Buffers | |
  • | | Stored Forward Pass Activations | +--------------------------------+ |
  • | | (Required for Backpropagation) | |
  • | +-----------------------------------+ |
  • | | Working Scratchpad Memory | |
  • | +-----------------------------------+ |
  • +-----------------------------------------------------------------------------+
  • Training Memory Components

For an N N N -parameter model trained with mixed precision and the AdamW optimizer:

Model Parameters: Stored in BF16/FP16 (2 bytes per parameter) or FP32 (4 bytes per parameter).

Gradients: Stored in BF16/FP16 (2 bytes per parameter) during backward accumulation.

Optimizer States: In standard AdamW, the system maintains FP32 master weights (4 bytes), first momentum estimates (4 bytes), and second momentum variance estimates (4 bytes), requiring 12 to 16 bytes per parameter .

Forward Activations: Intermediate layer outputs generated during the forward pass must be retained in memory until the corresponding layer's backward pass calculates parameter gradients. In long-context models, activations frequently exceed model weight memory unless mitigated by Activation Checkpointing (Recomputation) .

Temporary Workspace: Buffers for cuDNN convolution plans, GEMM workspaces, and NCCL communication rings. Total Training Memory ≈ 16 N (Static Weights/States) + Memory Activations + Memory Workspace \text{Total Training Memory} \approx 16N \text{ (Static Weights/States)} + \text{Memory}_{\text{Activations}} + \text{Memory}_{\text{Workspace}} Total Training Memory ≈ 16 N (Static Weights/States) + Memory Activations ​ + Memory Workspace ​

Inference Memory Components

Inference eliminates gradients, optimizer states, and stored backward activations entirely:

Model Weights: Static baseline footprint. For example, an unquantized 70B parameter model in FP16 requires 70 × 10 9 × 2 bytes ≈ 140 GB 70 \times 10^9 \times 2 \text{ bytes} \approx 140 \text{ GB} 70 × 1 0 9 × 2 bytes ≈ 140 GB of HBM. In FP8, this drops to ≈ 70 GB \approx 70 \text{ GB} ≈ 70 GB .

Key-Value (KV) Cache (LLMs): Autoregressive models cache previous token Key and Value projection vectors to avoid redundant attention recalculations. The KV cache size per token across batch size b b b , context length s s s , number of layers l l l , number of KV heads h k v h_{kv} h k v ​ , and head dimension d d d is: Memory KV = 2 × b × s × l × h k v × d × bytes per element \text{Memory}_{\text{KV}} = 2 \times b \times s \times l \times h_{kv} \times d \times \text{bytes per element} Memory KV ​ = 2 × b × s × l × h k v ​ × d × bytes per element In multi-user serving environments, the KV cache dynamically expands and can easily consume 50–70% of available GPU memory.

Activation Working Memory: Minimal ephemeral memory allocated only for the current token or batch forward evaluation, immediately overwritten by subsequent layers.

3. Deconstructing LLM Inference: Prefill vs. Decoding Phase

Large Language Model serving consists of two computationally distinct phases that stress different hardware subsystems within the GPU. [ USER PROMPT: N Tokens ]

  • │
  • ▼
  • ┌────────────────────────────────────────────────────────────────────────┐
  • │ PHASE 1: PREFILL (PROMPT PROCESSING) │
  • │ - Ingests all input tokens concurrently in parallel │
  • │ - Dense Matrix-Matrix Multiplication (GEMM) │
  • │ - Arithmetic Intensity: HIGH (Hundreds of FLOPs/byte) │
  • │ - Primary Hardware Bound: COMPUTE-BOUND (Tensor Cores) │
  • │ - Key Performance Metric: Time to First Token (TTFT) │
  • │ - Output: Initial token + Populated Initial KV Cache │
  • └───────────────────────────────────┬────────────────────────────────────┘
  • │
  • ▼
  • ┌────────────────────────────────────────────────────────────────────────┐
  • │ PHASE 2: DECODING (AUTOREGRESSIVE GENERATION) │
  • │ - Generates tokens sequentially, one token at a time │
  • │ - Matrix-Vector Multiplication (GEMV) │
  • │ - Arithmetic Intensity: LOW (Single-digit FLOPs/byte) │
  • │ - Primary Hardware Bound: MEMORY-BANDWIDTH-BOUND (HBM Streaming) │
  • │ - Key Performance Metric: Inter-Token Latency (ITL) / TPOT │
  • │ - Mechanism: Reads entire model weight matrix from HBM for EACH token│
  • └────────────────────────────────────────────────────────────────────────┘
  • Detailed Phase Comparison
  • Operational Characteristic Phase 1: Prefill (Context Phase) Phase 2: Decoding (Generation Phase)
  • Input Size Entire user prompt ( N N N tokens simultaneously) Single generated token ( 1 1 1 token per step)
  • Mathematical Operation Matrix-Matrix Multiplication (GEMM) Matrix-Vector Multiplication (GEMV)
  • Hardware Bottleneck Compute-Bound (Tensor Core FLOPS limit) Memory-Bandwidth-Bound (HBM read speed)
  • Arithmetic Intensity High (>100 FLOPs/byte) Very Low (<10 FLOPs/byte at batch size = 1)
  • Critical SLA Metric Time to First Token (TTFT) Time Per Output Token (TPOT) / Inter-Token Latency

Optimization Tactics Chunked prefill, FlashAttention, Tensor Parallelism Dynamic batching (vLLM/Triton), PagedAttention, Speculative Decoding, FP8/INT4 weight quantization

4. The Roofline Model & Operational Intensity

The Roofline Model is a foundational performance model used by systems architects to diagnose whether an AI kernel or application is bounded by GPU compute capacity (Tensor Core FLOPs) or memory subsystem bandwidth (HBM bytes/sec). Attainable Performance (TFLOPS)

  • ▲
  • │ /──────────────────────── Peak Compute Ceiling
  • │ / (e.g., H100 FP16: 989 TFLOPS)
  • │ / COMPUTE-BOUND REGION
  • │ / (Performance limited by Tensor Cores)
  • │ /│
  • │ / │
  • │ / │
  • │ MEMORY-BOUND / │
  • │ REGION / │
  • │ (Performance / │
  • │ limited by / │
  • │ HBM Bandwidth/ │
  • │ / │
  • │ / │
  • │ / │
  • │ / │
  • └───────────┴────────────┴───────────────────────────► Operational Intensity
  • ▲ (FLOPs / Byte)
  • │
  • RIDGE POINT (Knee)
  • = Peak Compute / Peak Memory Bandwidth
  • Key Mathematical Formulations

Operational Intensity (Arithmetic Intensity): The ratio of total floating-point operations executed to the total bytes of data transferred to/from device memory (HBM): Operational Intensity = Total Floating Point Operations (FLOPs) Total DRAM / HBM Bytes Transferred \text{Operational Intensity} = \frac{\text{Total Floating Point Operations (FLOPs)}}{\text{Total DRAM / HBM Bytes Transferred}} Operational Intensity = Total DRAM / HBM Bytes Transferred Total Floating Point Operations (FLOPs) ​

Attainable Performance Ceiling: Attainable Performance = min ⁡ ( Peak Compute Capacity , Operational Intensity × Memory Bandwidth ) \text{Attainable Performance} = \min\left(\text{Peak Compute Capacity}, \text{Operational Intensity} \times \text{Memory Bandwidth}\right) Attainable Performance = min ( Peak Compute Capacity , Operational Intensity × Memory Bandwidth )

The Ridge Point (Knee of the Curve): The operational intensity threshold where a GPU transitions from memory-bandwidth-bound to compute-bound: Ridge Point = Peak Compute FLOP/s Peak Memory Bandwidth (Bytes/s) \text{Ridge Point} = \frac{\text{Peak Compute FLOP/s}}{\text{Peak Memory Bandwidth (Bytes/s)}} Ridge Point = Peak Memory Bandwidth (Bytes/s) Peak Compute FLOP/s ​

Practical GPU Example: NVIDIA H100 SXM5

Peak FP16 Tensor Core Compute: 989 × 10 12 FLOP/s 989 \times 10^{12} \text{ FLOP/s} 989 × 1 0 12 FLOP/s (without structured sparsity).

Peak HBM3 Memory Bandwidth: 3.35 × 10 12 Bytes/s 3.35 \times 10^{12} \text{ Bytes/s} 3.35 × 1 0 12 Bytes/s ( 3.35 TB/s 3.35 \text{ TB/s} 3.35 TB/s ).

Ridge Point Calculation: Ridge Point = 989 × 10 12 3.35 × 10 12 ≈ 295.2 FLOPs/Byte \text{Ridge Point} = \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295.2 \text{ FLOPs/Byte} Ridge Point = 3.35 × 1 0 12 989 × 1 0 12 ​ ≈ 295.2 FLOPs/Byte

Operational Takeaway:

If an operation (such as single-token LLM decoding or DLRM embedding table lookup) achieves an operational intensity of only 15 FLOPs/Byte 15 \text{ FLOPs/Byte} 15 FLOPs/Byte (well below 295.2 295.2 295.2 ), it resides squarely in the memory-bound region . Upgrading compute cores will yield zero performance gain; only increasing memory bandwidth or batch size will improve throughput.

If an operation (such as large-batch training GEMM or prompt prefill) achieves an operational intensity of 500 FLOPs/Byte 500 \text{ FLOPs/Byte} 500 FLOPs/Byte (well above 295.2 295.2 295.2 ), it resides in the compute-bound region , fully saturating Tensor Cores.

5. Precision Formats & Numerical Representation

Modern accelerated computing leverages reduced-precision arithmetic formats to maximize Tensor Core execution speed and reduce memory consumption while preserving numerical gradient stability.

Format Total Bits Bit Layout (Sign : Exponent : Mantissa) Dynamic Range Primary Advantages & Applications

FP32 32 1 : 8 : 23 1 : 8 : 23 1 : 8 : 23 Large (~10^±38) Master optimizer weights, loss scaling, stable reduction operations

FP16 16 1 : 5 : 10 1 : 5 : 10 1 : 5 : 10 Limited (~10^±5) High precision mantissa; requires loss scaling to prevent underflow during backpropagation

BF16 16 1 : 8 : 7 1 : 8 : 7 1 : 8 : 7 Same as FP32 (~10^±38) Preserves FP32 dynamic range; standard format for deep learning training without loss scaling

FP8 (E4M3) 8 1 : 4 : 3 1 : 4 : 3 1 : 4 : 3 Higher precision within FP8 Forward pass GEMMs, weights, and activations in NVIDIA Transformer Engine

FP8 (E5M2) 8 1 : 5 : 2 1 : 5 : 2 1 : 5 : 2 Higher dynamic range within FP8 Backward pass gradient accumulation where wide exponent range is critical

INT8 / INT4 8 / 4 Integer / Quantized scale Fixed discrete High-throughput low-latency inference quantization (via TensorRT)

Loading diagram... GPU Roofline Model Architecture

Operational Intensity Comparison (FLOPs / Byte)