Study guide
1.2 AI, Machine Learning & Deep Learning Workloads
NVIDIA-Certified Associate: AI Infrastructure and Operations NCA-AIIO
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview, AI Fundamentals & Adoption Drivers
1.1 Exam Structure, Logistics & Certification Roadmap 1.2 AI, Machine Learning & Deep Learning Workloads 1.3 Training vs. Inference Computational Profiles 1.4 Why AI Adoption Accelerated: Data, Compute & Algorithms 1.5 Key AI Use Cases Across Industries
2 Chapter 2: GPU Architecture & CUDA Fundamentals
2.1 GPU vs. CPU Architecture & Parallel Execution 2.2 CUDA Programming Model & Hierarchy 2.3 GPU Memory Hierarchy & Bandwidth
3 Chapter 3: NVIDIA AI Software Stack, Solution Portfolio & Lifecycle
3.1 CUDA-X Libraries & Accelerated Computing 3.2 NVIDIA AI Enterprise (NVAIE) & NeMo Framework 3.3 TensorRT, TensorRT-LLM & NIM Microservices 3.4 The NVIDIA Solution Portfolio: Domain Platforms & Blueprints 3.5 Software Components Across the AI Development & Deployment Lifecycle
4 Chapter 4: GPU Microarchitecture, Tensor Cores & Precision
4.1 Streaming Multiprocessors & Tensor Core Evolution 4.2 Numerical Precision Formats & Transformer Engine 4.3 High-Bandwidth Memory (HBM) & Memory Reliability
5 Chapter 5: Interconnects, NVLink & GPUDirect Technologies
5.1 PCIe, NVLink Generations & NVSwitch Architecture 5.2 GPUDirect Technologies: P2P, RDMA & Storage (GDS) 5.3 Distributed Training Interconnects & NCCL Collectives
6 Chapter 6: AI Systems, Networking, Facilities & Deployment Models
6.1 NVIDIA DGX Systems & HGX Architecture 6.2 AI Data Center Networking: InfiniBand & Spectrum-X 6.3 BlueField DPUs, Power, Cooling & SuperPOD Scale 6.4 On-Premises vs. Cloud AI Infrastructure & DGX Cloud 6.5 Data Center Facility Requirements & Site Readiness
7 Chapter 7: Workload Orchestration, Scheduling & Run:ai
7.1 AI Workload Scheduling: Slurm & Kubernetes 7.2 NVIDIA Run:ai Architecture & Dynamic Orchestration 7.3 NVIDIA Container Toolkit & Kubernetes GPU Operator
8 Chapter 8: GPU Virtualization, Partitioning & Multi-Instance GPU (MIG)
8.1 Multi-Instance GPU (MIG) Architecture & Profiles 8.2 NVIDIA vGPU Software & Virtualization Models 8.3 Multi-Tenant Workload Isolation & Resource Slicing
9 Chapter 9: AI Operations — Telemetry, DCGM, Triton Serving & Cluster Management
9.1 GPU Telemetry, nvidia-smi & DCGM Architecture 9.2 DCGM Health Checks, Diagnostics & Prometheus Monitoring 9.3 Triton Inference Server Architecture & Deployment 9.4 AI Data Center Management: Base Command Manager, BMC & Out-of-Band Networks
1.2 AI, Machine Learning & Deep Learning Workloads
Key Takeaways Explain This Topic AI
Artificial Intelligence is the broad domain of computational intelligence, containing Machine Learning as a statistical subset, Deep Learning as a multi-layer neural network subset, and Generative AI as an emergent content-creation paradigm.
Deep learning model training executes through an iterative cyclic pipeline: forward pass, loss calculation via an objective function, backward pass gradient computation via the chain rule, and weight optimization using SGD, Adam, or AdamW.
Activation functions introduce non-linearities into neural networks, evolving from classical step/sigmoid to ReLU, smooth GELU, and modern gated SwiGLU architectures used in foundation LLMs.
Computer vision workloads rely heavily on 2D/3D convolutions and Vision Transformers (ViTs) with high spatial locality and dense General Matrix Multiply (GEMM) operations.
Large Language Models (LLMs) depend on multi-head self-attention with quadratic context scaling and KV caching, while Recommender Systems are bounded by massive, sparse embedding tables in system memory.
Last updated: August 2026
1.2 AI, Machine Learning & Deep Learning Workloads
Core Concept: Understanding the architectural taxonomy of modern artificial intelligence and the mathematical mechanics of neural networks is fundamental for designing and operating GPU-accelerated computing infrastructure. Different model families—such as convolutional vision networks, autoregressive large language models, and deep recommendation systems—exhibit vastly different compute, memory bandwidth, and interconnect scaling requirements.
1. The AI Taxonomy: AI vs. ML vs. DL vs. GenAI
Modern artificial intelligence is structured as a series of nested subfields, where each subsequent tier introduces greater computational complexity, specialized data structures, and distinct hardware acceleration requirements. ┌────────────────────────────────────────────────────────────────────────┐
- │ ARTIFICIAL INTELLIGENCE (AI) │
- │ Rule-based systems, expert systems, symbolic logic, search heuristics │
- │ ┌──────────────────────────────────────────────────────────────────┐ │
- │ │ MACHINE LEARNING (ML) │ │
- │ │ Statistical algorithms, feature engineering, regression, SVMs │ │
- │ │ ┌────────────────────────────────────────────────────────────┐ │ │
- │ │ │ DEEP LEARNING (DL) │ │ │
- │ │ │ Multi-layer neural networks, backpropagation, GEMM cores │ │ │
- │ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │
- │ │ │ │ GENERATIVE AI (GenAI) │ │ │ │
- │ │ │ │ Transformers, LLMs, Diffusion Models, Autoregressive │ │ │ │
- │ │ │ └──────────────────────────────────────────────────────┘ │ │ │
- │ │ └────────────────────────────────────────────────────────────┘ │ │
- │ └──────────────────────────────────────────────────────────────────┘ │
- └────────────────────────────────────────────────────────────────────────┘
- Detailed Taxonomy Comparison
Paradigm Mathematical Basis Feature Engineering Primary Hardware Bottleneck Representative Workloads
Artificial Intelligence (AI) Deterministic logic, decision trees, heuristic search Handcrafted rules and explicit conditional statements CPU instruction throughput, branch prediction Expert systems, chess engines (Stockfish), rule engines
Machine Learning (ML) Statistical optimization, gradient descent, kernel methods Manual feature extraction (PCA, TF-IDF, normalization) CPU memory bandwidth, PCIe bus transfer speeds Random Forests, XGBoost, k-Means, Logistic Regression
Deep Learning (DL) Hierarchical multi-layer neural networks, tensor transformations Automated feature representation learned directly from data GPU Tensor Core compute (GEMM), High Bandwidth Memory (HBM) ResNet-50, YOLO, BERT, Speech Recognition (Conformer)
Generative AI (GenAI) Self-attention transformers, denoising diffusion, autoregression Unsupervised tokenization and positional embeddings Inter-GPU interconnect bandwidth (NVLink), HBM capacity & bandwidth LLaMA, GPT-4, Stable Diffusion, Nemotron, Whisper
2. Neural Network Computational Mechanics
Deep neural networks process information through multidimensional tensor transformations across layers of artificial neurons. Understanding the cyclical execution flow is crucial for diagnosing GPU utilization and distributed cluster bottlenecks. ┌───────────────────────────────────────────────────────────────────┐
- │ 1. FORWARD PASS │
- │ Input X ───► [ Z = W · X + b ] ───► [ A = σ(Z) ] ───► Output ŷ │
- └─────────────────────────────────┬─────────────────────────────────┘
- │
- ▼
- ┌───────────────────────────────────────────────────────────────────┐
- │ 2. LOSS CALCULATION │
- │ Compute Error via Loss Function: L = Loss(ŷ, y_true) │
- └─────────────────────────────────┬─────────────────────────────────┘
- │
- ▼
- ┌───────────────────────────────────────────────────────────────────┐
- │ 3. BACKWARD PASS (BACKPROP) │
- │ Compute Gradients via Chain Rule: ∂L/∂W = (∂L/∂ŷ)·(∂ŷ/∂Z)·(∂Z/∂W)│
- └─────────────────────────────────┬─────────────────────────────────┘
- │
- ▼
- ┌───────────────────────────────────────────────────────────────────┐
- │ 4. OPTIMIZER WEIGHT UPDATE │
- │ Update Parameters: W_new = W_old - η · Optimizer(∂L/∂W) │
- │ (SGD, Momentum, Adam, AdamW Tracking First & Second Moments) │
- └───────────────────────────────────────────────────────────────────┘
- Step 1: Forward Pass (Inference & Activation Computation)
In the forward pass, input data X X X (represented as a batch of multidimensional tensors) propagates through consecutive network layers. Each linear layer performs a General Matrix Multiply (GEMM) followed by a bias addition: Z = W ⋅ X + b Z = W \cdot X + b Z = W ⋅ X + b
Where W W W represents the learnable weight matrix, X X X is the input tensor, and b b b is the bias vector. The linear output Z Z Z is then passed through an activation function σ ( Z ) \sigma(Z) σ ( Z ) to produce activation tensor A A A .
Step 2: Loss Function & Objective Evaluation
The model's predicted output y ^ \hat{y} y ^ is compared against the ground-truth target y y y using a domain-specific mathematical loss function:
Cross-Entropy Loss: Used for multi-class classification and autoregressive token generation in LLMs: L C E = − ∑ i y i log ( y ^ i ) L_{CE} = -\sum_{i} y_i \log(\hat{y}_i) L CE = − i ∑ y i lo g ( y ^ i )
Mean Squared Error (MSE): Used for continuous regression tasks: L M S E = 1 N ∑ i = 1 N ( y i − y ^ i ) 2 L_{MSE} = \frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2 L MSE = N 1 i = 1 ∑ N ( y i − y ^ i ) 2
Step 3: Backward Pass & Gradient Computation (Backpropagation)
Backpropagation utilizes the calculus chain rule to compute the partial derivative of the scalar loss L L L with respect to every learnable parameter (weights W W W and biases b b b ) across all layers in reverse order: ∂ L ∂ W = ∂ L ∂ A ⋅ ∂ A ∂ Z ⋅ ∂ Z ∂ W \frac{\partial L}{\partial W} = \frac{\partial L}{\partial A} \cdot \frac{\partial A}{\partial Z} \cdot \frac{\partial Z}{\partial W} ∂ W ∂ L = ∂ A ∂ L ⋅ ∂ Z ∂ A ⋅ ∂ W ∂ Z
These partial derivatives, known as gradients , indicate the direction and magnitude of parameter adjustment required to minimize total loss. The backward pass requires approximately 2× the floating-point operations (FLOPs) of the forward pass.
Step 4: Optimizer Parameter Updates
The calculated gradients are ingested by an optimization algorithm to update the model parameters. Different optimizers exhibit varying memory overheads:
Stochastic Gradient Descent (SGD): Directly updates weights along the negative gradient vector: W ← W − η ∇ W L W \leftarrow W - \eta \nabla_W L W ← W − η ∇ W L , where η \eta η is the learning rate. Low memory overhead (0 extra bytes per parameter).
SGD with Momentum: Tracks an exponential moving average of past gradients to accelerate through flat loss surfaces. Requires 4 additional bytes per parameter (FP32 momentum vector).
Adam (Adaptive Moment Estimation): Maintains running estimates of both the uncentered first moment (mean gradient m t m_t m t ) and second moment (uncentered variance v t v_t v t ) for every parameter: m t = β 1 m t − 1 + ( 1 − β 1 ) g t , v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2 m t = β 1 m t − 1 + ( 1 − β 1 ) g t , v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 Requires 8 additional bytes per parameter (two FP32 state vectors).
AdamW (Decoupled Weight Decay): The industry standard for training Transformers and LLMs. Decouples L 2 L_2 L 2 weight decay regularization from gradient updates, preventing weight decay from being distorted by the adaptive learning rate.
3. Evolution of Activation Functions
Activation functions introduce non-linear mapping capabilities, enabling deep networks to approximate complex mathematical functions beyond simple linear hyperplanes.
Activation Function Mathematical Formulation Characteristics & Trade-Offs Primary Architectural Use Case
Sigmoid / Logistic σ ( x ) = 1 1 + e − x \sigma(x) = \frac{1}{1 + e^{-x}} σ ( x ) = 1 + e − x 1 S-shaped curve bounded in ( 0 , 1 ) (0, 1) ( 0 , 1 ) . Prone to vanishing gradients during backprop when saturated. Binary classification output layers, gating mechanisms
ReLU (Rectified Linear) f ( x ) = max ( 0 , x ) f(x) = \max(0, x) f ( x ) = max ( 0 , x ) Piecewise linear; extremely fast compute on CUDA cores. Suffers from "dying ReLU" if neurons become permanently inactive. Classical CNNs (ResNet), early multi-layer perceptrons
Leaky ReLU / ELU f ( x ) = max ( α x , x ) f(x) = \max(\alpha x, x) f ( x ) = max ( αx , x ) Introduces a small positive slope α \alpha α for x < 0 x < 0 x < 0 to prevent dead neurons. Deep convolutional architectures, generative adversarial networks (GANs)
GELU (Gaussian Error Linear) f ( x ) = x ⋅ Φ ( x ) ≈ 0.5 x ( 1 + tanh ( 2 / π ( x + 0.044715 x 3 ) ) ) f(x) = x \cdot \Phi(x) \approx 0.5x(1 + \tanh(\sqrt{2/\pi}(x + 0.044715x^3))) f ( x ) = x ⋅ Φ ( x ) ≈ 0.5 x ( 1 + tanh ( 2/ π ( x + 0.044715 x 3 ))) Smooth, probabilistic non-linearity. Scales input by probability of dropping out under Gaussian distribution. Transformer backbones (BERT, GPT-2, GPT-3, ViT)
SwiGLU (Swish Gated Linear) SwiGLU ( x ) = Swish ( x W 1 ) ⊗ ( x W 2 ) \text{SwiGLU}(x) = \text{Swish}(x W_1) \otimes (x W_2) SwiGLU ( x ) = Swish ( x W 1 ) ⊗ ( x W 2 ) Gated non-linear combination using SiLU/Swish. Superior empirical convergence in foundation models at the cost of an extra GEMM. State-of-the-art LLMs (LLaMA 1/2/3, Mistral, Gemma, Nemotron)
4. Workload Computational Profiles & System Demands
Different enterprise AI workloads place distinct stresses on GPU compute, memory subsystem capacity, and distributed networking. +-----------------------------------------------------------------------------+
- | WORKLOAD COMPUTATIONAL PROFILES |
- | |
- | [ COMPUTER VISION ] [ LARGE LANGUAGE MODELS ] [ RECOMMENDER ] │
- | - 2D/3D Convolutions - Multi-Head Self-Attention - Massive Sparse│
- | - Dense GEMM Operations - Matrix Multiplication - Embedding Tbls│
- | - High Spatial Locality - KV Cache Dynamics - Low Compute / │
- | - High Arithmetic Intensity - All-Reduce / TP Comm - High DRAM Band│
- | ==> COMPUTE-BOUND ==> COMPUTE & MEMORY BOUND ==> MEMORY-BOUND│
- +-----------------------------------------------------------------------------+
- 1. Computer Vision (CV): Convolutional Networks & Vision Transformers
Mechanics: Classical CV relies on 2D/3D convolutional kernels sliding across pixel grids, implemented via im2col tensor flattening mapped to Tensor Core GEMM operations. Vision Transformers (ViTs) divide images into 16 × 16 16 \times 16 16 × 16 pixel patches treated as sequential tokens.
Computational Profile: High arithmetic intensity (FLOPs per byte). High spatial data reuse in GPU SRAM/L2 cache. Generally compute-bound during both training and batch inference.
2. Natural Language Processing & Large Language Models (NLP / LLMs)
Mechanics: Built upon the Transformer architecture utilizing Multi-Head Self-Attention (MHSA) : Attention ( Q , K , V ) = softmax ( Q K T d k ) V \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V Attention ( Q , K , V ) = softmax ( d k Q K T ) V Where Query ( Q Q Q ), Key ( K K K ), and Value ( V V V ) projections generate attention score matrices with O ( N 2 ) O(N^2) O ( N 2 ) computational and memory complexity relative to sequence length N N N .
Computational Profile: Highly bifurcated profile. Training and prompt prefill are compute-bound dense GEMMs requiring massive Tensor Core throughput and inter-GPU communication (Tensor Parallel All-Reduce). Autoregressive token decoding is memory-bandwidth bound, constrained by streaming model weights and the Key-Value (KV) cache from HBM for every single generated token.
3. Deep Learning Recommendation Models (DLRM)
Mechanics: DLRMs (e.g., Meta DLRM, NVIDIA Merlin) process two distinct data streams: dense numerical features (processed via compute-heavy Bottom MLPs) and sparse categorical features (user IDs, search terms, item IDs).
Computational Profile: Sparse features are indexed into massive Embedding Tables spanning hundreds of gigabytes to terabytes of memory. Table lookups perform memory gathers (sparse memory lookups) with very low arithmetic intensity. Scaling DLRMs requires massive GPU HBM capacity and high All-to-All network interconnects to distribute embedding tables across cluster nodes.