Study guide
5.1 PCIe, NVLink Generations & NVSwitch Architecture
NVIDIA-Certified Associate: AI Infrastructure and Operations NCA-AIIO
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview, AI Fundamentals & Adoption Drivers
1.1 Exam Structure, Logistics & Certification Roadmap 1.2 AI, Machine Learning & Deep Learning Workloads 1.3 Training vs. Inference Computational Profiles 1.4 Why AI Adoption Accelerated: Data, Compute & Algorithms 1.5 Key AI Use Cases Across Industries
2 Chapter 2: GPU Architecture & CUDA Fundamentals
2.1 GPU vs. CPU Architecture & Parallel Execution 2.2 CUDA Programming Model & Hierarchy 2.3 GPU Memory Hierarchy & Bandwidth
3 Chapter 3: NVIDIA AI Software Stack, Solution Portfolio & Lifecycle
3.1 CUDA-X Libraries & Accelerated Computing 3.2 NVIDIA AI Enterprise (NVAIE) & NeMo Framework 3.3 TensorRT, TensorRT-LLM & NIM Microservices 3.4 The NVIDIA Solution Portfolio: Domain Platforms & Blueprints 3.5 Software Components Across the AI Development & Deployment Lifecycle
4 Chapter 4: GPU Microarchitecture, Tensor Cores & Precision
4.1 Streaming Multiprocessors & Tensor Core Evolution 4.2 Numerical Precision Formats & Transformer Engine 4.3 High-Bandwidth Memory (HBM) & Memory Reliability
5 Chapter 5: Interconnects, NVLink & GPUDirect Technologies
5.1 PCIe, NVLink Generations & NVSwitch Architecture 5.2 GPUDirect Technologies: P2P, RDMA & Storage (GDS) 5.3 Distributed Training Interconnects & NCCL Collectives
6 Chapter 6: AI Systems, Networking, Facilities & Deployment Models
6.1 NVIDIA DGX Systems & HGX Architecture 6.2 AI Data Center Networking: InfiniBand & Spectrum-X 6.3 BlueField DPUs, Power, Cooling & SuperPOD Scale 6.4 On-Premises vs. Cloud AI Infrastructure & DGX Cloud 6.5 Data Center Facility Requirements & Site Readiness
7 Chapter 7: Workload Orchestration, Scheduling & Run:ai
7.1 AI Workload Scheduling: Slurm & Kubernetes 7.2 NVIDIA Run:ai Architecture & Dynamic Orchestration 7.3 NVIDIA Container Toolkit & Kubernetes GPU Operator
8 Chapter 8: GPU Virtualization, Partitioning & Multi-Instance GPU (MIG)
8.1 Multi-Instance GPU (MIG) Architecture & Profiles 8.2 NVIDIA vGPU Software & Virtualization Models 8.3 Multi-Tenant Workload Isolation & Resource Slicing
9 Chapter 9: AI Operations — Telemetry, DCGM, Triton Serving & Cluster Management
9.1 GPU Telemetry, nvidia-smi & DCGM Architecture 9.2 DCGM Health Checks, Diagnostics & Prometheus Monitoring 9.3 Triton Inference Server Architecture & Deployment 9.4 AI Data Center Management: Base Command Manager, BMC & Out-of-Band Networks
5.1 PCIe, NVLink Generations & NVSwitch Architecture
Key Takeaways Explain This Topic AI
PCIe Gen 5 x16 provides 128 GB/s bidirectional throughput (64 GB/s per direction), doubling Gen 4 bandwidth but remaining a severe communication bottleneck for distributed deep learning workloads.
NVIDIA NVLink provides high-bandwidth, energy-efficient, point-to-point GPU interconnects with native memory load/store semantics, scaling from 160 GB/s in Pascal to 1.8 TB/s per GPU in Blackwell.
NVSwitch implements a non-blocking multi-port crossbar fabric, enabling all 8 GPUs in an HGX/DGX chassis to communicate at full native line rate simultaneously without PCIe or CPU contention.
Hopper HGX H100 integrates 4 third-generation NVSwitch chips delivering 7.2 TB/s aggregate crossbar switching capacity, while Blackwell NVL72 scales NVLink across 72 GPUs in a single rack-scale domain via 18 NVLink Switch trays.
Selecting appropriate interconnect topologies directly dictates multi-GPU scaling efficiency, as communication latency and bandwidth bound intra-node Tensor Parallelism and inter-node AllReduce operations.
Last updated: August 2026
5.1 PCIe, NVLink Generations & NVSwitch Architecture
Core Concept: In modern deep learning clusters, raw computational throughput (FLOPs) is only as effective as the underlying communication fabric. As neural network models scale into hundreds of billions and trillions of parameters, the volume of intermediate activations, weight gradients, and key-value tensors exchanged between GPUs surpasses the capacity of traditional computer buses. NVIDIA NVLink and NVSwitch architectures bypass legacy Peripheral Component Interconnect Express (PCIe) bottlenecks by establishing direct, cache-coherent, high-bandwidth interconnects across accelerator silicon.
1. Host Interconnects: PCIe Gen 4 vs. PCIe Gen 5 & The DL Bottleneck
Traditional server architectures connect hardware accelerators (GPUs, DPUs, FPGAs) to the host central processing unit (CPU) and system DRAM via the PCIe (Peripheral Component Interconnect Express) bus standard. ┌────────────────────────────────────────────────────────────────────────┐
- │ TRADITIONAL PCIE SYSTEM TOPOLOGY │
- │ │
- │ ┌──────────────────┐ ┌──────────────────┐ │
- │ │ Host CPU 0 ├───────────────┤ Host CPU 1 │ │
- │ │ (NUMA Node 0) │ UPI/QPI │ (NUMA Node 1) │ │
- │ └────────┬─────────┘ └────────┬─────────┘ │
- │ │ PCIe Root Complex │ PCIe Root Complex │
- │ ┌────────┴─────────┐ ┌────────┴─────────┐ │
- │ │ PCIe Switch │ │ PCIe Switch │ │
- │ └───┬───────────┬──┘ └───┬───────────┬──┘ │
- │ │ │ │ │ │
- │ ▼ ▼ ▼ ▼ │
- │ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ │
- │ │ GPU 0 │ │ GPU 1 │ │ GPU 2 │ │ GPU 3 │ │
- │ └───────┘ └───────┘ └───────┘ └───────┘ │
- └────────────────────────────────────────────────────────────────────────┘
- PCIe Generational Bandwidth & Physical Characteristics
PCIe utilizes serial point-to-point differential signaling lanes. Bandwidth scales with the number of physical lanes (typically × 16 \times 16 × 16 for discrete enterprise GPUs) and the signaling rate per generation:
PCIe Gen 4: Operates at a raw signaling rate of 16.0 GT/s (Gigatransfers per second) per lane using 128b/130b line encoding (yielding ~98.5% payload efficiency). A full × 16 \times 16 × 16 slot provides approximately 31.5 GB/s in each direction , or ~63–64 GB/s total bidirectional throughput .
PCIe Gen 5: Doubles the raw signaling rate to 32.0 GT/s per lane using identical 128b/130b encoding. A standard × 16 \times 16 × 16 slot achieves 63.0 GB/s per direction , resulting in ~126–128 GB/s total bidirectional throughput .
PCIe Gen 6 (Emerging): Transitions from Non-Return-to-Zero (NRZ) 2-level signaling to PAM4 (4-level Pulse Amplitude Modulation) and abandons 128b/130b entirely in favour of FLIT-based 1b/1b encoding with mandatory Forward Error Correction (FEC) . A × 16 \times 16 × 16 link reaches 128 GB/s per direction / ~256 GB/s bidirectional .
PCIe Comparison Matrix
Interconnect Standard Raw Transfer Rate Line Encoding Unidirectional Bandwidth ( × 16 \times 16 × 16 ) Bidirectional Bandwidth ( × 16 \times 16 × 16 ) Typical Transfer Latency
- PCIe Gen 3 8.0 GT/s 128b/130b 15.75 GB/s 31.5 GB/s ~2.0–2.5 μ \mu μ s
- PCIe Gen 4 16.0 GT/s 128b/130b 31.51 GB/s 63.02 GB/s ~1.5–2.0 μ \mu μ s
- PCIe Gen 5 32.0 GT/s 128b/130b 63.02 GB/s 126.04 GB/s ~1.0–1.5 μ \mu μ s
- PCIe Gen 6 64.0 GT/s (PAM4) FLIT mode, 1b/1b + FEC 128.0 GB/s 256.0 GB/s ~1.0–1.2 μ \mu μ s
- The PCIe Bottleneck in Deep Learning
While 128 GB/s bidirectional throughput is sufficient for standard storage I/O and conventional database workloads, it represents a severe bottleneck in distributed deep learning for several structural reasons:
Bandwidth Asymmetry: An NVIDIA H100 GPU features an internal High-Bandwidth Memory (HBM3) subsystem capable of delivering 3.35 TB/s (3,350 GB/s) of memory bandwidth. A PCIe Gen 5 host interface (63 GB/s unidirectional) exposes less than 1.9% of the GPU's memory speed . Any workload requiring continuous cross-GPU synchronization over PCIe starves the GPU's arithmetic Tensor Cores.
NUMA Crossing Penalties: In dual-socket CPU servers, GPUs connected to CPU 0 must traverse the inter-socket CPU coherent link (Intel Ultra Path Interconnect - UPI, or AMD Infinity Fabric) to exchange data with GPUs connected to CPU 1. This inter-socket bridge introduces substantial latency spikes (~3–5 μ \mu μ s) and severe bandwidth contention.
PCIe Tree Switch Congestion: When multiple GPUs communicate across intermediate PCIe switches (such as PLX/Broadcom switches), packet contention on the root port upstream links degrades all-to-all communication primitives (such as AllReduce) into serialized bottlenecks.
2. NVLink Generational Evolution: Pascal to Blackwell
To overcome PCIe bandwidth and protocol limitations, NVIDIA developed NVLink —a dedicated, high-speed, direct GPU-to-GPU interconnect protocol. Unlike PCIe, which treats GPUs as standard peripheral I/O devices, NVLink provides native memory load/store and atomic semantics , allowing a GPU to directly read, write, and perform atomic operations on remote GPU memory with sub-microsecond latency. ┌────────────────────────────────────────────────────────────────────────┐
- │ NVLINK GENERATIONAL BANDWIDTH ROADMAP │
- │ │
- │ Blackwell (NVLink 5) : [====================================] 1800 GB/s│
- │ Hopper (NVLink 4) : [==================] 900 GB/s │
- │ Ampere (NVLink 3) : [============] 600 GB/s │
- │ Volta (NVLink 2) : [======] 300 GB/s │
- │ Pascal (NVLink 1) : [===] 160 GB/s │
- │ PCIe Gen 5 (x16) : [=] 128 GB/s │
- └────────────────────────────────────────────────────────────────────────┘
- Architectural Features of the NVLink Protocol
Physical Layer (PHY): Utilizes high-frequency differential pairs with advanced signaling techniques (NRZ in earlier generations, transitioning to 50 Gbaud / 100 Gbps PAM4 in NVLink 4 and NVLink 5).
Data Link Layer: Provides hardware-level cyclic redundancy checking (CRC), packet framing, credit-based flow control, and link-level packet replay with zero operating system or CPU overhead.
Transaction Layer: Native support for 64-bit Virtual Memory addressing across connected GPUs (Unified Virtual Addressing / Shared Memory Architecture). Supports direct remote reads, remote writes, atomic additions ( atomicAdd ), compare-and-swap, and cache coherence protocols.
Detailed Generational Specifications
Generation Architecture & GPU Year Physical Links / Lanes Signaling Type Total Bidirectional Bandwidth Bandwidth vs. PCIe Gen 4 x16
- NVLink 1 Pascal (Tesla P100) 2016 4 links (8 sub-links) 20 Gbps NRZ 160 GB/s 2.5 × \times ×
- NVLink 2 Volta (Tesla V100) 2017 6 links (12 sub-links) 25 Gbps NRZ 300 GB/s 4.7 × \times ×
- NVLink 3 Ampere (A100 SXM4) 2020 12 links (4 differential pairs/link) 50 Gbps NRZ 600 GB/s 9.4 × \times ×
- NVLink 4 Hopper (H100 SXM5) 2022 18 links (2 lanes/link) 100 Gbps PAM4 (50 Gbaud) 900 GB/s 14.1 × \times ×
NVLink 5 Blackwell (B200 / GB200) 2024 18 links (high-speed differential) 200 Gbps PAM4 (100 Gbaud) 1,800 GB/s (1.8 TB/s) 28.1 × \times ×
Architectural Evolution Highlights
Pascal (NVLink 1 - 160 GB/s): Replaced SLI bridges and PCIe staging for 4-GPU and 8-GPU systems, establishing point-to-point mesh topologies (DGX-1 used hybrid cube-mesh wiring).
Volta (NVLink 2 - 300 GB/s): Increased link count to 6 per GPU. Enabled direct hardware atomics and cache coherency, powering the 16-GPU DGX-2 with the first-generation NVSwitch.
Ampere (NVLink 3 - 600 GB/s): Doubled links to 12 per GPU. Standardized SXM4 form factors with 6 on-board NVSwitches in HGX A100 baseboards, delivering 4.8 TB/s aggregate bidirectional bandwidth across 8 GPUs.
Hopper (NVLink 4 - 900 GB/s): Transitioned to 100 Gbps PAM4 signaling across 18 links. Introduced direct hardware acceleration for Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) collectives within the NVSwitch fabric and hardware-accelerated Tensor Memory Accelerator (TMA) transfers.
Blackwell (NVLink 5 - 1.8 TB/s): Achieves a massive 1.8 TB/s bidirectional bandwidth per GPU (900 GB/s in each direction), equivalent to the aggregate bandwidth of 14 separate PCIe Gen 5 × 16 \times 16 × 16 slots. Integrates second-generation SHARP in-network computing, doubling collective AllReduce throughput across massive LLM clusters.
3. NVSwitch Crossbar Architecture & Full-Mesh Topologies
While point-to-point NVLink direct links work effectively for 2 or 4 GPUs, scaling direct connections across 8 or more GPUs encounters geometric routing limits: a fully connected direct-mesh graph requires N ( N − 1 ) 2 \frac{N(N-1)}{2} 2 N ( N − 1 ) bidirectional links. For 8 GPUs, direct full-mesh connectivity would require 28 discrete multi-lane link groups, exceeding physical die perimeter boundaries and motherboard trace routing constraints. ┌────────────────────────────────────────────────────────────────────────┐
- │ HGX H100 8-GPU NVSWITCH CROSSBAR TOPOLOGY │
- │ │
- │ [GPU 0] [GPU 1] [GPU 2] [GPU 3] [GPU 4] [GPU 5] [GPU 6] [GPU 7] │
- │ │ │ │ │ │ │ │ │ │
- │ └───┬───┴───┬───┴───┬───┴───┬───┴───┬───┴───┬───┴───┬───┘ │
- │ │ │ │ │ │ │ │ │
- │ ┌────┴───────┴───────┴───────┴───────┴───────┴───────┴────┐ │
- │ │ 4× THIRD-GENERATION NVSWITCHES │ │
- │ │ (Non-Blocking All-to-All Crossbar) │ │
- │ │ 7.2 TB/s Aggregate Bidirectional Bandwidth │ │
- │ └─────────────────────────────────────────────────────────┘ │
- │ │
- │ Result: ANY GPU can write to ANY GPU at full 900 GB/s line rate! │
- └────────────────────────────────────────────────────────────────────────┘
- The NVSwitch Crossbar Solution
NVIDIA solved this scaling challenge by introducing NVSwitch —a high-radix, non-blocking, multi-port crossbar packet switch on a dedicated silicon chip:
Non-Blocking Switching: Any GPU connected to the NVSwitch fabric can communicate with any other GPU in the system at full line rate simultaneously without packet collision, head-of-line blocking, or CPU mediation.
Address Translation & Routing: The NVSwitch ASIC inspects incoming NVLink transaction packets, performs physical-to-logical address mapping across the unified global GPU memory space, and routes packets to destination GPU ports with sub-100ns switching latency.
Integrated Hardware Reduction Engine: NVSwitches embed on-chip arithmetic logic units that execute in-network reduction operations (SHARP) directly inside the switch fabric, offloading gradient aggregation during distributed training.
Generational NVSwitch Comparison
Switch Model Silicon Architecture Port Count & Signaling Aggregate Full-Duplex Bandwidth HGX Deployment Count System Implementation
NVSwitch 1 Volta 18 NVLink 2 ports (25 Gbps NRZ) 900 GB/s (7.2 Tbps) 12 switches per chassis DGX-2 (16x V100)
NVSwitch 2 Ampere 36 NVLink 3 ports (50 Gbps NRZ) 2.4 TB/s (19.2 Tbps) 6 switches per baseboard DGX A100 (8x A100)
NVSwitch 3 Hopper 64 NVLink 4 ports (100 Gbps PAM4) 3.2 TB/s (25.6 Tbps) 4 switches per baseboard DGX H100 (8x H100)
NVSwitch 4 Blackwell 72 NVLink 5 ports (200 Gbps PAM4) 7.2 TB/s (57.6 Tbps) 2 switches per tray NVL72 Rack (72x B200)
The HGX H100 Crossbar Layout
In the NVIDIA HGX H100 8-GPU baseboard:
Each H100 SXM5 GPU exposes 18 NVLink 4 links , providing 900 GB/s bidirectional throughput per GPU.
The baseboard integrates 4 third-generation NVSwitch chips .
Each GPU divides its 18 links across the 4 NVSwitches (4–5 links per switch).
The aggregate crossbar switching bandwidth across the 4 NVSwitch chips is 7.2 TB/s bidirectional (3.6 TB/s per direction) .
This ensures that every GPU can broadcast or exchange data with any other GPU at full 900 GB/s wire speed , completely eliminating intra-node communication contention.
4. Scaling Beyond the Chassis: NVLink Network & Blackwell NVL72
Historically, NVLink fabric was restricted to a single server chassis (up to 8 or 16 GPUs). Communicating between nodes required traversing PCIe to an InfiniBand Host Channel Adapter (HCA). With the introduction of the NVLink Network protocol and the NVIDIA GB200 NVL72 system, NVIDIA scaled the NVLink crossbar to encompass entire data center racks. ┌────────────────────────────────────────────────────────────────────────┐
- │ BLACKWELL NVL72 RACK-SCALE ARCHITECTURE │
- │ │
- │ ┌──────────────────────────────────────────────────────────────────┐ │
- │ │ 18 Compute Trays (36 GB200 Superchips = 72 Blackwell B200 GPUs) │ │
- │ │ Total GPU HBM3e Memory: 13.5 TB (at 576 TB/s aggregate bandwidth)│ │
- │ └──────────────────────────────────┬───────────────────────────────┘ │
- │ │ Over-Rack NVLink Spine │
- │ │ (5,000+ Passive Copper Cables) │
- │ ┌──────────────────────────────────┴───────────────────────────────┐ │
- │ │ 9 NVLink Switch Trays x 2 NVLink Switch Chips = 18 NVSwitches │ │
- │ │ 130 TB/s Aggregate Bidirectional NVLink Switching Fabric │ │
- │ └──────────────────────────────────────────────────────────────────┘ │
- │ │
- │ Single Unified Virtual GPU: 72 GPUs with full-mesh all-to-all NVLink! │
- └────────────────────────────────────────────────────────────────────────┘
- Mechanics of the NVL72 Rack Architecture
Unified NVLink Domain: The NVL72 connects 72 NVIDIA Blackwell B200 GPUs (housed across 18 compute trays, with each tray containing 2 GB200 Grace Blackwell Superchips) into a single, massive, non-blocking NVLink switching domain.
NVLink Spine & Direct Copper Interconnect: Rather than using optical transceivers that consume substantial power and add latency, the NVL72 links compute trays to switch trays via a cartridge-based blind-mate passive copper backplane comprising over 2 miles (5,000+ discrete conductors) of high-speed copper cabling. This saves approximately 20 kW of power per rack compared to optical interconnects.
Switch Trays: 9 dedicated NVLink Switch trays each house 2 fifth-generation NVLink Switch chips — 18 NVSwitch ASICs in total — delivering 130 TB/s aggregate bidirectional fabric bandwidth .
Operational Impact on Model Parallelism: In traditional clusters, intra-node Tensor Parallelism is limited to 8 GPUs over NVLink, requiring slower Pipeline Parallelism across InfiniBand for larger dimensions. In an NVL72 rack, a 70-billion or 500-billion parameter model can run Tensor Parallelism across all 72 GPUs simultaneously , delivering up to a 30 × \times × inference speedup and 4 × \times × training throughput improvement over traditional multi-node clusters.