In 2017, in response to NVIDIA’s first introduction of Tensor Cores into the CUDA platform, I wrote an article called Don’t Move The Data that highlighted that data availability has become the limiting reagent of all compute. Tensor Cores promised big performance improvements, but that came with a caveat: it made the GPUs even hungrier for data, exacerbating the need to improve both bandwidths and latencies of available interconnects to keep the GPUs from starving. To that end, NVIDIA already had invested in technologies like NVLink, their proprietary interconnect, and GPUDirect for copy elision (i.e. to avoid round trips through CPU memory); but in 2019, NVIDIA announced the acquisition of Mellanox, the preeminent vendor of Infiniband networking adapters, and they have continued developing that technology so performance of multi-GPU setups can continue to scale healthily as the GPUs’ performance continues to increase.
The three technologies discussed in this chapter include the venerable and pervasive PCI Express; NVLink, a high-bandwidth interconnect that started out as an alternative to PCI Express for CPU-GPU connection, and since has evolved into a fabric that can connect multiple servers; and RDMA-capable network adapters, which carry GPU traffic between servers and clusters.
PCI Express (PCIe) was the industry standard for PC platforms by the time CUDA first shipped in 2007. As a bus, PCIe serves as the transport for many different peripherals across the system: not just GPUs, but disk and network adapters as well. Because GPUs demand more bandwidth than any other peripheral, they are built for the widest, 16-lane slots, and each generation of the standard has roughly doubled the per-lane rate—from about 500 MB/s per lane on PCIe 2.0 (c. 2007) to 8 GB/s on 6.0 (c. 2022). In practice, due to packet overhead, the bandwidths delivered by PCIe are about 25% lower than the theoretical line rates. A 16-lane PCIe 4.0 link, typical of Ampere-generation systems, offers roughly 32 GB/s per direction in theory and, after packet overhead, about 25 GB/s in practice; Chapters 6 and 11 have microbenchmarks that measure these numbers directly.
Despite the increases in bandwidth across the years, PCIe has remained significantly slower than its contemporaneous GPUs’ local memory bandwidth. For workloads that stream data through the GPU rather than keeping it resident—the transfer-bound cases of Chapters 6 and 11—PCIe usually is the bottleneck, which is why the techniques for overlapping compute (by both the CPU and the GPU) with PCIe transfers are so important: pinned memory, and asynchronous copies moderated by CUDA streams and events.
For communication among GPUs in the same server, NVIDIA supplements or replaces PCIe with NVLink, a proprietary point-to-point interconnect introduced with Pascal (c. 2016). An NVLink connection is built from a number of links, each carrying tens of gigabytes per second per direction, and a GPU aggregates several of them; total NVLink bandwidth has grown from about 160 GB/s to 900 GB/s on Hopper (c. 2022).
In addition to the bandwidth advantages over PCIe, NVLink was designed to service cache coherency traffic: Starting with supercomputers that NVIDIA co-designed with IBM in the Pascal and Volta generations, the latest superchips use NVLink’s cache coherency technology to make the disjoint CPU and GPU memories appear unified. The first NVIDIA-proprietary deployment of this technology was the Grace Hopper superchip, and NVIDIA has continued to roll out improved versions of those CPUs as well as GPUs.
NVIDIA also has pursued enabling GPUs to communicate directly over NVLink. To scale this communication beyond 2-4 GPUs, they introduced NVSwitch, a crossbar built from the same NVLink signaling, to connect every GPU in a node to every other at full NVLink bandwidth, turning a chassis of GPUs into a tightly coupled fabric in which any GPU can read or write any other’s memory at hundreds of gigabytes per second. On such systems—NVIDIA’s DGX servers and the HGX baseboards that OEMs build on—multi-GPU algorithms that PCIe would throttle, such as the all-reduce of distributed training or the domain decomposition of an N-body or PDE solver, instead scale nearly linearly.
Scaling past a single server—to the dozens or thousands of GPUs in a cluster—means moving GPU data over the network. To minimize superfluous data copies, GPUs and network adapters implement remote direct memory access (RDMA): one node can read or write memory on another node directly, without interrupting either CPU, which keeps latency low and sustains high throughput. The memory in question may be attached to either CPUs or GPUs.
RDMA is provided natively by InfiniBand and, over Ethernet, by RoCE (RDMA over Converged Ethernet). NVIDIA’s ConnectX network adapters and BlueField DPUs—the product line acquired with Mellanox—implement both, at port rates that have climbed to 200 and 400 gigabits per second, or roughly 25 and 50 GB/s.
What makes an RDMA NIC a GPU interconnect is GPUDirect RDMA, which lets the adapter DMA directly to and from GPU memory across PCIe, with no staging copy through host memory: a GPU on one node can send a buffer straight into the memory of a GPU on another, leaving both CPUs out of the data path. NVIDIA’s collective-communications library, NCCL, composes these tiers automatically—NVLink and NVSwitch within a node, GPUDirect RDMA between nodes—so a collective such as an all-reduce runs over whichever links are fastest. The result is an interconnect hierarchy, from on-board NVLink down to inter-server RDMA, whose bandwidth falls at each step but which lets one programming model span from a single GPU to an entire data center.