Many CUDA systems have more than one GPU, and the count can range from two in a workstation to tens of thousands across a data center. What a program can do with those GPUs—and how fast—depends as much on how they are interconnected as on the GPUs themselves. This section surveys the range at increasing scale; the interconnects (PCI Express, NVLink, and RDMA networking) are described in detail in Section 2.7. Throughout, each GPU is assumed to have its own dedicated memory.
The simplest multi-GPU system places several GPUs in one node, each in a PCI Express slot (Figure 2-13). The GPUs share the host’s PCIe fabric, and a CUDA program can copy between them—or map one GPU’s memory into another’s address space—using peer-to-peer access (Section 2.4). On a large node, the placement matters: as with the NUMA considerations of Section 2.1, data transfer performance will vary for GPUs attached to different I/O hubs or CPU sockets, and if the intervening topology includes a proprietary cache coherency link such as Intel’s QPI, peer-to-peer access may not span those boundaries at all.
Figure 2-13. Multiple GPUs in a node, connected by PCI Express.
The earliest multi-GPU configurations targeted consumer gaming applications: NVIDIA’s SLI (Scalable Link Interface) and dual-GPU boards such as the GeForce GTX 295 used a PCIe bridge chip to enable two GPUs to appear as one faster GPU for graphics. By changing the bridge chip configuration to consider each GPU as separate, CUDA developers could manage each GPU explicitly. Those SLI-era consumer designs have largely given way to the NVLink-connected systems described below.
PCI Express is a general-purpose bus, shared with every other peripheral and far slower than a GPU’s device memory. To enable GPUs to cooperate more effectively, NVIDIA created a proprietary interconnect called NVLink, tying all of a server’s GPUs together through one or more NVSwitch chips (Figure 2-14). In an HGX or DGX server, any of the eight GPUs can read, write, or perform atomics on any other’s memory at NVLink bandwidth—roughly an order of magnitude beyond PCIe—which is what makes model- and data-parallel training across a server practical.
Figure 2-14. An eight-GPU server fully connected by NVSwitch.
To scale up to multiple servers, NVIDIA created the NVLink Switch System introduced in the DGX GH200 (c. 2023). The GB200 NVL72 (Figure 2-15), announced the following year, fills a liquid-cooled rack with 18 compute trays—36 Grace CPUs and 72 Blackwell GPUs—and 9 NVLink switch trays, wiring all 72 GPUs into a single NVLink domain. The 72 remain 72 distinct CUDA devices, but each can reach every other’s memory directly at NVLink speed, across a pooled 13.5 TB of HBM3e at an aggregate 130 TB/s. Workloads that previously had to be partitioned across a slow network now sit within one fast interconnect.
Figure 2-15. A GB200 NVL72: 72 GPUs wired into a single rack-scale NVLink domain.
A single NVLink domain is not the end of the line. To go even larger, racks are connected over an RDMA network—InfiniBand or RoCE Ethernet—as in Figure 2-16. GPUDirect RDMA lets a network adapter move data directly between GPUs in different racks without staging through host memory, and libraries such as NCCL orchestrate the collective operations—all-reduce and its kin—that distributed training depends on. The full interconnect hierarchy, NVLink within a domain and RDMA networking between domains, is the subject of Section 2.7.
Figure 2-16. Scaling out: NVLink domains joined across the data center by an RDMA network.
How many of these concerns a CUDA program must contend with depends on how far it scales, and each level of scale brings an API to learn. A single-GPU application need only learn about GPU memory allocation, copying, asynchrony and synchronization, and how to write and launch kernels. Scaling up—adding GPUs within one tightly coupled system, from a node to a full NVLink domain—relies on peer-to-peer access, where GPUs can access other GPUs’ memory directly (Section 2.4); the interconnect determines the speed and coherency properties of that access. Scaling out—adding systems across a network—rests instead on collective communication, through libraries such as NCCL and in-network protocols such as SHARP, which perform reductions within the network fabric itself. A developer need only learn APIs as needed by the application’s scale.