Every technique so far in this chapter moves data between GPUs, or between the CPU and a GPU, through pointers the CUDA driver manages. GPUDirect is the umbrella name for a family of hardware and software features that give data a direct path into and out of GPU memory – from a peer GPU, a network adapter, or a storage device – with no staging copy through host memory. The members pursue that one goal over different links.
GPUDirect Peer-to-Peer is the member this chapter
has already used. On a system whose topology allows it, one GPU reads
and writes another’s memory directly over PCIe or NVLink, and
cudaMemcpyPeer() (Section 9.2) moves data from GPU to GPU with no host
bounce buffer.
GPUDirect RDMA extends the idea to the network. A
third-party device on the PCIe bus – most often an InfiniBand or RoCE
network adapter – can DMA directly to and from GPU memory, so a GPU on
one node sends a buffer straight into a GPU on another with both CPUs
out of the data path. The hardware is described in Section 2.7.3, and a
device reports support through the gpuDirectRDMASupported
property.
GPUDirect Storage applies the idea to files, through
the cuFile() API. cuFileRead() and cuFileWrite(), operating on registered
file handles and GPU buffers, let a storage device – a local NVMe drive
or an NVMe-over-Fabrics target – DMA data directly between the file and
GPU memory. The conventional path reads a file into a host buffer and
then copies it across PCIe to the GPU: two transfers, plus the host
memory bandwidth and CPU time the staging copy consumes. GPUDirect
Storage removes that copy, which matters most in the data-loading phase
of training, where a working set larger than device memory is streamed
continuously from disk.
GPUDirect Async moves control onto the GPU rather than data. The stream memory operations of Section 6.10 let the GPU ring a network adapter’s doorbell and wait on its completion flag itself, so a communication step need not return to the CPU to be issued.
Across all four, the GPU stops being an accelerator the CPU feeds and becomes a peer that exchanges data with other GPUs, the network, and storage without routing it through the host. The collective-communication libraries of the next section build on that substrate.