The CUDA Handbook ▸ Chapter 5. Memory

5.1 Host Memory

In CUDA, host memory refers to memory accessible to the CPU(s) in the system. By default, this memory is pageable, meaning the operating system may move the memory or evict it out to disk. Because the physical location of pageable memory may change without notice, it cannot be accessed by peripherals like GPUs. To enable “direct memory access” (DMA) by hardware, operating systems allow host memory to be “page-locked,” and for performance reasons, CUDA includes APIs that make these operating system facilities available to application developers. So-called pinned memory that has been page-locked and mapped for direct access by CUDA GPU(s) enables:

• Faster transfer performance,

• Asynchronous memory copies (i.e. memory copies that return control to the caller before the memory copy necessarily has finished – the GPU does the copy in parallel with the CPU),

• Mapped pinned memory that can be accessed directly by CUDA kernels.

Because the virtual→physical mapping for pageable memory can change unpredictably, GPUs cannot access pageable memory at all. CUDA copies pageable memory using a pair of staging buffers of pinned memory that are allocated by the driver when a CUDA context is allocated. Chapter 6 includes illustrative pageable memcpy routines that use CUDA events to do the synchronization needed to manage this double-buffering.

5.1.1 Allocating Pinned Memory

Pinned memory is allocated and freed using special functions provided by CUDA: cudaHostAlloc()/cudaFreeHost(). These functions work with the host operating system to allocate page-locked memory and map it for DMA by the GPU(s).

CUDA keeps track of memory it has allocated, and transparently accelerates memory copies that involve host pointers allocated with cuMemHostAlloc()/cudaHostAlloc(). Additionally, some functions (notably the asynchronous memcpy functions) require pinned memory.

The bandwidthTest SDK sample enables developers to easily compare the performance of pinned memory versus normal pageable memory. The --memory=pinned option causes the test to use pinned memory instead of pageable memory. Here are the bandwidthTest numbers for a p3.2xlarge instance in Amazon EC2, running Linux (numbers in MB/s):

Table 5-1. Pinned v. Pageable Bandwidth

Because it may involve a significant amount of work for the host, such as a kernel transition, allocating pinned memory is expensive.

CUDA provides several features for pinned memory: portable pinned memory can be accessed by any GPU, and mapped pinned memory is mapped into the CUDA address space for direct access by CUDA kernels.

A further option, the cudaHostAllocWriteCombined flag, allocates the pinned buffer as write-combined memory. The CPU does not cache write-combined pages, and does not snoop them when the GPU reads across PCI Express, so both the host’s writes into the buffer and the GPU’s transfers out of it run faster. The cost is that CPU reads from such memory are very slow, since each bypasses the cache and goes all the way to DRAM – so write-combined memory fits a buffer the host only fills on its way to the device, and penalizes one the host reads back.

CUDA provides two important features that pertain to host memory: host memory registration enables existing host memory ranges to be page-locked for GPU access, and Unified Virtual Addressing (UVA) enables all pointers to be unique process-wide, including host and device pointers.

When UVA is in effect, the system generally can introspect pointers: given a pointer, CUDA can infer from the address range whether memory is host memory or device memory, and even identify the "owning" device. Additionally, when UVA is in effect, all pinned memory allocations are also portable and mapped.

The exceptions to this rule are write-combined memory and registered memory: for those, the device pointer may differ from the host pointer, and applications still must query it with cudaHostGetDevicePointer()/cuMemHostGetDevicePointer().

UVA is supported on all 64-bit platforms except Windows. On Windows since Windows Vista, only the TCC driver supports UVA. Applications can query whether UVA is in effect by calling cudaGetDeviceProperties() and examining the cudaDeviceProp::unifiedAddressing structure member.

For purposes of this book, we will assume that multi-GPU CUDA applications are built on UVA-capable platforms.

5.1.2 Registering Pinned Memory

CUDA developers don’t always get the opportunity to allocate host memory that they want the GPU(s) to access directly. For example, a large, extensible application may have an interface that passes pointers to CUDA-aware plugins; or the application may be using an API for some other peripheral (notably high-speed networking) that has its own dedicated allocation function for much the same reason CUDA does. To accommodate these usage scenarios, CUDA provides the ability to register pinned memory.

Pinned memory registration decouples allocation from the page-locking and mapping of host memory: it takes an already-allocated virtual address range, page-locks it, and maps it for the GPU. Just as with cudaHostAlloc(), the memory optionally may be mapped into the CUDA address space or made portable (accessible to all GPUs).

The cudaHostRegister()and cudaHostUnregister() functions register and unregister host memory for access by the GPU(s), respectively. While the memory is registered with CUDA, it may be accessed directly by the GPU.

The memory range to register must be page-aligned: in other words, both the base address and the size must be evenly divisible by the page size of the operating system. Applications can allocate page-aligned address ranges in two ways:

• Allocate the memory with operating system facilities that traffic in whole pages, such as VirtualAlloc() on Windows or valloc() or mmap()¹on other platforms;

• Given an arbitrary address range (say, memory allocated with malloc() or operator new[]), clamp the address range to the next-lower page boundary, and pad to the next page size.

5.1.3 Pinned Memory Usage

For applications whose performance is limited by CPU↔︎GPU transfer performance, mapped pinned memory can be a boon. Since the GPU can read or write host memory directly from kernels, it eliminates the need to perform some memory copies, reducing overhead. Some common idioms for using mapped pinned memory are as follows:

• Posting writes to host memory: Multi-GPU applications often must stage results back to system memory for further processing by the CPU; writing these results via mapped pinned memory avoids an extraneous device→host memory copy.

• Streaming: These workloads otherwise would use CUDA streams to coordinate concurrent memcpy’s to and from device memory, while kernels do their processing on device memory.

• “Copy with panache”: Some workloads benefit from performing computations as data is transferred across PCI Express. For example, the GPU may compute subarray reductions while transferring data for Scan.

Caveats

Mapped pinned memory is not a panacea. Some caveats to consider when using it include the following:

• Texturing from mapped pinned memory is possible, but very slow.

• It is important that mapped pinned memory be accessed with coalesced memory transactions (see Section 5.2.8). The performance penalty for uncoalesced memory transactions ranges from 6x to 2x. But even on SM 2.x and later GPUs, whose caches were supposed to make coalescing an obsolete consideration, the penalty is significant.

• Polling host memory with a kernel (e.g., for CPU/GPU synchronization) is not recommended.

• Do not try to use atomics on mapped pinned host memory, either for the host (locked compare-exchange) or the device (atomicAdd()). On the CPU side, the facilities to enforce mutual exclusion for locked operations are not visible to peripherals on the PCI Express bus. Conversely, on the GPU side, atomic operations only work on local device memory locations because they are implemented using the GPU’s local memory controller.

5.1.4 NUMA, Thread Affinity, and Pinned Memory

On modern systems, CPU memory controllers are integrated directly into CPUs, so each CPU has its own dedicated pool of “local” physical memory that is directly attached to that CPU. Although any CPU can access any other CPU’s memory, “nonlocal” accesses – accesses by one CPU to memory attached to another CPU – incur latency penalties and bandwidth limitations. To contrast with the uniform memory access times exhibited by older systems whose CPUs did not have integrated memory controllers, these system architectures are known as NUMA for “nonuniform memory access.”

As you can imagine, performance of multithreaded applications can be heavily dependent on whether memory references are local to the CPU that is running the current CPU thread. For most applications, however, the higher cost of a nonlocal access is offset by the CPUs’ on-board caches: once nonlocal memory is fetched into a CPU, it remains in-cache until evicted or needed by a memory access to the same page by another CPU. In fact, it is common for NUMA systems to include a System BIOS option to “interleave” memory physically between CPUs. When this BIOS option is enabled, the memory is evenly divided between CPUs on a per-cache line (typically 64 bytes) basis, so for example on a 2-CPU system, about 50% of memory accesses will be nonlocal on average.

For CUDA applications, PCI Express transfer performance can be dependent on whether memory references are local. If there is more than one I/O hub (IOH) in the system, the GPU(s) attached to a given IOH have better performance, and reduce demand for QPI bandwidth, when the pinned memory is local. Because some high-end NUMA systems are hierarchical, but don’t associate the pools of memory bandwidth strictly with CPUs, NUMA APIs refer to nodes that may or may not strictly correspond with CPUs in the system.

If NUMA is enabled on the system, it is good practice to allocate host memory on the same node as a given GPU. Unfortunately, there is no official CUDA API to associate a GPU with a given CPU. Developers with a priori knowledge of the system design may know which node to associate with which GPU; then platform-specific, NUMA-aware APIs may be used to perform these memory allocations, and host memory registration (Section 5.1.2) can be used to pin those virtual allocations and map them for the GPU(s).

Listing 5-1 gives a code fragment to perform NUMA-aware allocations on Linux², and Listing 5-2 gives a code fragment to perform NUMA-aware allocations on Windows³.

boolnumNodes( int *p ){    if ( numa_available() >= 0 ) {        *p = numa_max_node() + 1;        return true;    }    return false;} void *pageAlignedNumaAlloc( size_t bytes, int node ){    void *ret;    printf( "Allocating on node %d\n", node ); fflush(stdout);    ret = numa_alloc_onnode( bytes, node );    return ret;} voidpageAlignedNumaFree( void *p, size_t bytes ){    numa_free( p, bytes );}

boolnumNodes( int *p ){    ULONG maxNode;    if ( GetNumaHighestNodeNumber( &maxNode ) ) {        *p = (int) maxNode+1;        return true;    }    return false;} void *pageAlignedNumaAlloc( size_t bytes, int node ){    void *ret;    printf( "Allocating on node %d\n", node ); fflush(stdout);    ret = VirtualAllocExNuma( GetCurrentProcess(),                               NULL,                              bytes,                              MEM_COMMIT | MEM_RESERVE,                              PAGE_READWRITE,                              node );    return ret;} voidpageAlignedNumaFree( void *p ){    VirtualFreeEx( GetCurrentProcess(), p, 0, MEM_RELEASE );}