To maximize performance, CUDA uses different types of memory depending on the expected usage.
Host memory refers to the memory attached to the CPU(s) in the system. CUDA provides APIs that enable faster access to host memory by page-locking and mapping it for the GPU(s).
Device memory is attached to the GPU and accessed by a dedicated memory controller. Device memory can be allocated and accessed in a variety of ways:
Global memory may be allocated statically or dynamically, and accessed via pointers in CUDA kernels;
Constant memory is read-only memory optimized for broadcast to multiple threads;
Local memory contains the stack: local variables that cannot be held in registers, parameters and return address for subroutines;
Texture memory (in the form of CUDA arrays) is accessed via texture and surface load/store instructions.
Constant memory and texture memory are accessed using different machine instructions than global memory, enabling the hardware to service the memory traffic with caches that are distinct from the caches that service global memory traffic. Caches for read-only memory traffic, such as reads from constant or texture memory, are more efficient to build in hardware because the hardware need only invalidate cache lines – there is no need to write cache contents back to memory.
Shared memory is an important type of memory in CUDA that is not backed by device memory. Instead, it is an abstraction for an on-chip “scratchpad” memory that can be used for fast data interchange between threads within a block. Physically, shared memory is built into each Streaming Multiprocessor.
Managed memory uses virtual memory manager (VMM) support in the CPU and GPU to migrate pages between host and device memory, as needed. Managed memory is a useful porting tool because it gives the illusion of unified memory, shared between the CPU and GPU. Applications that use managed memory don’t have to concern themselves with explicitly copying data between host and device memory. Managed memory is discussed in Section 5.7.