Device memory (or linear device memory) resides in the CUDA address space, and may be accessed by CUDA kernels via normal C/C++ pointer and array dereferencing operations. Most GPUs have a dedicated pool of device memory that is directly attached to the GPU, and accessed by an integrated memory controller.
Unlike CPU applications, which can allocate more virtual memory than
there is physical memory in the system, device memory allocations are
backed by actual physical memory, and CUDA’s memory allocation
facilities fail when the physical memory is exhausted. Only managed
memory (allocated with cudaMallocManaged(), described in Section 5.7) is
exempt: on Pascal and later GPUs, which support demand paging (see
Section 2.4.2), managed allocations may exceed physical device memory,
with pages migrating on demand.
The details of how to allocate, free, and access device memory are given in Section 5.2.
CUDA runtime applications may query the total amount of device memory
available on a given device by calling cudaGetDeviceProperties() and
examining cudaDeviceProp::totalGlobalMem.
cudaMalloc() and cudaFree() allocate and free device memory,
respectively.
cudaMallocPitch() allocates pitched memory; cudaFree() may be used to
free it.
cudaMalloc3D() performs a 3D allocation of pitched memory.
Driver API applications may query the total amount of device memory
available on a given device by calling cuDeviceTotalMem().
Alternatively, the cuMemGetInfo() function may be used to query the
amount of free device memory as well as the total. cuMemGetInfo() can
only be called when a CUDA context is current to the CPU thread.
cuMemAlloc() and cuMemFree() allocate and free device memory,
respectively.
cuMemAllocPitch() allocates pitched memory; cuMemFree() may be used
to free it.