This chapter gives a detailed description of how kernels are executed on the GPU: how they are launched, their execution characteristics, how they are organized into grids of blocks of threads, and resource management considerations. The section on Dynamic Parallelism, the feature that enables CUDA kernels to launch work for the GPU, is followed by sections on newer CUDA features such as Independent Thread Scheduling, cooperative kernel launches, green contexts, and thread block clusters.
CUDA kernels execute on the GPU and, since the very first version of CUDA, always have executed concurrently with the CPU. In other words, kernel launches are asynchronous: control is returned to the CPU before the GPU has completed the requested operation. When CUDA was first introduced, there was no need for developers to concern themselves with the asynchrony (or lack thereof) of kernel launches; data had to be copied to and from the GPU explicitly, and the memcpy commands would be enqueued after the commands needed to launch kernels. It was not possible to write CUDA code that exposed the asynchrony of kernel launches; the main side effect was to hide driver overhead when performing multiple kernel launches consecutively.
With the introduction of mapped pinned memory – host memory that can be directly accessed by the GPU – the asynchrony of kernel launches becomes more important, especially for kernels that write their output to host memory. Explicit synchronization sometimes is not needed for kernels that read via mapped pinned memory, since any pending writes by the CPU will be posted before the kernel launches; but for kernels that are returning results to CPU by writing to mapped pinned memory, synchronizing to avoid write-after-read hazards is essential.
Once a kernel is launched, it runs as a grid of
blocks of threads. Not all blocks run concurrently,
necessarily; each block is assigned to a Streaming Multiprocessor (SM),
and each SM can maintain the context for multiple blocks. To cover both
memory and instruction latencies, the SM generally needs more warps than
a single block can contain. The maximum number of blocks per SM is at
least 8, and can be queried by calling cudaGetDeviceProperties() and
examining maxBlocksPerMultiProcessor.
The programming model makes no guarantees whatsoever as to the order of execution, or whether certain blocks or threads can run concurrently. Developers can never assume that all the threads in a kernel launch are executing concurrently – it is easy to launch more threads than the machine can hold, and some will not start executing until others have finished. Given the lack of ordering guarantees, even initialization of global memory at the beginning of a kernel launch is a difficult proposition.
Dynamic parallelism is a mature, little-used feature that enables kernels to launch other kernels and coordinate the work among them.
Independent Thread Scheduling, a new feature added with the Volta V100, enables threads within the same warp to execute concurrently: a significant departure from the “lockstep” execution model that CUDA hardware always had implemented before. Our testing, documented in Section 7.5, illustrates that CUDA developers are well-served by continuing to adhere to the historical guidance of avoiding divergence and planning for threads to execute within warps, 32 at a time.
Cooperative Kernel Launches are a new feature that enables the kernel launch to guarantee that all warps will be resident and executing concurrently on the GPU. If some aspect of the kernel configuration precludes that, a cooperative kernel launch will return an error.
Green Contexts enable the GPU to be hard-partitioned into sections that operate with total autonomy, to enable concurrent execution of differently-configured kernels or even applications.
Thread Block Clusters, added with the Hopper architecture, insert an optional level between the grid and the block: a small group of co-scheduled thread blocks that can read and write one another’s shared memory as a single distributed address space.