The Streaming Multiprocessors (SMs) are the part of the GPU that runs our CUDA kernels. Each SM contains:
thousands of registers that can be partitioned among threads of execution;
several caches:
shared memory for fast data interchange between threads,
constant cache for fast broadcast of reads from constant memory,
texture cache to aggregate bandwidth from texture memory,
L1 cache to reduce latency to local or global memory.
warp schedulers that can quickly switch contexts between threads and issue instructions to warps that are ready to execute;
Execution cores for integer and floating-point operations:
Integer and single-precision floating point operations,
Double-precision floating point,
Special Function Units (SFUs) for single-precision floating-point transcendental functions.
The reason there are many registers, and the reason the hardware can context switch between threads efficiently, is to maximize throughput of the hardware. The GPU is designed to have enough state to cover both execution latency and the memory latency of hundreds of clock cycles that it may take for data from device memory to arrive after a read instruction is executed.
The SMs are general-purpose processors, but they are designed very differently than the execution cores in CPUs: they target much lower clock rates; they support instruction-level parallelism, but not branch prediction or speculative execution; and they have less cache, if they have any cache at all. For suitable workloads, the sheer computing horsepower in a GPU more than makes up for these disadvantages.
The design of the Streaming Multiprocessors has been evolving
continuously since the introduction of the first CUDA-capable hardware
in 2006, through major revisions codenamed Tesla (SM 1.x), Fermi (2.x),
Kepler (3.x), Maxwell (5.x), Pascal (6.x), Volta (7.0), Turing (7.5),
Ampere (8.0-8.7), Ada Lovelace (8.9), Hopper (9.0), and Blackwell –
which, in a break with tradition, spans three major versions: 10.x for
the data center parts, 11.x for the embedded and automotive line, and
12.x for the consumer and workstation family. (Beginning with CUDA 12.9,
the compiler also accepts family-specific targets such as compute_100f
and compute_120f that cover every minor version within a family.)
Developers can query the compute capability by calling
cudaGetDeviceProperties() and examining cudaDeviceProp.major and
cudaDeviceProp.minor, or via cuDeviceGetAttribute() in the driver API.
Table 8-1 summarizes the capabilities added in each generation of the
Streaming Multiprocessor hardware.
| Compute Level | Architecture | Introduced… |
|---|---|---|
| 1.1 | Tesla | Global memory atomics; mapped pinned memory; debuggable (e.g. breakpoint instruction) |
| 1.2 | Tesla | Relaxed coalescing constraints; warp voting (any() and all()
intrinsics); atomic operations on shared memory |
| 1.3 | Tesla | Double precision support |
| 2.0 | Fermi | 64-bit addressing; L1 and L2 cache; concurrent kernel execution;
configurable 16K or 48K shared memory; bit manipulation instructions
(__clz(), __popc(), __ffs(), __brev()); directed rounding for single
precision floating point values; fused multiply-add; 64-bit clock
counter; surface load/store; 64-bit global atomic add, exchange, and
compare-and-swap; global atomic add for single-precision floating point
values; warp voting (ballot() intrinsic); assertions and formatted
output (printf) |
| 2.1 | Fermi | Stack-based API enables function calls and indirect calls in kernels |
| 3.0 | Kepler | Increased maximum grid size; warp shuffle; permute; 32K/32K shared memory configuration; configurable shared memory (32- or 64-bit mode) |
| 3.5 | Kepler | Bindless textures (“texture objects”); faster global atomics; 64-bit atomic min, max, AND, OR, and XOR; 64-bit funnel shift; read global memory via texture; dynamic parallelism |
| 5.0/5.2 | Maxwell | Unified L1/texture cache with dedicated shared memory; hardware shared-memory atomics for 32-bit values |
| 5.3 | Maxwell | Native half-precision arithmetic (see Section 8.3.4) |
| 6.0/6.1/6.2 | Pascal | Hardware demand paging and managed memory oversubscription (Section
2.4.2); full-rate half-precision arithmetic (6.0); double-precision
atomicAdd(); NVLink; __dp4a()/__dp2a() 8-bit integer dot products
(6.1) |
| 7.0 | Volta | Independent thread scheduling (Section 7.5); first-generation Tensor Cores (half-precision matrix multiply); _sync variants of the warp intrinsics |
| 7.5 | Turing | Integer (8-bit and 4-bit) Tensor Core modes; uniform datapath and register file |
| 8.0/8.6/8.7 | Ampere | TF32, bfloat16, and double-precision Tensor Cores; asynchronous copy from global to shared memory, bypassing registers; asynchronous barriers; L2 cache residency control; 2:4 structured sparsity |
| 8.9 | Ada Lovelace | FP8 Tensor Cores |
| 9.0 | Hopper | Thread block clusters with distributed shared memory; Tensor Memory Accelerator (TMA); asynchronous transaction barriers; DPX dynamic-programming instructions; FP8 transformer engine |
| 10.0/10.3 | Blackwell (data center) | Fifth-generation Tensor Cores with FP4 and FP6 (Section 8.3.5) and dedicated tensor memory; CTA-pair matrix operations; decompression engine; NVLink 5 |
| 11.0 | Blackwell (embedded) | Jetson Thor automotive/robotics line (numbered 10.1 before CUDA 13.0) |
| 12.0/12.1 | Blackwell (consumer) | GeForce RTX 50 series, RTX PRO workstation parts, and the GB10 of DGX Spark (12.1); FP4 Tensor Cores without the data center parts’ clusters or tensor memory capacity |
Table 8-1. SM Capabilities
In the chapter on Hardware Architecture, Figures 2-29 through 2-32 show block diagrams of different Streaming Multiprocessors. CUDA cores can execute integer and single-precision floating point instructions; one double-precision unit implements double-precision support, if available; and Special Function Units implement reciprocal, reciprocal square root, sine/cosine, and logarithm/exponential functions. Warp schedulers dispatch instructions to these execution units as the resources needed to execute the instruction become available.
This chapter will focus on the instruction set capabilities of the
SM. As such, it sometimes refers to the “SASS” instructions, the native
instructions into which ptxas or the CUDA driver translate intermediate
PTX code. Developers are not able to author SASS code directly; instead,
NVIDIA has made these instructions visible to developers through the
cuobjdump utility, so developers can direct optimizations of their
source code by examining the compiled microcode.