Most of this book is about writing CUDA kernels, but a great deal of
GPU computing is done without writing a kernel at all, by calling one of
NVIDIA’s libraries. They divide by domain. Dense linear
algebra is served by cuBLAS – the BLAS, plus the more flexible
cublasLtMatmul() for mixed-precision and block-scaled matrix multiply –
and, one level down, by CUTLASS, a template library for building custom
GEMM and convolution kernels. Sparse problems have
cuSPARSE for sparse matrix operations and cuSOLVER for dense and sparse
factorizations and eigensolvers. Signal processing has
cuFFT for fast Fourier transforms; random number
generation has cuRAND; image and signal
primitives have NPP, the NVIDIA Performance Primitives.
Deep learning rests on cuDNN, which supplies the
convolution, attention, normalization, and pooling primitives that the
frameworks are built on.
Alongside these device-side libraries are two header-only template
libraries a program uses from its own kernels and host code.
Thrust is an STL-like layer of parallel algorithms –
sort, scan, reduce, transform – over device vectors, written for
productivity. CUB exposes the same class of primitives
at the block, warp, and device level as composable building blocks,
written for performance; it is what the algorithm chapters of this book
reach for when a hand-written kernel needs a well-tuned scan or
reduction (Chapters 12 and 13). The rule of thumb is the one those
chapters keep returning to: call a library first, and write a kernel
only where profiling shows that no library can be made to fit.