The CUDA Handbook ▸ Chapter 3. Software Architecture

3.4 Modules and Functions

Modules are collections of code and related data that are loaded together, analogous to DLLs on Windows or DSOs on Linux. Like CUDA contexts, modules were for most of CUDA’s history available only in the driver API; CUDA 12.0 exposed them to the CUDA runtime as “libraries,” via cudaLibraryLoadData(), cudaLibraryGetKernel(), and related functions⁶.

CUDA does not have a linker, so CUDA does not have an intermediate structure analogous to object files that can be synthesized into a CUDA module; instead, nvcc directly emits files that can be loaded as CUDA modules:

• .cubin files that target specific SM versions, or

• .ptx files that can be compiled onto the hardware by the driver.

This data needn’t be sent to end users in the form of these files: CUDA includes APIs to load modules as NULL-terminated strings that can be embedded in executable resources or elsewhere⁷.

Once a CUDA module is loaded, the application can query for the resources contained in it:

• globals, and

• functions (kernels).

As an important note: all of these resources are created when the module is loaded, so the query functions cannot fail due to a lack of resources.

Like contexts, the CUDA runtime hides the existence and management of modules; all modules are loaded at the same time CUDART is initialized. For applications with large amounts of GPU code, the ability to explicitly manage residency by loading and unloading modules is one of the principal reasons to use the driver API instead of the CUDA runtime.

Modules are built by invoking nvcc, which can emit different types of modules depending on the command line parameters, as summarized in Table 3-3.

Table 3-3. nvcc Module Types

Since cubins have been compiled to a specific GPU architecture, they do not have to be compiled “just in time,” and are faster to load. But they are neither backward compatible (e.g., cubins compiled onto SM 2.x cannot run on SM 1.x architectures) nor forward compatible (e.g., cubins compiled onto SM 2.x architectures will not run on SM 3.x architectures). As a result, only applications with a priori knowledge of their target GPU architectures (and hence, cubin versions) can use cubins without also embedding PTX versions of the same modules to use as backup.

PTX is the intermediate language used as a source for the driver’s just-in-time compilation. Because this compilation can take a significant amount of time, the driver saves compiled modules and reuses them for a given PTX module, provided the hardware and driver have not changed. If the driver or hardware changes, all PTX modules must be recompiled.

With fatbins, the CUDA runtime automates the process of using a suitable cubin, if available, and compiling PTX otherwise. The different versions are embedded as strings in the host C++ code emitted by nvcc. Applications using the driver API have the advantage of finer-grained control over modules – for example, they can be embedded as resources in the executable, encrypted, or generated at runtime – but the process of using cubins if available and compiling PTX otherwise must be implemented explicitly.

Modules also need not originate as files produced ahead of time by nvcc. NVRTC, the runtime compilation library introduced in CUDA 7.0, compiles CUDA C++ source into PTX while the application runs: the program hands NVRTC a string of device code, receives PTX back, and loads it like any other module – with cuModuleLoadData() (or, since CUDA 12.0, cudaLibraryLoadData()), followed by cuModuleGetFunction() and cuModuleGetGlobal() to reach its kernels and globals. Compiling source at runtime lets a program specialize device code in ways that aren’t possible at compile time – a data type, an operator, or values that otherwise would have to be fixed as compile-time template parameters, resulting in a Cartesian explosion of generated object code that otherwise would have to be compiled, stored, and dispatched at runtime. APIs that explicitly divide the specification of an operation from its execution have a rich history, going at least as far back as fftw (the “fastest FFT in the West”), which used problem size specifications to develop “plans” that generated bespoke code to compute FFTs as fast as possible. Section 12.5 uses NVRTC to illustrate how to build a reduction whose operator is supplied at runtime.

Table 3-4. Module query functions.