The CUDA Handbook ▸ Chapter 4. Software Environment

Prefer to read without ads? Become a member — from $10/month — and support the work. Already a member? Log in to read ad-free on this device.

4.1 nvcc (CUDA compiler driver)

nvcc is the compiler driver used by CUDA developers to translate source code into functional CUDA applications. It can perform many functions, as complex as compiling, linking and executing a sample program in one command (a usage encouraged by many of the sample programs in this book) or as simple as a targeted compilation of a GPU-only .cu file.

As a compiler driver, nvcc does nothing more than set up a build environment and spawn a combination of native tools (such as the C compiler installed on the system) and CUDA-specific command-line tools (such as ptxas) to build the CUDA code. It implements many sensible default behaviors that can be overridden by command-line options; its exact behavior depends on which “compile trajectory” is requested by the main command-line option.

Table 4-1 gives the file extensions understood by nvcc, and the default behavior implemented for them. (NOTE: some intermediate file types, like the .i/.ii files that contain host code generated by CUDA’s front end, are omitted here.)

File Extension Default Behavior
.c/.cc/.cpp/.cxx Preprocess, compile, link
.cu Split host and device cost, compile them separately
.o(bj) Link
.ptx PTX-assemble into cubin

Table 4-1. Extensions for nvcc Input Files

Table 4-2 gives the compilation stage options and corresponding compile trajectory.

Option Trajectory
--cuda Compile all .cu input files to .cu.cpp.ii output.
--cubin Compile all .cu/ptx/.gpu files to .cubin files*.
--fatbin Compile all .cu/ptx/.gpu files to PTX and/or device-only binaries, as specified by --arch and/or --code, and output the result into the fat binary file specified with the –o option*.
--ptx Compile all .cu/.gpu files to device-only .ptx files*.
--gpu Compile all .cu files to device-only .gpu files*.
--preprocess (-E) Preprocess all .c/.cc/.cpp/.cxx/.cu input files.
--generate-dependencies (-M) Generate for the one .c/.cc/.cpp/.cxx/.cu input file (more than one input file is not allowed in this mode) a dependency file that can be included in a makefile.
--compile (-c) Compile each .c/.cc/.cpp/.cxx/.cu input file into an object file.
--link (-link) Compile and link all inputs (this is the default trajectory).
--lib (-lib) Compile all inputs into object files (if necessary) and add the results to the specified output library file.
--x (-x) Explicitly specify the language for the input files, rather than letting the compiler choose a default based on the file name suffix. Allowed values: 'c','c++','cu'.
--run (-run) Compiles and links all inputs into an executable, then runs it. If the input is an executable, runs it without any compiling or linking.

* These command line options discard any host code in the input file.

Table 4-2. Compilation Trajectories

Option Description

--output-file <file>

(-o)

 Specify name and location of the output file. Only a single input file is allowed when this option is present in nvcc non- linking/archiving mode.

--pre-include <include-file>

(-include)

 Specify header files that must be preincluded during preprocessing.

--library <library>

(-l)

 Specify libraries to be used in the linking stage. The libraries are searched for on the library search paths that have been specified using –L.

--define-macro <macrodef>

(-D)

 Specify macro definitions to define during preprocessing or compilation.

--undefine-macro

(-U)

 Specify macro definitions to undefine during preprocessing or compilation.

--include-path <include-path>

(-I)

 Specify include search paths.

--system-include <include-path>

-isystem

 Specify system include search paths.

--library-path

(-L)

 Specify library search paths.

--output-directory

(-odir)

 Specify the directory of the output file. This option is intended to enable the dependency generation step (--generate-dependencies) to generate a rule that defines the target object file in the proper directory.

--compiler-bindir <path>

(--ccbin)

 Specify the directory in which the compiler executable (Microsoft Visual Studio cl, or a gcc derivative) resides. By default, this executable is expected in the current executable search path. For a different compiler, or to specify these compilers with a different executable name, specify the path to the compiler including the executable name.

--cl-version <cl-version-number>

(-cl-version)

 Specify the version of Microsoft Visual Studio installation. Allowed values for this option: 2005, 2008, 2010. This option is required if --use-local-env is specified.

Table 4-3. nvcc Options (Environment)

Option

--profile

(-pg)

Description

Instrument generated code/executable for use by gprof (Linux only).

--debug

(-g)

 Generate debug information for host code.

--device-debug<level>

(-G)

 Generate debug information for device code, plus also specify the optimization level (0-3) for the device code in order to control its ‘debuggability.’

--optimize <level>

-O

 Specify optimization level for host code.

--shared

-shared

 Generate a shared library during linking.
--machine [32:64]  | Specify 32 vs. 64 bit architecture. Allowed values for this option: 32, 64. | | -m | |

Table 4-4. Options for specifying behavior of compiler/linker

Option Description

--compiler-options <options>

(-Xcompiler)

 Specify options directly to the compiler/preprocessor.

--linker-options <options>

-Xlinker

 Specify options directly to the linker.

--archive-options <options>

(-Xarchive)

 Specify options directly to library manager.

--cudafe-options <options>

-Xcudafe

 Specify options directly to cudafe.

--ptx-options <options>

-Xptxas

 Specify options directly to the PTX optimizing assembler.

Table 4-5. nvcc options for passthrough

Option Description

--dont-use-profile

(-noprof)

 Do not use nvcc.profiles file for compilation.

--dryrun

(-dryrun)

 Suppresses execution of the compilation commands.

--verbose

(-v)

 List the commands generated by nvcc.

--keep

(-keep)

 Keep all the intermediate files generated during internal compilation steps.

--keep-dir

(-keep-dir)

 Specifies the directory where files specified by --keep should be written.

--save-temps

(-save-temps)

 (same as --keep)

--clean-targets

(-clean)

 Causes nvcc to delete all of the non-temporary files that otherwise would be created by nvcc.

--run-args <arguments>

(-run-args)

 If --run is specified, this option specifies command line arguments to pass to the executable.

--input-drive-prefix <prefix>

(-idp)

 Windows specific: specifies prefix for absolute paths of input files. For Cygwin users, specify ‘-idp /cygwin/’; for Mingw, specify ‘-idp /’

--dependency-drive-prefix <prefix>

(-ddp)

 Windows specific: specifies prefix for absolute paths when generating dependency files (--generate-dependencies). For Cygwin users, specify ‘-idp /cygwin/’; for Mingw, specify ‘-idp /’

--drive-prefix <prefix>

(-dp)

 Specifies prefix to use for both input files and dependency files.
--no-align-double Specifies that –malign-double should not be passed as a compiler argument on 32-bit platforms. NOTE: For certain 64-bit types, this option makes the ABI incompatible with CUDA’s kernel ABI.

Table 4-6. Miscellaneous nvcc options.

Option Description

--gpu-architecture <gpu architecture name>

-arch

 Specify the virtual NVIDIA GPU architectures to compile for. This option specifies which version of PTX to target. Valid options include: compute_10, compute_11, compute_12, compute_13, compute_20, compute_30, compute_35, sm_10, sm_11, sm_12, sm_13, sm_20, sm_21, sm_30, sm_35.

--gpu-code <gpu architecture name>

-code

 Specify the actual GPU architecture to compile for. This option specifies which SM versions to compile for. If left unspecified, this option is inferred to be the SM version corresponding to the PTX version specified by --gpu-architecture. Valid options include: compute_10, compute_11, compute_12, compute_13, compute_20, compute_30, compute_35, sm_10, sm_11, sm_12, sm_13, sm_20, sm_21, sm_30, sm_35.

--generate-code

(-gencode)

 Specifies a tuple of virtual and actual GPU architectures to target. --generate-code arch=<arch>,code=<code> is equivalent to --gpu-architecture <arch> --gpu-code <code>.
--export-dir Specify the name of the directory to which all device code images will be copied.

--maxregcount <N>

(-maxregcount)

 Specify the maximum number of registers that GPU functions can use.

--ftz [true,false]

(-ftz)

 Flush-to-zero: when performing single-precision floating point operations, denormals are flushed to zero. --use-fast-math implies --ftz=true. The default is false.

--prec-div [true, false]

(-prec-div)

Precise division: if true, single-precision floating point division and reciprocals are performed to full precision (round-to-nearest-even with 0 ulps in error).

--use-fast-math implies --prec-div=false. The default value is true.
--prec-sqrt [true, false]

Precise square root: if true, single-precision floating point square root is performed to full precision (round-to-nearest-even with 0 ulps in error).

--use-fast-math implies --prec-sqrt=false. The default value is true.
--fmad [true, false] Enables or disables the contraction of floating-point multiplies and adds/subtracts into floating-point multiply-add (FMAD) instructions. This option is supported only when --gpu-architecture is compute_20, sm_20, or higher. For other architecture classes, the contraction is always enabled. -use_fast_math implies --fmad=true.
--use-fast-math Make use of fast math library. Besides implying --prec-div false, --prec-sqrt false, --fmad true, the single-precision runtime math functions are compiled directly to SFU intrinsics.

Table 4-7. nvcc options for code generation.

The --gpu-architecture and --gpu-code options are especially confusing. The former controls which virtual GPU architecture to compile for (i.e., which version of PTX to emit), while the latter controls which actual GPU architecture to compile for (i.e., which version of SM microcode to emit). The --gpu-code option must specify SM versions that are at least as high as the versions specified to --gpu-architecture.

The --export-dir option specifies a directory where all device code images will be copied. It is intended as a device code repository that can be inspected by the CUDA driver when the application is running (in which case the directory should be in the CUDA_DEVCODE_PATH environment variable). The repository can be either a directory or a ZIP file. In either case, CUDA will maintain a directory structure to facilitate code lookup by the CUDA driver. If a filename is specified but does not exist, a directory structure (not a ZIP file) will be created at that location.

Compiling for AMD GPUs

nvcc has a counterpart in AMD’s ROCm toolkit: hipcc, a compiler driver that plays the same role for AMD hardware, spawning the underlying amdclang++ and device tools much as nvcc drives ptxas and the host compiler. What hipcc consumes is HIP, a dialect close enough to CUDA that most kernels port by renaming the runtime calls—cudaMalloc becomes hipMalloc, cudaMemcpy becomes hipMemcpy—while __global__, <<<>>>, __shared__, and the thread built-ins carry over unchanged. HIP compiles for NVIDIA too, atop CUDA, so a single HIP source can target both vendors.

AMD ships hipify (as hipify-perl and hipify-clang) to automate that renaming, but it is a source-to-source translator, and source-to-source translation makes sense only as a one-time port—the way a codebase is migrated from 32-bit to 64-bit, after which the original is abandoned and only the translated copy is maintained. A project that keeps shipping on both NVIDIA and AMD never reaches that endpoint: every change to the CUDA source has to be re-translated, and the HIP copy drifts into a second codebase to maintain. For code that must stay portable, a single source guarded by a thin compatibility layer—as the samples in this book do, branching on __HIPCC__—is the durable arrangement.

A different approach keeps the CUDA source untouched and retargets the compiler instead. SCALE, from Spectral Compute, is an nvcc-compatible toolchain that compiles unmodified CUDA—the same <<<>>> launches, the same nvcc-style flags—directly for AMD GPUs, reimplementing the CUDA runtime and much of the dialect atop ROCm. Where its coverage reaches, an existing CUDA project builds for AMD with no source changes at all; where it does not—a CUDA-only library such as NPP, say—the affected code has to be excluded or replaced.