Because a well-written CUDA program is limited by a hardware resource
– memory bandwidth, arithmetic throughput, or the latency between
dependent operations – optimizing one begins with measurement. The
current profilers are Nsight Systems and Nsight Compute, which replace
the first edition’s Visual Profiler and the nvprof
command-line tool, both since deprecated by NVIDIA.
Nsight Systems (nsys) is the system-wide profiler and
the right place to start. It records a timeline of the entire
application – CPU threads, CUDA API calls, kernel launches, memory
copies, and stream activity – and shows how they overlap, which answers
the questions that determine what to optimize: whether the GPU is busy
or waiting on the CPU, whether copies overlap with kernels or serialize
against them, whether one long kernel dominates or a thousand short ones
do, and where the gaps on the timeline are.
nsys profile ./myApp writes a report that opens in the
Nsight Systems GUI, and nsys profile --stats=true ./myApp
prints a summary to the console. Marking regions of interest with NVTX
ranges makes the timeline considerably easier to read.
Nsight Compute (ncu) is the kernel-level profiler, to be
reached for once Nsight Systems has identified which kernel matters. It
replays a single kernel launch to collect detailed hardware metrics –
achieved occupancy, memory throughput at each level of the hierarchy,
warp stall reasons, and instruction mix – and presents them with guided
analysis and a roofline chart that shows how close the kernel runs to
the memory and compute limits. ncu ./myApp profiles
interactively; ncu --set full -o report ./myApp collects
the full metric set into a file; and options such as
-k <regex> and --launch-count restrict
collection to the launches of interest, which matters because a full
profile replays each kernel many times. As with the sanitizer, building
with -lineinfo lets Nsight Compute attribute metrics to
individual lines of source and SASS. The two tools form one workflow:
Nsight Systems to find the kernel or the gap that dominates the run,
then Nsight Compute to understand and remove the limit within it.