Streams (Section 3.7) let the application overlap CPU and GPU work, but the application still issues each operation itself: every kernel launch and every memory copy is a separate driver call, and for each one the driver spends CPU time validating arguments and writing commands to the GPU. For a workload built from many small operations, typical of deep-learning inference and iterative solvers, that per-launch CPU cost can rival the GPU work it dispatches.
A CUDA graph is a record of a set of operations and the dependencies among them: a directed acyclic graph whose nodes are kernel launches, memory copies, memsets, host callbacks, and child graphs. The graph is defined once – either by capturing an existing stream of operations or by building the nodes explicitly – and then instantiated, in effect compiled: the driver resolves the dependencies and validates the arguments ahead of time, producing an executable graph that the application launches as a unit.
The motivation is to move that per-operation cost off the critical path. Instantiation does the expensive setup once; launching the instantiated graph replays every operation it contains from a single driver call, at lower CPU cost than issuing the operations one at a time. A workload that repeats the same sequence each iteration – a training step, a simulation timestep, a filter pipeline – incurs the dispatch overhead once and amortizes it over every launch that follows.
Because the work is described rather than issued, the driver also sees the entire dependency structure at once and can schedule it with less overhead than it would spend reconstructing that structure from a stream of individual calls. Section 6.9 develops the graph APIs – stream capture versus explicit construction, updating an instantiated graph, and launching graphs from the device – and measures the CPU savings against equivalent stream code.