Much ink has been spilled describing the GPU revolution in computing. I have read with interest, because I myself got involved very early. While at Microsoft in the mid-1990s, I was the development lead for Direct3D when Intel and AMD were introducing the first multimedia instruction sets to accelerate floating point computation. Intel had already tried (unsuccessfully) to forestall the migration of clock cycles for 3D rasterization from their CPUs, by working with Microsoft to ship rasterizers that used their MMX instruction set. I knew that effort was doomed when we found that the MMX rasterizer, running on a yet-to-be-released Pentium 2 processor, was half as fast as a humble S3 Virge GX rasterizer that was available for sale.
For Direct3D 6.0, we worked with CPU vendors to integrate their code into our geometry pipeline so developers could transparently benefit from vendor-optimized code paths that used new instruction sets from Intel and AMD. Game developers embraced the new geometry pipeline; but it did not forestall the continued migration of clock cycles from the CPU to the GPU, as the new instruction sets were used to generate vertex data for consumption by GPUs’ hardware geometry pipelines.
About this time, the number of transistors on GPUs overtook the number of transistors on CPUs. The crossover was in 1997-1998, when the Pentium 2 and the NVIDIA RIVA TNT both had transistor counts of about 8M. Subsequently, the Geforce 256 (15M transistors), Geforce 2 (28M transistors), and Geforce 3 (63M transistors) all had more transistors than contemporary CPUs. Additionally, the architectural differences between the two devices were becoming clear: most of the die area for CPUs was dedicated to cache, while most of the die area for GPUs was dedicated to logic. Intel was able to add significant new instruction set extensions (MMX, SSE, SSE2, etc.) with negligible area cost because their CPUs were mostly cache. GPUs were designed for parallel throughput processing; their small caches were intended more for bandwidth aggregation than to reduce latency.
While companies like ATI and NVIDIA were building GPUs that were both faster and increasingly capable, CPU vendors continued to drive clock rates higher as Moore's Law enabled both increased transistor budgets and increased clock speeds. The first Pentium (c. 1993) had a clock rate of 60 MHz, while MMX-enabled Pentiums c. 1997 had clock rates of 200 MHz. By the end of the decade, clock rates had exceeded 1,000 MHz. But shortly thereafter, an important event in the history of computing took place: Moore's Law hit a wall. The transistors would continue to shrink, but clock rates could not continue to increase.
The event was not unexpected. Pat Gelsinger of Intel, delivering a keynote at the 2001 IEEE Solid-State Circuits Conference, said that if chips continued on their current design path, they would be as hot as nuclear reactors by the end of the decade and as hot as the surface of the sun by 2015. In the future, performance would have to come from “simultaneous multithreading” (SMT), possibly supported by putting multiple CPU cores on a single chip. Indeed, that is exactly what CPU vendors have done – today, it is difficult to impossible to find a desktop PC with a single-core CPU. But the decades-long free ride delivered by Moore's Law, in which increased clock rates enabled applications to run faster with little to no effort on the part of software developers, was over. Multicore CPUs require multithreaded applications. Only applications that benefit from parallelism can expect increased performance from CPUs with a larger number of cores.
GPUs were well-positioned to take advantage of this new trend in Moore's Law. While CPU applications that had not been authored with parallelism in mind would require extensive refactoring (if they could be made parallel at all), graphics applications were already formulated in a way that exploited the inherent parallelism between independent pixels. For GPUs, increasing performance by increasing the number of execution cores was a natural progression. In fact, GPU designers tend to prefer more cores over more-capable cores: they eschew strategies that CPU manufacturers take for granted, like maximizing clock frequency (GPUs never have, and still do not, run at clock rates approaching the limits of transistor fabrication), speculative execution, branch prediction, and store forwarding.
And to prevent this ever-more-capable processor from becoming I/O bound, GPU designers integrated memory controllers and worked with DRAM manufacturers to enable bandwidths that far exceeded the amount of bandwidth available to CPUs.
But that abundant horsepower was difficult for non-graphics developers to exploit. Some adventurous souls used graphics APIs such as Direct3D and OpenGL to subvert graphics hardware to perform non-graphics computations. The term “GPGPU” (general-purpose GPU programming) was invented to describe this approach; but for the most part, the computational potential of GPUs remained untapped until CUDA. Ian Buck, whose Brook project at Stanford enabled simplified development of GPGPU applications, came to NVIDIA and led development of a new set of development tools that would enable non-graphics applications to be authored for GPUs much more easily. The result is CUDA, a proprietary toolchain from NVIDIA that enables C programmers to write parallel code for GPUs using a few easy-to-use language extensions.
Since its introduction in 2007, CUDA has been well-received. Tens of thousands of academic papers have been written that make use of the technology. It has been used in commercial software packages as varied as Adobe's CS5 and Manifold's GIS (Geographic Information System). For suitable workloads, CUDA-capable GPUs range from 5x to 400x faster than contemporary CPUs. The sources of these speedups vary. Sometimes the GPUs are faster because they have more cores; sometimes because they have higher memory bandwidth; sometimes because the application can take advantage of specialized GPU hardware not present in CPUs, like the texture hardware or the SFU unit that can perform fast transcendentals. Not all applications can be implemented in CUDA – in fact, not all parallel applications can be implemented in CUDA. But it has been used in a wider variety of applications than any other GPU computing technology.
I hope this book enables accomplished CUDA developers to get the most out of CUDA.