Most CUDA programming is based on a simple, abstract hardware model, as shown in Figure 1-1.
The CPU runs most of the application software, the operating system, and the CUDA driver software that operates the GPU.
The GPU is a peripheral that can run select parallel programs (called “kernels”). Generally the GPU has its own dedicated “device memory”; input data must be copied to device memory in order for the GPU to process it, and the GPU output must be copied from device memory. Usually these data copies are performed over the PCI Express bus.
Figure 1-1. CPU GPU Architecture
Following CUDA terminology, we will interchangeably use the terms “CPU” and “host”, “GPU” and “device.”
The remainder of Part I gives architectural overviews of CUDA hardware and software:
Chapter 2 expands on the simplified hardware model of Figure 1-1, detailing both the CUDA hardware platforms and the GPUs themselves;
Chapter 3 similarly covers the CUDA software architecture, and
Chapter 4 covers the CUDA software environment, including descriptions of CUDA software tools and cloud computing environments.
In Part II, Chapters 5-10 will cover various aspects of the CUDA programming model in great depth:
Chapter 5 covers memory, including device memory, constant memory, shared memory and texture memory;
Chapter 6 covers streams and events, the mechanisms used for “coarse-grained” parallelism between the CPU and GPU, between hardware units of the GPU such as copy engines and the Streaming Multiprocessors, or between discrete GPUs;
Chapter 7 covers kernel execution, including the dynamic parallelism feature introduced with SM 3.5;
Chapter 8 covers every aspect of Streaming Multiprocessors;
Chapter 9 covers multi-GPU applications, including peer-to-peer operations and embarrassingly parallel operations with N-body as an example;
Chapter 10 covers every aspect of CUDA texturing.
Finally, in Part III, Chapters 11-16 will discuss various targeted CUDA applications:
Chapter 11 describes bandwidth-bound, streaming workloads such as elementwise vector scaling;
Chapters 12 and 13 describe reduction and parallel prefix sum (otherwise known as scan), both important building blocks in parallel programming;
Chapter 14 describes N-body, an important family of applications with high computational density that derive an especial benefit from GPU computing; and
Chapter 15 takes an in-depth look at an image processing operation called normalized cross-correlation that is used for feature extraction. Chapter 15 features the only code in the book that uses texturing and shared memory together to deliver optimal performance.
Chapter 16 examines histograms and their relationship with Counting Sort and Radix Sort, which do not require comparisons to compute the sorted output and so are not subject to the worst-case O(NlgN) runtime of comparison-based sorts.