When the first edition of this book was written, renting a CUDA-capable server was a novelty offered by a single vendor; today, GPU capacity is a headline product of every major cloud and of an entire second tier of GPU-first providers. The economics that made the cloud attractive then are unchanged: customers pay per instance-hour for potentially unlimited compute without investing in their own infrastructure. What has changed is the scale and the stakes – the largest machine learning training runs are executed on thousands of cloud GPUs at a time, and for many developers, a cloud instance is the only practical way to touch data center hardware at all.
One of the subtle benefits of cloud computing is that it empowers developers to specify the hardware their software will run on: packaged software of the past had detailed “System Requirements” boxes that described the hardware their customer would have to procure to run the product well, but cloud developers can launch virtual machines with exactly the characteristics they need, and fit them together using the hardware and software primitives provided by the cloud vendor. GPU instances confer an exaggerated version of this benefit, partly because top-of-the-line GPUs require exotic power and cooling solutions, and partly because hyperscalers are better positioned to build out the data center interconnects that can keep the GPUs from starving.
A survey of cloud computing is outside the scope of this book, but a
few practical notes apply everywhere: every provider below offers NVIDIA
data center GPUs with machine images that come with drivers and the CUDA
toolkit preinstalled; nvidia-smi (Section 4.6) is the
quickest way to verify what you have been given; and pricing varies
enormously – between providers, between on-demand, reserved, and spot
capacity, and over time – so measure performance per dollar rather than
making assumptions. The benchmark suite that accompanies this book on cudahandbook.com/benchmarks
is one way to do exactly that.
The preeminent infrastructure-as-a-service vendor, and the pioneer of GPU instances. CUDA capacity spans the P family for training (P4d with A100, P5 with H100, and P6 with Blackwell-generation parts) and the G family for inference and graphics (G5 with A10G, G6 with L4 and L40S). Multi-node training is served by the Elastic Fabric Adapter interconnect and preassembled UltraClusters, and scarce accelerators can be reserved in advance through Capacity Blocks. The operational primitives the first edition described in detail – instances, block storage, object storage, key pairs, security groups – persist essentially unchanged; the Deep Learning AMIs spare you from installing the toolkit yourself. Container orchestration is fully supported – ECS, EKS, and Batch all schedule GPU containers – but the unit of capacity remains the instance: AWS’s serverless container offering, Fargate, does not extend to GPUs.
Compute Engine’s accelerator-optimized families cover the same ground: A2 (A100), A3 (H100 and H200), and G2 (L4), with Deep Learning VM images playing the role of AWS’s prebuilt AMIs. Google is also the one major cloud with a first-party alternative to NVIDIA hardware – its TPU family – which colors its GPU story: GPUs are offered as the compatibility choice, TPUs as the house specialty. Spot capacity is generously priced when available.
Azure’s GPU lineage splits into the NC family for general GPU compute and the ND family for deep learning, currently anchored by H100 v5 variants of each; ND-series clusters are joined by NVLink within a node and InfiniBand across nodes. Azure operates some of the largest GPU fleets ever assembled – it is the infrastructure behind OpenAI’s training runs – and its enterprise integration story is the strongest of the majors.
A late entrant that bought its way into relevance with aggressive pricing and an unusual offering: bare-metal GPU shapes (for example, eight-way H100 machines) with no virtualization layer at all, joined by RDMA cluster networking into superclusters of tens of thousands of GPUs. Several prominent AI labs train on OCI for exactly these reasons.
Alongside the hyperscalers, a class of GPU-first providers – CoreWeave, Lambda, Crusoe, and Nebius among them – buys accelerators at scale and rents little else. Born variously from cryptocurrency mining fleets and machine learning tooling companies, they compete on price, on early access to new parts, and on availability when the majors are capacity-constrained. Their operational idioms skew toward the HPC world – Kubernetes- and SLURM-native scheduling, InfiniBand fabrics as table stakes – and their catalogs are narrow by design: if what you need is GPUs, and only GPUs, they are frequently the cheapest and fastest way to get them.