I have been programming computers professionally since 1982, and on the way, have read huge numbers of books, articles (both academic and trade press), blogs, and other materials. There is a lot of great material out there!1

At the same time, some content distinguishes itself because I find myself referencing it over, and over and over across the years. The small list I’ve compiled reflects how I have spent most of my career twiddling bits in user mode.

Without further ado, and in no particular order, here’s a list of resources that have been on my shelves, serving me well, in some cases for decades.

Textbooks

Introduction to Algorithms, 4th ed. by Cormen, Leiserson, Rivest and Stein.

Now in its fourth edition, this book has been the best text on the subject since it came out in 1990, just in time for the undergraduate course in algorithms I took as a sophomore. Since the book was first published, algorithm design necessarily has evolved to center parallel architectures to a greater degree; and these days, memory operations matter a lot more than the number of instructions executed. But the math and the tools described to prove algorithms correct, and analyze their performance, are timeless. The chapters naturally progress in complexity, from simple sorting algorithms and data structures such as binary trees, to chapters on Dynamic Programming, Greedy Algorithms, and Amortized Analysis, to advanced topics where they really can only give an overview and point the reader at resources.

The only nit I can pick with this book is that early editions had a chapter on parallel algorithms that covered PRAM, a good fit for massively parallel architectures like GPUs; but it has since been replaced by a chapter on threading, perhaps due to co-author Charles Leiserson’s affinity for the Cilk programming language. But parallel programming is such a broad topic that it could only get an introductory chapter with references, anyway.

This book is a gem.

Programming Pearls, 2nd ed. by John Bentley

Back in the 1980s, CMU professor Jon Bentley wrote a monthly column for Communications of the ACM called “Programming Pearls,” and this book is a compendium of the best of those columns. As a freshman at Duke University (c. 1988), this book was on the must-buy list for the mandatory intro-to-programming course, even though it did not figure into the curriculum as far as I could tell. I was immediately hooked: he talks about everything from DSLs (though he calls them “little languages”) to binary heaps (though he just calls them “heaps”). Bentley’s evident love for the craft of programming, coupled with clear explanations in an approachable style, still serve as an inspiration.

Hacker’s Delight, 2nd ed. by Henry S. Warren

With a title like that, who can resist? This book is the densest collection of bit twiddling recipes you can find anywhere. Whether you need a branchless conditional negate, need to figure out how to enlist floating point hardware to find the most significant set bit, or want to understand Gray codes or why integer division is so damn difficult, this book’s got your back! It’s well-organized, but if you are anything like me, you may lose yourself just perusing it, then going back to find the little kernels of wisdom when needed.

Some of the tricks described in this book have been subsumed into hardware (his panoply of population count algorithms has been obsoleted on some platforms by native machine instructions), and compilers have improved at making it unnecessary to spell out some of the optimizations (see ‘branchless conditional negate’ above) - largely because compiler writers have spent the last few decades poring over this book and figuring out ways to translate code to use the idioms described here.

By the way, if you enjoy Hacker’s Delight, you need to know about the Stanford bit twiddling hacks maintained by Sean Eron Anderson.

The Mythical Man-Month, by Fred Brooks

This oft-cited text on software engineering is worth a re-read every couple of years. If your first thought on hearing the title is, Oh that’s the one where he discovered that doubling the size of a software engineering team causes schedules to slip, because of communications overhead… drop everything, buy a copy if you don’t already have one, and reread this book. Every time I read it, I’m struck anew by some durable truth Brooks articulates, and I think, should we be struck at how much software engineering has changed in the last fifty years, or how little? Reading this book will incline you to the latter. No one was talking about “scrums” or “agile programming” when Brooks wrote this book, but many key insights he shares still hold true today. His observation that some software engineers are substantially more productive (one or two orders of magnitude) than average (we’ve all seen this) is immediately followed by the observation that there just aren’t enough such engineers to do all the required work. In his discussion of roles, from QA to software architects to language lawyers (and he uses that term), is as relevant today as it was back then.

Come to think of it, I have this book on my Kindle and it has been a while since I reread it2.

Performance Aides

A Primer on Memory Consistency and Cache Coherence, by Vijay Nagarajan, Daniel J. Sorin, Mark D. Hill, and David A. Wood

When the first edition of this novella-sized treatise was published, John Montrym handed it to me and said: “Read. This.” Montrym was a legendary GPU architect before he joined NVIDIA in the late 1990s, and he’s as warm and generous with his time as he is brilliant; so if he tells you to read something, you drop everything and comply.

This masterwork is required reading for anyone interested in parallel computing architectures, whether they be multicore CPUs, multi-socket servers, GPUs, or modern servers that feature both. It starts by defining Consistency and Coherence, their motivations, scalability problems that have arisen over the years, and solutions to those scalability problems. It includes a detailed description of TSO/x86, Intel’s “total store ordering” memory consistency model that was esoteric enough that it was nothing more than poorly-documented tribal knowledge before the Linux kernel team considered an optimization of their spin lock. According to this account (which has reference links):

Various experts went back and forth over whether the final MOV that sets a lock variable to 1 needed to be prefixed by LOCK or not. The discussion ended when Linus Torvalds said “I know that it is needed”. Only to see an Intel architect finally intervene and say “you know, really, it isn’t needed”. This was followed by a series of releases of Intel manuals documenting the x86 memory model, with increasing precision in each release. Intel also actually changed the published rules along the road, withdrawing some optimizations as they realized that they would break existing software.

The Primer is now in its second edition, which added a chapter on “accelerators” (mostly GPUs). It includes ample references for you to chase down for further study.

Intel Intrinsics Guide, Intel Corporation.

This resource is for every developer interested in accessing esoteric x86 instructions - especially SIMD instructions - unless you are a masochist, or have a solid engineering reason to bypass the compiler’s ability to use intrinsics to automate register allocation and instruction scheduling.

These days, writing SIMD code for x86 chips is akin to writing a legal brief: extensive research is needed, in part because there are annoying orthogonality misses in the ways the instructions were implemented. You have to confirm that the instruction set actually supports a feature (for example: until AVX512, there was no SIMD double-to-int64 conversion instruction. You could only do one of those at a time!), and that it’s supported across the full SIMD width of the instruction set.

The Intrinsics Guide lets you filter by ISA and search for either instruction mnemonics or intrinsic names, and once you find them, it has details on exactly how the instruction behaves, which Intel chips implemented it, which header file to include, and performance guidance (latency in clocks and throughput in CPI (clocks per instruction).

Optimization Manuals, Agner Fog.

If his Web site is any indication, Agner Fog is an accomplished Renaissance man; but for me, his manuals on x86 microarchitecture and x86 instruction tables are worth their weight in platinum.

To convey the scope of achievement encapsulated in these works, consider the microarchitecture manual is copyrighted 1996-2025. (It was last updated in September!) Execution pipelines, micro-op caches, instruction decoding, branch prediction, cache architectures, partial register access, store forwarding… these are just a few of the details of CPU implementation you can glean from these pages.

Historical References

The Story of ISPC, by Matt Pharr.

ISPC (“Implicit SPMD Program Compiler”) is a toolchain designed to simplify SIMD instruction set programming: in contrast with vectorizing compilers, it strikes a more natural balance between enabling programmers to express their intent, and the compiler’s ability to generate the corresponding code. It hasn’t achieved as much adoption as I expected, though it has notched some significant wins of mindshare: DreamWorks has open-sourced their MoonRay renderer that uses ISPC.

Computer graphics nerds know Matt Pharr for his masterful book Physically Based Rendering, now in its fourth edition; but in ISPC, he also created one of the only viable ways to program SIMD instruction sets with human-readable syntax and without intrinsics3.

To me, this blog series is an important historical reference on the early GPU computing landscape, when CUDA was new. Starting with the early history of Larrabee (Intel’s manycore entree into the parallel computing competition of the late 2000s), Pharr details how his experiments with vectorizing compiler technology led to what we now know as ISPC, and how ISPC came to be open sourced by Intel. There are some interesting stories in there.

Oral history of Intel, by Bob Colwell

Bob Colwell was a CPU architect at Intel, starting in 1990, and worked on the Pentium Pro, the first CPU to implement a 4/1/1 decoder for the x86 instruction set and one that substantially improved on the Pentium’s limited superscalar features. He was the chief architect for the Pentium 4 and reportedly included 64-bit features in that chip which then were disabled.

This oral history is the size of a medium-sized book. Where the discussion touches on Itanium is particularly interesting to me, but there’s much more.

No matter your opinion of Intel, its place in history is secure, and Colwell both had an outsized hand in writing that history and also bore firsthand witness to many events that are recounted here.