Most covert caches hold information derived from state that already exists, such as page tables, instruction bytes, or the history of control flow and memory accesses. The store buffer holds the opposite: data that the program has written, but that memory (including the cache hierarchy) has not yet received. When a store instruction executes, its address and data are added to the store buffer, where they wait until the store retires and the core acquires ownership of the destination cache line; only then is the data written into the L1 cache and made visible to other cores. In the meantime, the store buffer is the only place in the machine where the new value exists, which makes it a cache in reverse, holding the future contents of memory that will become canonical once the stores have been retired1.

The store buffer also is consulted like a cache: a load that reads bytes written by an earlier store must return that store’s data, so every load searches the store buffer in parallel with the L1 cache, and a hit forwards the data directly from the store to the load. That lookup is what the optimization manuals call store forwarding. Judged as a cache, the store buffer rarely hits: in a simulation of a Skylake-like core, only 8.1 percent of the loads in the SPEC CPU2006 benchmarks found their data there, since the buffer drains as quickly as it can, and stays mostly empty2. The low hit rate is also why the lookup runs in parallel with the L1 cache rather than ahead of it, because serializing the two would delay the 92 percent of loads that miss in the store buffer. The store buffer also is responsible for the one reordering that x86’s otherwise-strict memory model permits: a load later in program order may be satisfied before an earlier store to a different address becomes visible, because the store is still waiting in the buffer while the load reads from the cache.

That strict memory model constrains everything Intel and AMD do in the load and store path. Intel’s manual calls it write ordered with store-buffer forwarding; the researchers who formalized it, after years in which the vendors’ prose descriptions left programmers guessing, call it x86-TSO, after the total store order model that SPARC had defined formally by 19923. How much guessing was involved can be gauged from a 1999 debate on the Linux kernel mailing list, 119 messages long, over whether a spinlock could be released with an ordinary MOV instead of a locked instruction, so that the releasing core need not wait for its store buffer to drain. Sewell and his coauthors write that the question “seems to have been resolved only with input from Intel staff,” a decade before x86-TSO would be able to show from the model alone why the optimization was sound4.

The best extended treatment of the model, which I have recommended before, is the chapter “Total Store Order and the x86 Memory Model” in A Primer on Memory Consistency and Cache Coherence, which motivates and formalizes TSO, compares it with sequential consistency, and describes how a TSO machine implements atomic instructions and fences5. The rules governing x86-TSO are few: loads must not be reordered with other loads; stores must not be reordered with other stores, or with earlier loads; and processors other than the ones performing them must agree on the order of any two stores. The single relaxation, a load passing an earlier store to a different location, is exactly the latitude needed by the store buffer, and store forwarding is its corollary: a core may read its own stores before the rest of the system can see them, and nobody else may. (That is why Dekker’s mutual exclusion algorithm, and any other code that writes one flag and then reads another, needs an MFENCE or a locked instruction between the two, either of which waits for the store buffer to drain.) Intel’s manual observes that the model is “virtually identical” to the one used by the Intel486 and the Pentium, and lists the only enhancements made since: speculative reads and store-buffer forwarding (covered in this article), and out-of-order stores from fast-string operations6. Every x86 core since the Pentium Pro, which in 1995 was the first Intel processor to execute instructions out of order, has executed loads and stores out of order, dozens of them in flight at once, while producing results indistinguishable from those of a processor that did neither.

The machinery that reconciles out-of-order execution with x86-TSO treats stores and loads differently. Stores enter the store buffer and leave it in program order, so a store that misses in the cache holds up every store behind it, including those whose lines are already present; the core can request ownership of those lines in advance, but it cannot make their stores visible early. Loads are more difficult, because a core that executed its loads strictly in program order would give up much of the benefit of out-of-order execution. So x86 cores execute loads early and out of order, then check the result. If another core’s store invalidates a line that a speculatively executed load has already read, the load may have returned a value disallowed by the memory model, so the core discards the load and every instruction that followed. Intel’s performance counters call the event a memory ordering machine clear, and list its causes as memory disambiguation (the memory dependence predictor having guessed wrong), an external snoop, or a store from the other hyperthread hitting the load buffer7.

The costs of the x86-TSO memory model are hard to quantify on an x86 processor, which has no other mode to compare against. Apple’s M-series cores provide a natural experiment we can examine: to enable Rosetta 2 to run translated x86 code without strengthening every memory access, Apple built a TSO mode into its ARM cores, enabled per thread by a bit in an implementation-defined control register, and patches to enable it in Linux were posted in 20248. Researchers at Leibniz Universität Hannover ran the multithreaded SPEC CPU 2017 floating-point benchmarks in both modes on an M1 Ultra, and found TSO 8.94 percent slower on average than ARM’s native, weaker ordering9. The figure does not transfer directly, since Apple’s cores were designed around ARM’s model and x86 cores have had thirty years to optimize for x86-TSO, but it indicates the scale of what Intel and AMD have spent those thirty years engineering around. It also explains why the exceptions Intel has allowed itself (non-temporal stores, and the stores issued by fast-string operations) are confined to specific instructions10.

The store buffer comes with a predictor of its own, because a load whose address is known cannot safely execute ahead of an earlier store whose address is not yet known, since the store might turn out to write the location the load reads; but waiting for every earlier store address to resolve would serialize much of the program. So the core predicts, and a memory dependence predictor (the classic design is Chrysos and Emer’s store sets11) remembers which loads have previously been caught reading a location that an earlier, unresolved store was about to write, and holds back those loads while letting all other loads run ahead. It is allowed to be wrong, like the BTB, and so, like the BTB, it became an attack surface: Speculative Store Bypass (Spectre variant 4, 2018) contrived for a load to execute speculatively ahead of an earlier store to the same address, so that the load observed stale data (for example, a pointer that the store was about to sanitize).

The store buffer’s lookup also has been attacked. To decide quickly whether a load might need an earlier store’s data, the hardware first compares only part of their addresses, and in 2019 two groups found that on Intel cores the partial match can be enough: when a load’s address translation fails, and the low 12 bits of its address match those of an earlier store, the store’s data is forwarded to the load anyway, in the interval before the fault is handled. Fallout used this behavior to let an unprivileged process read values that the kernel had just written, and Store-to-Leak Forwarding used whether forwarding succeeded to distinguish mapped kernel addresses from unmapped ones, locating the randomized kernel in 42 microseconds on a Coffee Lake processor12. Intel classified the first as Microarchitectural Store Buffer Data Sampling, and neither attack was found to work on AMD or ARM cores.

How to Detect It

Names are Intel’s for Skylake and Ice Lake, and AMD’s for Zen 4.

• Cycles stalled because the store buffer is full.

  • Intel: RESOURCE_STALLS.SB

  • AMD: de_dis_dispatch_token_stalls1.store_queue_rsrc_stall

• Loads that could not be forwarded.

  • Intel: LD_BLOCKS.STORE_FORWARD

  • AMD: ls_bad_status2.stli_other

• 4K aliasing.

  • Intel: LD_BLOCKS_PARTIAL.ADDRESS_ALIAS

  • AMD: no event

• Machine clears for memory ordering.

  • Intel: MACHINE_CLEARS.MEMORY_ORDERING

  • AMD: resyncs_or_nc_redirects, which combines every cause of a resync

Sapphire Rapids renamed LD_BLOCKS_PARTIAL.ADDRESS_ALIAS to LD_BLOCKS.ADDRESS_ALIAS, and AMD’s cores have no event for the effect at all, although it occurs on them too, at a spacing of 1 KB on Zen 4 and Zen 5. AMD’s store queue event has been renamed twice: it is de_dis_dispatch_token_stalls1.store_queue_token_stall on Zen 2, and de_dispatch_stall_cycle_dynamic_tokens_part1.store_queue_rsrc_stall on Zen 5.

Working Around It

Reload data as it was stored. When data must make a round trip through memory rather than staying in registers, the store buffer can hand it directly to the dependent load. The one case every core forwards is a load that starts at the same address as the store, is no wider than the store, and is naturally aligned. Beyond that, how far a narrower load may be offset within the store differs by core13:

• Core 2, Nehalem, Silvermont, Goldmont, and Tremont forward to a narrower load only if it starts at the same address as the store.

• Skylake and Ice Lake forward to a narrower load at any offset within a store of 64 bits or less, at a cost of 1 to 3 cycles on Skylake and little or none on Ice Lake, but from a store of 128 bits or more, only to a load of one of its naturally aligned halves or quarters; any other partial read costs Skylake about 11 cycles, and fails to forward on Ice Lake.

• Zen 1 through Zen 3 forward “in all cases, including reads from a part of the written data.”

• Zen 4 and Zen 5 forward to a narrower load contained anywhere in the store, but charge about 11 and up to 28 cycles, respectively, when that load is unaligned.

A load that needs bytes from two stores, or that is wider than the store it overlaps (two 32-bit stores followed by a 64-bit load of the same location, e.g. when a structure is written field by field and then copied as a whole), generally cannot be forwarded, and must wait until the stores have left the store buffer. Applications with this memory traffic pattern may benefit from reformatting the data in SIMD registers before the store.

Keep loads and stores from colliding. Some collisions are only apparent, the most common being 4K aliasing: to decide quickly whether a load might depend on an earlier store, the hardware initially compares only the low 12 bits of their addresses, so a load from one buffer and a store to another whose addresses differ by a multiple of 4,096 are provisionally treated as dependent. On Zen 4 and Zen 5, Agner Fog finds the same false dependence at a spacing of only 1,024 bytes14. It is the BTB’s partial tag again, in a different structure, and the remedy is to offset buffers that are processed in lockstep (a source and a destination allocated as page-aligned blocks, for example) by an amount that is not a multiple of 1 KB, which avoids the effect on every core.

Only ask for the ordering you need. Because x86 already gives every ordinary load and store most of the ordering that a C++ program can request, acquire loads and release stores compile to ordinary MOV instructions, and the expensive cases are the ones that ask for more. A sequentially consistent std::atomic store must not be reordered with later loads, which is the one reordering x86-TSO permits, so on x86 the store compiles to an instruction that waits for the store buffer to drain: XCHG, or a MOV followed by MFENCE. Code that needs no more than release semantics should use memory_order_release. Spin-wait loops should contain a PAUSE instruction. Intel’s optimization manual explains that a processor exiting a spin-wait loop “detects a possible memory order violation and flushes the core processor’s pipeline,” and that PAUSE lets it “avoid the memory order violation and prevent the pipeline flush.”15 The MACHINE_CLEARS.MEMORY_ORDERING counter shows whether a program is paying for those flushes.

Leave the memory dependence predictor on unless the threat model rules it out. The store buffer itself cannot be switched off, but the prediction that lets loads run ahead of it can be. The Linux kernel describes Speculative Store Bypass as “a common industry wide performance optimization ... in which recent stores to the same memory location may not be observed by later loads during speculative execution,” and its spec_store_bypass_disable= parameter turns that optimization off, either for the whole machine or for chosen processes. The default on x86 is prctl, which leaves the optimization on until a process asks for the mitigation, and the seccomp setting extends the mitigation to every sandboxed thread that does not opt out16. A future article on memory renaming will cover the control in more detail, including the 2024 research that found it effective where the narrower predictive-store-forwarding control was not.

Further Reading

• Nagarajan, Sorin, Hill, and Wood, A Primer on Memory Consistency and Cache Coherence — Chapter 4 motivates total store order, formalizes it, and shows how fences and atomics are implemented under it. Free from the authors.

• Sewell and coauthors, x86-TSO: A Rigorous and Usable Programmer’s Model — The formalization that ended years of guessing at what the vendors’ prose meant.

• Henry Wong, Store-to-Load Forwarding and Memory Disambiguation in x86 Processors — Microbenchmarks of when forwarding succeeds and when it fails, on nine Intel and AMD microarchitectures from Core 2 to Haswell and Piledriver.

• Alves, Ros, Black-Schaffer, and Kaxiras, Filter Caching for Free: The Untapped Potential of the Store-Buffer — How rarely the store buffer hits, and what it could do if it were allowed to act as a cache on purpose.

• Wrenger, Töllner, and Lohmann, Analyzing the Memory Ordering Models of the Apple M1 — What strict ordering costs, measured on a core that can switch it off.