A stack1 is a region of memory with entries that are allocated in last-in, first-out order: in combination with a pointer that tracks the “top” of the stack, a push moves the pointer and writes, and a pop reads and moves the pointer back. Whenever a subroutine is called, a CPU’s native hardware support for stacks is invoked: the address of the next instruction is pushed, along with the subroutine’s parameters and local variables. When it’s time to return, those operations are reversed and control is returned to the address after popping it off the stack. Because running code uses different regions of stack from any subroutines it may call, subroutines may be nested, with each invocation pushing its own return address, arguments, and local variables. Nesting is bounded only by the memory set aside for the stack, and a program that uses more stack memory than was preallocated produces the error that gave Stack Overflow its name2.

Familiar as the arrangement may be, stacks had to be invented: Turing described them in the 1945 report proposing the ACE, where a subroutine’s return address was filed away by “a standard instruction table BURY, and the disinterring by the table UNBURY,” and Klaus Samelson and Friedrich Bauer applied in 1957 for a patent on the Keller, the cellar, whose contents are taken out in the reverse of the order they were put in.

Stacks were a well-understood feature of computer architecture long before x86 was designed in the mid-1970s. The Burroughs B 5000 had operated “on a last in, first out principle” since 1961, and allowed “complete freedom for using recursive procedures,” and the PDP-11 dedicated register 6 as its “hardware stack pointer,” which “even permits a routine to call itself in those cases where this is meaningful.”3

On x86, procedure calls are initiated with the CALL instruction, which pushes the return address onto the stack and then transfers control to the specified location in memory. RET performs the inverse operation: it loads the pushed address (named by Intel the return-instruction pointer), adjusts the stack pointer, and branches back to the caller.

Programmers were entrusted with a great deal of creative control over how to use these features. Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, section 6.2.4.2, states: “the processor does not keep track of the location of the return-instruction pointer. It is thus up to the programmer to ensure that stack pointer is pointing to the return-instruction pointer on the stack, prior to issuing a RET instruction.” The same chapter describes the optional argument to RET, which “allows a program to increment the stack pointer on a return to release parameters from the stack.”4

        CALL    compute         ; push the address of the next instruction
        MOV     RBX, RAX        ; <-- the pushed address points here
        ...
compute:
        ...
        RET                     ; pop that address into RIP, and jump to it

On x86, the target of every RET is in memory5, and in the early 1990s, Intel realized that stalling for the duration of that load was becoming a performance bottleneck for their new superscalar processors. The return stack buffer, added in the Pentium Pro (c. 1995), removes the wait by keeping its own copy. When the branch predictor identifies a CALL, it pushes the address of the instruction following it onto the return stack buffer, and when it identifies a RET, it pops the address on top and uses it as the prediction, so that the front end need not wait for a memory-dependent return target6.

The Pentium Pro’s return stack buffer held sixteen entries, a number that Intel kept through the Skylake generation (c. 2015).7 More recent architectures have increased the size of the buffer: AMD documents 32 entries on Zen 1 and Zen 2 (shared between hyperthreads), 32 per thread on Zen 3 and Zen 4, and 52 per thread on Zen 5. Intel published 16 for the Skylake generation, in the document that introduced retpolines, and Agner Fog measures 22 on Ice Lake and 128 on Gracemont8. When calls nest deeper than the return stack buffer size, the oldest entry is discarded, and the returns that eventually unwind past the buffer’s reach trigger a condition known as underflow. Underflowing the return stack buffer is not an error condition - it just reintroduces the stall that prompted Intel to create it in the first place.

A return address on the program’s stack sits in ordinary writable memory, so an attacker can send the program wherever they like by overwriting it. Stringing together such returns is enough to compose whole programs out of fragments of the victim’s own code, a technique Hovav Shacham named return-oriented programming. In response, Intel’s control-flow enforcement technology added a shadow stack, which records the return address of every call rather than the most recent handful; it shipped on Tiger Lake in 2020, and on AMD from Zen 3. RET pops from both the data stack and the shadow stack, and compares them, raising a control protection exception when they differ9. According to Intel’s manual, the shadow stack “is not used to store data, hence is not explicitly writeable by software,” and is “protected from tamper through the page table protections such that regular store instructions cannot modify the contents of the shadow stack.” The return stack buffer guesses and lets the architecture catch it later; the shadow stack checks and faults. The two coexist in the same core, one a covert cache that is permitted to be wrong, the other an architectural structure that is not. Intel’s manual is careful to say that enabling the shadow stack does not stop the speculation, since a return whose target has been predicted by the return stack buffer may still execute speculatively at that target before the check completes.

When the Hardware Misbehaves

Being incorrect costs only time, and AMD’s documentation estimates the penalty to be similar to a mispredicted conditional branch. Two patterns provoke it: overflowing the buffer (making more calls than the buffer has slots), and CALL and RET instructions that are not paired, which leaves the shadow stack out of step with the real stack. Code that leaves them unpaired includes pushing a return address and jumping to a routine, returning to an address other than the one the matching call stored, using a return as an indirect jump, and the long jump out of setjmp and longjmp.

Both vendors special-case one unpaired call, the zero-displacement CALL that x86 code has long used to discover its own address, which they exclude from the prediction rather than let it corrupt the stack; AMD recommends LEA RAX, [RIP+0] instead10.

The errata are all cases of the structure persisting across a boundary it was expected to respect. Post-barrier RSB predictions, disclosed in 2022, allow the linear address following the most recent near call before a virtual machine exit to serve as the prediction for the first near return afterward, which Linux’s commit message described as enhanced IBRS not working as documented11. A cross-thread bug on AMD’s Family 17h allows a change of simultaneous multithreading mode to move the return address pointers, so that one thread uses its sibling’s return targets. And an indirect branch prediction barrier, the instruction whose whole purpose is to discard what the predictors learned, has been found on some cores to leave return predictions behind.

The security history is longer. In 2018, ret2spec and SpectreRSB showed the buffer could be steered directly, and cheaply: a single store to the top of the stack, followed by a return, sends speculation wherever the attacker likes12. It mattered more than it otherwise would have, because the principal mitigation for Spectre v2 had recently made returns central to the defense. Retpolines replace each indirect branch with a return, precisely so that the prediction comes from the return stack buffer instead of the BTB, and arrange for the misprediction to land in a harmless infinite loop. Then, in 2022, Retbleed showed that returns could themselves be predicted from the BTB, which left the retpoline’s returns, and with them the mitigation’s whole premise, open to the poisoning they were designed to evade. The two vendors exhibit different failure modes: on Skylake-generation Intel cores, the fallback happens when a deep call chain has emptied the buffer, whereas on Zen 1 and Zen 2, a BTB entry takes precedence over the return stack buffer for any return at all, underflow or not.

In 2023, Inception took the opposite approach. Rather than emptying the buffer, it overflows one: a mispredicted branch that the machine briefly believes to be a call pushes a return address for a call that never happened, and recursion then overflows the stack with entries of the attacker’s choosing, leaking at 39 bytes per second on Zen 4. AMD tracks it as speculative return stack overflow, affecting Zen 1 through Zen 4, with microcode needed only for Zen 3 and Zen 4 because the earlier cores already flush branch type predictions; Linux now lists Zen 5 among the affected as well.

What happens when the buffer runs empty was believed to be settled for several years, but was not. The 2018 taxonomy had Intel cores wrapping around to reuse old entries and AMD cores simply declining to predict, but measurements published in 2025 find Zen cores reusing one of their oldest entries rather than stopping, and find the alternate-predictor behavior on some recent Intel cores and not others13.

How to Detect It

Names are Intel’s for Skylake and Ice Lake, and AMD’s for Zen 4.

• Retired return mispredictions, against retired returns.

  • Intel: BR_MISP_RETIRED.RET against BR_INST_RETIRED.NEAR_RETURN

  • AMD: ex_ret_near_ret_mispred against ex_ret_near_ret

Skylake counts retired returns, but it has no retired return-mispredict event at all, and BR_MISP_EXEC.INDIRECT excludes returns by definition. Retbleed’s authors worked around it by running a loop of nothing but calls and returns and subtracting retired call mispredictions from retired taken-branch mispredictions.

Two signals outside the performance counters say more about the mitigations than the structure. The files under /sys/devices/system/cpu/vulnerabilities/ report what the kernel thinks it is defending against, with spectre_v2 printing whether RSB filling is in effect, and spec_rstack_overflow reporting the Inception mitigation. The kernel also announces “Filling RSB on context switch and VMEXIT” in dmesg at boot.

Working Around It

Pair every call with a return. The buffer is only useful while its contents match the real stack, so the remedies are the ones that keep them matched:

• Return to the address the CALL pushed.

• Use a tail JMP rather than a CALL when the callee’s return can serve as the caller’s.

• Prefer LEA over a zero-displacement CALL to read the instruction pointer.

• Use inlining to reduce recursion depth that may exceed the buffer.

Intel’s optimization manual has given versions of this advice, in its rules for branch prediction, since long before any of it had any security implications.

Expect to pay for the mitigations. Filling the buffer with known-safe addresses at privilege boundaries, which Linux has done since January 2018 and made unconditional later that year, costs 32 writes at every context switch and virtual machine exit14. Beyond that lie the heavier measures: IBRS or call-depth tracking for Retbleed on Intel, where one kernel developer’s measurements put hackbench 21 percent slower under IBRS and 11 percent under stuffing; Safe RET for Inception on AMD, which works by forcing every function return to mispredict; and an indirect branch prediction barrier on kernel entry, which on Zen 1 more than tripled a single-core UnixBench run. Newer hardware has begun absorbing the cost, with Zen 4 and later clearing the return address predictor on a barrier, and Zen 5 tagging its entries by guest and host so that a virtual machine exit need not discard them.

Further Reading

• Retbleed — Returns predicted from the branch target buffer, and the difference between how Intel and AMD get there.

• Inception — The attack from the other end, overflowing the buffer with returns for calls that never happened.

• Wikner and Razavi, Breaking the Barrier: Post-Barrier Spectre Attacks — Current measurements of what each core does when the buffer runs empty, correcting a taxonomy that stood for seven years.