Technique:Recognizing compiler-generated code/Stack probes
Appearance
Topic page of Technique:Recognizing compiler-generated code.
Calls and loops that touch every page of a large stack frame so the guard page is hit in order: MSVC __chkstk, MinGW/Clang ___chkstk_ms, AArch64 __chkstk and Rust's __rust_probestack, with their register conventions.
A thread's stack grows into a guard page one page at a time. A function whose frame is larger than a page (4096 bytes) must touch each page in order, otherwise a skipped guard page lets code run into unmapped or foreign memory. The compiler emits a probe call or loop in the prologue.
Recognition
- x64 Windows:
mov eax, <frame size>; call __chkstk; sub rsp, rax. MSVC's x64__chkstktakes the size in RAX, preserves it and does not adjust RSP (compiler-rt note: MSVC x64's __chkstk and MinGW's ___chkstk_ms do not adjust RSP). Unverified: on x86,_chkstk/_alloca_probetake the size in EAX and the helper itself moves ESP (not read in a source). - MinGW (GCC) and Clang on Windows x64/x86:
___chkstk_ms(x64) /__chkstk_ms(x86), in compiler-rtx86_64/chkstk.S: size in RAX, preserves RAX and RCX and does not adjust RSP; the caller subtracts afterwards. Loop: fromrsp+24,sub rcx, 0x1000; test [rcx], rcxfor each full page, then one more probe for the remainder. - AArch64 Windows:
mov x15, #<size/16>; bl __chkstk; sub sp, sp, x15, lsl #4. The helper clobbers x16 and x17, probes withldr xzr, [x17]one page at a time, and does not modify SP (compiler-rtaarch64/chkstk.S; ARM64EC uses__chkstk_arm64ec). - Rust on non-Windows x86/x86-64:
__rust_probestack(rust-lang/compiler-builtinsprobestack.rs): frame size in RAX, loopsub rsp, 0x1000; test [rsp+8], rspwhile more than a page remains, then a finalsub rsp, remainder; test, finally restores RSP (add rsp, rax) so it does not change RSP or RAX. LLVM calls it via the function attributeprobe-stack. - Inline probing: with
inline-asmstack probes (-fstack-clash-protection, Rustinline-stack-probes) no call appears; a loop ofsub rsp, 0x1000; mov [rsp], 0oror [rsp], 0is inlined instead. - Default probe size is 4096; the function attribute
stack-probe-sizechanges it (llvmX86TargetLowering::getStackProbeSize). LLVM emits the probe call only on Windows/UEFI targets by default (getStackProbeSymbolName) unless the function hasprobe-stack.
Why a reverser cares
- A function that calls
__chkstkhas locals of at least ~4 KB (often a big array or struct); the immediate before the call is the frame size. - It is not a real call: do not let it end the analysis; model it as probing memory only, with the register effects of the variant above (AArch64 clobbers x16, x17).
- Frames under 4096 bytes need no probe, so a very small immediate is not a probe argument.
Evidence & references
- llvm-project
compiler-rt/lib/builtins/x86_64/chkstk.S,i386/chkstk.S,aarch64/chkstk.S;llvm/lib/Target/X86/X86ISelLowering.cpp(getStackProbeSymbolName). - rust-lang/compiler-builtins
compiler-builtins/src/probestack.rs.