Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Technique:Recognizing compiler-generated code/Stack probes

From ReverseEngineering.wiki

Calls and loops that touch every page of a large stack frame so the guard page is hit in order: MSVC __chkstk, MinGW/Clang ___chkstk_ms, AArch64 __chkstk and Rust's __rust_probestack, with their register conventions.

A thread's stack grows into a guard page one page at a time. A function whose frame is larger than a page (4096 bytes) must touch each page in order, otherwise a skipped guard page lets code run into unmapped or foreign memory. The compiler emits a probe call or loop in the prologue.

Recognition

  • x64 Windows: mov eax, <frame size>; call __chkstk; sub rsp, rax. MSVC's x64 __chkstk takes the size in RAX, preserves it and does not adjust RSP (compiler-rt note: MSVC x64's __chkstk and MinGW's ___chkstk_ms do not adjust RSP). Unverified: on x86, _chkstk/_alloca_probe take the size in EAX and the helper itself moves ESP (not read in a source).
  • MinGW (GCC) and Clang on Windows x64/x86: ___chkstk_ms (x64) / __chkstk_ms (x86), in compiler-rt x86_64/chkstk.S: size in RAX, preserves RAX and RCX and does not adjust RSP; the caller subtracts afterwards. Loop: from rsp+24, sub rcx, 0x1000; test [rcx], rcx for each full page, then one more probe for the remainder.
  • AArch64 Windows: mov x15, #<size/16>; bl __chkstk; sub sp, sp, x15, lsl #4. The helper clobbers x16 and x17, probes with ldr xzr, [x17] one page at a time, and does not modify SP (compiler-rt aarch64/chkstk.S; ARM64EC uses __chkstk_arm64ec).
  • Rust on non-Windows x86/x86-64: __rust_probestack (rust-lang/compiler-builtins probestack.rs): frame size in RAX, loop sub rsp, 0x1000; test [rsp+8], rsp while more than a page remains, then a final sub rsp, remainder; test, finally restores RSP (add rsp, rax) so it does not change RSP or RAX. LLVM calls it via the function attribute probe-stack.
  • Inline probing: with inline-asm stack probes (-fstack-clash-protection, Rust inline-stack-probes) no call appears; a loop of sub rsp, 0x1000; mov [rsp], 0 or or [rsp], 0 is inlined instead.
  • Default probe size is 4096; the function attribute stack-probe-size changes it (llvm X86TargetLowering::getStackProbeSize). LLVM emits the probe call only on Windows/UEFI targets by default (getStackProbeSymbolName) unless the function has probe-stack.

Why a reverser cares

  • A function that calls __chkstk has locals of at least ~4 KB (often a big array or struct); the immediate before the call is the frame size.
  • It is not a real call: do not let it end the analysis; model it as probing memory only, with the register effects of the variant above (AArch64 clobbers x16, x17).
  • Frames under 4096 bytes need no probe, so a very small immediate is not a probe argument.

Evidence & references

  • llvm-project compiler-rt/lib/builtins/x86_64/chkstk.S, i386/chkstk.S, aarch64/chkstk.S; llvm/lib/Target/X86/X86ISelLowering.cpp (getStackProbeSymbolName).
  • rust-lang/compiler-builtins compiler-builtins/src/probestack.rs.