Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Technique:Recognizing compiler-generated code/Stack cookies

From ReverseEngineering.wiki

MSVC /GS security cookies (__security_cookie, __security_check_cookie, __GSHandlerCheck) and the GCC/Clang stack protector (fs:0x28 canary, __stack_chk_fail): what the prologue and epilogue look like and which values and symbols identify them.

A stack cookie (canary) is a value stored between local buffers and the return address in the prologue and compared in the epilogue; a mismatch calls a failure routine. It is compiler scaffolding: skip it when reading.

GCC and Clang (stack protector)

  • On Linux/glibc, musl, Android and Fuchsia targets LLVM reads the guard from a thread-control-block slot: %fs:0x28 on x86-64 (%gs:0x28 with the kernel code model) and %gs:0x14 on i386, unless -mstack-protector-guard-* options change register, offset or symbol. Fuchsia uses offset 0x10. Source: llvm X86TargetLowering::getIRStackGuard in llvm/lib/Target/X86/X86ISelLoweringCall.cpp.
  • Pattern: prologue mov rax, fs:[28h]; mov [rsp+X], rax; xor eax, eax; epilogue mov rax, [rsp+X]; sub rax, fs:[28h] (or xor/cmp); jne to a block that calls __stack_chk_fail (no return). Where the guard is a global instead, the variable is named __stack_chk_guard.
  • GCC's libssp (libssp/ssp.c) defines uintptr_t __stack_chk_guard, fills it from the OS random source in a constructor (CryptGenRandom on Windows, /dev/urandom elsewhere) and, if no randomness is available, falls back to a terminator canary: lowest byte 0, second-highest byte \n (0x0A), highest byte 0xFF. Unverified: glibc's own canary also has a zero low byte (not read here).
  • The failing path calls __stack_chk_fail (libssp prints *** stack smashing detected ***: terminated); the string is a quick identifier of -fstack-protector builds. Unverified: exact message text varies by libc.

MSVC (/GS)

  • The cookie is the CRT global __security_cookie (its address is also stored in the PE load configuration directory, see Format:PE (Portable Executable)/Load config directory). __security_init_cookie runs at startup; the default (uninitialized) value is 0xBB40E64E on x86 and 0x00002B992DDFA232 on x64. Checked: ReactOS sdk/lib/crt/startup/gs_support.c (master 52eaf78aa0, lines 18-25 and 49-50, 61, 91-92 read) defines DEFAULT_SECURITY_COOKIE as 0x00002B992DDFA232ll under _WIN64 and 0xBB40E64E otherwise, initialises __security_cookie to it (with __security_cookie_complement = its bitwise not), skips randomisation when the cookie already differs from the default, and adds 1 if the random result equals the default. This is a reimplementation of Microsoft's gs_support.c, so real MSVC binaries are expected to match; not compared with a Microsoft CRT binary.
  • Prologue (x64): mov rax, cs:__security_cookie; xor rax, rsp; mov [rsp+...], rax. Epilogue: mov rcx, [rsp+...]; xor rcx, rsp; call __security_check_cookie. On x86 the xor operand is ebp (xor eax, ebp) and the check is mov ecx, [ebp-4]; xor ecx, ebp; call @__security_check_cookie@4 (fastcall, argument in ECX). Clang for MSVC targets does the same: LLVM declares the global __security_cookie and the fastcall __security_check_cookie with the argument in a register (X86ISelLoweringCall.cpp, insertSSPDeclarations).
  • The check function compares ECX/RCX to the global and on mismatch calls __report_gsfailure (fast-fail). Unverified.
  • __GSHandlerCheck and __GSHandlerCheck_SEH appear as the language-specific handler in x64 UNWIND_INFO of functions with /GS and cookie-checked exception data; the handler data holds a GS cookie offset. See Format:PE (Portable Executable)/Exception directory. Unverified.
  • x86 SEH4 frames also xor their scope table pointer with the cookie: see Technique:Recognizing compiler-generated code/x86 SEH frames.

Pitfalls

  • Decompilers show the xor-with-rsp as odd arithmetic on a local; ignore it. Functions without arrays or address-taken locals have no cookie, so its absence says nothing about the build flags.
  • Wine's msvcrt does not validate cookies in _except_handler4_common (source comment FIXME: no cookie validation yet).

Evidence & references

  • llvm-project llvm/lib/Target/X86/X86ISelLoweringCall.cpp (getIRStackGuard, insertSSPDeclarations).
  • gcc-mirror/gcc libssp/ssp.c (guard setup, terminator canary).
  • wine dlls/msvcrt/except_i386.c.