Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Technique:Graphics API vtable hooking

From ReverseEngineering.wiki
Technique
On this wiki Facts · Code · Notes · Topics

How overlays and capture tools reach a game's Direct3D/DXGI calls: COM objects are a pointer to a vtable shared by all instances, so patching one slot (e.g. IDXGISwapChain::Present, slot 8) intercepts the frame.

Idea

Every D3D9/DXGI/D3D11/D3D12 COM object starts with a pointer to a table of function pointers; the order is fixed by the interface (see Library:DXGI, Library:Direct3D 9, Library:Direct3D 11, Library:Direct3D 12 for slot tables). All objects of one implementation class share the table, so a hook placed on a slot of a dummy object also catches the game's own object.

Typical steps

  1. Create a hidden window and a dummy device (and swap chain), read *(void***)obj, take the slot address (slot8 on x64, slot4 on x86).
  2. Hook that function (inline detour) or overwrite the table entry (the table is in read-only memory; change protection first).
  3. Frame boundary: D3D9 Present 17 (or EndScene 42), DXGI IDXGISwapChain::Present 8 plus ResizeBuffers 13 (reset your resources there), Present1 22. D3D12 additionally needs the command queue (ExecuteCommandLists slot 10 of ID3D12CommandQueue).
  4. Draw an overlay with the matching backend of Library:Dear ImGui.

Notes

  • Another way in is a proxy d3d9.dll/dxgi.dll: Technique:DLL proxy loading.
  • Both big tools avoid dummy objects: they intercept the creation entry points and take the real objects the game gets.
    • ReShade (crosire/reshade @ 7bf9de8, BSD-3-Clause), source/dll_main.cpp: when ReShade is loaded as a proxy DLL (dxgi.dll, d3d9.dll, d3d11.dll, opengl32.dll...) it registers the system modules d3d9, d3d10, d3d10_1, d3d11, d3d12, dxgi, d2d1 and opengl32 with its hook manager (MinHook-based, source/hook.cpp), which inline-hooks the exported creation functions (CreateDXGIFactory/1/2, D3D11CreateDevice*, Direct3DCreate9...). In source/dxgi/dxgi.cpp (CreateDXGIFactory1, ~line 631-655) the returned real factory then gets its vtable slots patched: IDXGIFactory::CreateSwapChain (slot 10), IDXGIFactory2::CreateSwapChainForHwnd (15), ForCoreWindow (16), ForComposition (24); D3D9 does the same with IDirect3D9::CreateDevice (slot 16) and CreateDeviceEx (20) in source/d3d9/d3d9.cpp. The hooked CreateSwapChain calls the original, then init_swapchain_proxy returns a proxy object (new DXGISwapChain(factory, device, swapchain), dxgi.cpp ~358-373) in place of the real swap chain; D3D11/D3D12 devices are likewise wrapped (new D3D11Device, new D3D12Device). So Present is not found by vtable slot of the swap chain: the game holds a wrapper whose Present (DXGISwapChain::Present, dxgi_swapchain.cpp) is ReShade's own method. Vulkan is not hooked this way: the Vulkan layer mechanism is used.
    • RenderDoc (baldurk/renderdoc @ 45e9166, MIT): hooks are applied by patching the Import Address Table of every loaded module (renderdoc/os/win32/win32_hook.cpp, ApplyHook: VirtualProtect the IAT slot, overwrite it with the hook), plus hooks on LoadLibrary*/GetProcAddress so modules loaded later are patched too. Each API registers a library and its exports, e.g. renderdoc/driver/dxgi/dxgi_hooks.cpp registers dxgi.dll CreateDXGIFactory, CreateDXGIFactory1, CreateDXGIFactory2, and renderdoc/driver/d3d11/d3d11_hooks.cpp registers d3d11.dll D3D11CreateDeviceAndSwapChain (the plain D3D11CreateDevice funnels into it). The hook calls the original and wraps the results: new WrappedIDXGISwapChain4(*ppSwapChain, hwnd, wrapDevice) (d3d11_hooks.cpp ~1257), and factories are wrapped in RefCountDXGIObject::HandleWrap. Again the game receives wrapper objects; no dummy device is created.
    • Common point: hook the module exports that create the factory/device (by IAT patch or inline detour), then wrap or patch what they return. Static analysis target: look for imports of CreateDXGIFactory*/D3D11CreateDevice* being redirected, and for swap chain objects whose vtable lies outside dxgi.dll.
  • Detection of such hooks by the game or anti-cheat is outside this page's scope.

Related

Why vtable offsets look like [[rcx]+n] and what slots 0-2 are: Technique:Reading COM and WinRT code.

Code & repositories

  • https://github.com/crosire/reshade @ 7bf9de8b33bcc76c3177007e65d73c72dd0f34c0 · license: BSD-3-Clause
    ReShade: proxy DLL hooking D3D9/10/11/12, DXGI and OpenGL creation entry points and wrapping devices and swap chains
    Tested on: source read, not run: source read, not run · checked 2026-10-18; last activity 2026-09-24
  • https://github.com/baldurk/renderdoc @ 45e91661ee457e64279ff86265abdbe214aa6be0 · license: MIT
    RenderDoc: IAT-patching hook layer and wrapped D3D/DXGI objects for frame capture
    Tested on: source read, not run: source read, not run · checked 2026-10-18; last activity 2026-10-15