Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Technique:Identifying compression libraries in binaries

From ReverseEngineering.wiki
Technique
On this wiki Facts · Code · Notes · Topics

How to tell which compression, database or scripting library a program carries, and its version, from stream magics seen in its data files and from constant tables, error strings and exported names in its code. A hub that links the per-library pages of the wiki.

Overview

A program's data files show which formats it uses (the magic of the first bytes), and its code shows which implementation reads them. The second matters: forks and re-implementations produce the same format but have different tables, strings and bugs. Work in this order: (1) classify the stream by magic, (2) search the binary for the implementation's tables, which a linker cannot drop because the decoder needs them, (3) read version strings or version constants, (4) check exported names and error strings.

Stream magics

Bytes Stream Page
1F 8B 08 gzip member Format:Gzip
78 01, 78 5E, 78 9C, 78 DA zlib stream (CMF/FLG) Format:Zlib stream
none raw DEFLATE (first byte holds BFINAL and BTYPE bits) Format:DEFLATE
28 B5 2F FD Zstandard frame Format:Zstandard
04 22 4D 18 LZ4 frame (legacy frame: 02 21 4C 18) Format:LZ4
FD 37 7A 58 5A 00 xz stream Format:XZ
5D 00 00 (properties byte usually 5D, then dictionary size) legacy .lzma Format:LZMA
none (bit stream, window size in the first bits) Brotli Format:Brotli
42 5A 68 ("BZh") + digit 31-39; blocks 31 41 59 26 53 59 (bit aligned) bzip2 Format:Bzip2, Library:Bzip2
FF 06 00 00 73 4E 61 50 70 59 Snappy framing stream (raw Snappy blocks have no magic) Format:Snappy, Library:Snappy
89 4C 5A 4F 00 0D 0A 1A 0A lzop file wrapper around LZO1X blocks (raw LZO has no magic) Library:LZO
"SQLite format 3" + 00 SQLite database Format:SQLite database file
1B 4C 75 61 PUC-Rio Lua chunk Format:Lua bytecode
1B 4C 4A LuaJIT chunk Format:LuaJIT bytecode
FF 20 47 6F 20 62 75 69 6C 64 69 6E 66 3A Go build info, dates a Go binary Format:Go buildinfo

Game and Microsoft-specific compressors (no file magic unless stated):

Bytes Stream Page
8C 06 / 8C 0A / 8C 0C (also 0C, 4C) RAD Oodle block: Kraken / Mermaid-Selkie / Leviathan; 8C 05 LZNA, 8C 0B BitKnit Library:RAD Oodle
low nibble E in the first byte (e.g. 9E, 1E) behind a zlib-like wrapper LZHAM (method 14) Format:LZHAM
10 or 11 + 24-bit LE size Nintendo LZ77 (GBA/DS BIOS LZ10, LZ11) Format:Nintendo LZ77 (LZ10 and LZ11)
none (container gives window) Microsoft LZX (CAB, CHM, WIM, XEX2) Format:LZX
256 bytes of 4-bit lengths Xpress Huffman (MS-XCA) Format:Xpress
16-bit LE chunk header, top nibble B LZNT1 (RtlCompressBuffer format 2) Format:LZNT1
none (68000 code) Sega Kosinski, Nemesis, Enigma Format:Kosinski, Format:Nemesis (Sega Mega Drive), Format:Enigma (Sega Mega Drive)

Oodle is recognised by the DLL name oo2core_N_win64.dll and the OodleLZ_Decompress export; see the library page.

Implementation tables and constants

  • zlib (madler): inflate tables lbase/lext/dbase/dext; the last two lext entries differ per release and give the version (Library:Zlib#Identification). zlib-ng copies the same inflate tables, so use its version strings (Library:Zlib-ng).
  • miniz: raw extra-bit tables (0,0,0,0,0,0,0,0,1,1,1,1,...,5,5,5,5,0,0,0), a 16-entry nibble CRC table, "out of memory"/"parameter error" strings (Library:Miniz).
  • libdeflate: unpadded 28/30-entry slot tables, libdeflate_* names, no run-time version string (Library:Libdeflate).
  • CRC-32 table (00000000 77073096 EE0E612C 990951BA little-endian words): shared by zlib, PNG, ZIP, Ethernet and countless private copies, so it is a weak hint (Fact:Zlib/signature/crc table start).
  • Zstandard: precomputed default FSE decoding tables and the level table (Library:Libzstd). LZ4: LZ4_versionNumber returns a constant (MAJOR10000 + MINOR100 + RELEASE) (Library:Liblz4). Brotli: the 122,784-byte static dictionary starting timedownlifeleftback (Library:Libbrotli). xz/LZMA: Library:Liblzma.
  • bzip2: BZ2_crc32Table is the MSB-first CRC (00000000 04C11DB7 09823B6E), version string 1.0.8, 13-Jul-2019, exports BZ2_bz* (Library:Bzip2). Snappy: no tables or version string; C names snappy_compress/snappy_uncompress (Library:Snappy). LZO: version string 2.10 with date Mar 01 2017 and lzo1x_* names (Library:LZO).
  • Cryptographic and checksum tables found next to compressors (CRC-32, Adler, hash constants): Technique:Identifying cryptographic constants.
  • SQLite: the library that last wrote a database file stores its SQLITE_VERSION_NUMBER at file offset 0x60, so data files date the embedded library (Library:SQLite).
  • Lua: PUC-Rio versions are told apart by the luaP_opmodes table (Library:Lua); LuaJIT has assembler VM core, the version string LuaJIT 2.1. and the luaJIT_version_2_1_* export (Library:LuaJIT).
  • Go: Format:Go buildinfo gives the toolchain and module list, Format:Go pclntab the functions.

See also

For compressed save files see Technique:Reverse engineering game save files.

Pitfalls

  • One binary often holds several copies (a browser or Node.js build carries zlib twice); search for every table and compare versions.
  • MSVC's linker drops unreferenced copyright banners; GCC and clang builds keep them. Tables survive in both.
  • A static table match proves the format, not the library: confirm with a second marker (name, string, version constant) before naming a product.
  • Obfuscated games change magics (a custom first byte, XOR over the header); check whether the stream after a few bytes of offset passes a real decoder.

Open questions

  • Version strings of older bzip2 releases (0.9.5, 1.0.0-1.0.7) and of lzop headers were not collected.