Technique:Identifying compression libraries in binaries
| On this wiki | Facts · Code · Notes · Topics |
|---|
How to tell which compression, database or scripting library a program carries, and its version, from stream magics seen in its data files and from constant tables, error strings and exported names in its code. A hub that links the per-library pages of the wiki.
Overview
A program's data files show which formats it uses (the magic of the first bytes), and its code shows which implementation reads them. The second matters: forks and re-implementations produce the same format but have different tables, strings and bugs. Work in this order: (1) classify the stream by magic, (2) search the binary for the implementation's tables, which a linker cannot drop because the decoder needs them, (3) read version strings or version constants, (4) check exported names and error strings.
Stream magics
| Bytes | Stream | Page |
|---|---|---|
| 1F 8B 08 | gzip member | Format:Gzip |
| 78 01, 78 5E, 78 9C, 78 DA | zlib stream (CMF/FLG) | Format:Zlib stream |
| none | raw DEFLATE (first byte holds BFINAL and BTYPE bits) | Format:DEFLATE |
| 28 B5 2F FD | Zstandard frame | Format:Zstandard |
| 04 22 4D 18 | LZ4 frame (legacy frame: 02 21 4C 18) | Format:LZ4 |
| FD 37 7A 58 5A 00 | xz stream | Format:XZ |
| 5D 00 00 (properties byte usually 5D, then dictionary size) | legacy .lzma | Format:LZMA |
| none (bit stream, window size in the first bits) | Brotli | Format:Brotli |
| 42 5A 68 ("BZh") + digit 31-39; blocks 31 41 59 26 53 59 (bit aligned) | bzip2 | Format:Bzip2, Library:Bzip2 |
| FF 06 00 00 73 4E 61 50 70 59 | Snappy framing stream (raw Snappy blocks have no magic) | Format:Snappy, Library:Snappy |
| 89 4C 5A 4F 00 0D 0A 1A 0A | lzop file wrapper around LZO1X blocks (raw LZO has no magic) | Library:LZO |
| "SQLite format 3" + 00 | SQLite database | Format:SQLite database file |
| 1B 4C 75 61 | PUC-Rio Lua chunk | Format:Lua bytecode |
| 1B 4C 4A | LuaJIT chunk | Format:LuaJIT bytecode |
| FF 20 47 6F 20 62 75 69 6C 64 69 6E 66 3A | Go build info, dates a Go binary | Format:Go buildinfo |
Game and Microsoft-specific compressors (no file magic unless stated):
| Bytes | Stream | Page |
|---|---|---|
| 8C 06 / 8C 0A / 8C 0C (also 0C, 4C) | RAD Oodle block: Kraken / Mermaid-Selkie / Leviathan; 8C 05 LZNA, 8C 0B BitKnit | Library:RAD Oodle |
| low nibble E in the first byte (e.g. 9E, 1E) behind a zlib-like wrapper | LZHAM (method 14) | Format:LZHAM |
| 10 or 11 + 24-bit LE size | Nintendo LZ77 (GBA/DS BIOS LZ10, LZ11) | Format:Nintendo LZ77 (LZ10 and LZ11) |
| none (container gives window) | Microsoft LZX (CAB, CHM, WIM, XEX2) | Format:LZX |
| 256 bytes of 4-bit lengths | Xpress Huffman (MS-XCA) | Format:Xpress |
| 16-bit LE chunk header, top nibble B | LZNT1 (RtlCompressBuffer format 2) | Format:LZNT1 |
| none (68000 code) | Sega Kosinski, Nemesis, Enigma | Format:Kosinski, Format:Nemesis (Sega Mega Drive), Format:Enigma (Sega Mega Drive) |
Oodle is recognised by the DLL name oo2core_N_win64.dll and the OodleLZ_Decompress export; see the library page.
Implementation tables and constants
- zlib (madler): inflate tables
lbase/lext/dbase/dext; the last twolextentries differ per release and give the version (Library:Zlib#Identification). zlib-ng copies the same inflate tables, so use its version strings (Library:Zlib-ng). - miniz: raw extra-bit tables (
0,0,0,0,0,0,0,0,1,1,1,1,...,5,5,5,5,0,0,0), a 16-entry nibble CRC table, "out of memory"/"parameter error" strings (Library:Miniz). - libdeflate: unpadded 28/30-entry slot tables,
libdeflate_*names, no run-time version string (Library:Libdeflate). - CRC-32 table (
00000000 77073096 EE0E612C 990951BAlittle-endian words): shared by zlib, PNG, ZIP, Ethernet and countless private copies, so it is a weak hint (Fact:Zlib/signature/crc table start). - Zstandard: precomputed default FSE decoding tables and the level table (Library:Libzstd). LZ4:
LZ4_versionNumberreturns a constant (MAJOR10000 + MINOR100 + RELEASE) (Library:Liblz4). Brotli: the 122,784-byte static dictionary startingtimedownlifeleftback(Library:Libbrotli). xz/LZMA: Library:Liblzma. - bzip2:
BZ2_crc32Tableis the MSB-first CRC (00000000 04C11DB7 09823B6E), version string1.0.8, 13-Jul-2019, exportsBZ2_bz*(Library:Bzip2). Snappy: no tables or version string; C namessnappy_compress/snappy_uncompress(Library:Snappy). LZO: version string2.10with dateMar 01 2017andlzo1x_*names (Library:LZO). - Cryptographic and checksum tables found next to compressors (CRC-32, Adler, hash constants): Technique:Identifying cryptographic constants.
- SQLite: the library that last wrote a database file stores its SQLITE_VERSION_NUMBER at file offset 0x60, so data files date the embedded library (Library:SQLite).
- Lua: PUC-Rio versions are told apart by the
luaP_opmodestable (Library:Lua); LuaJIT has assembler VM core, the version stringLuaJIT 2.1.and theluaJIT_version_2_1_*export (Library:LuaJIT). - Go: Format:Go buildinfo gives the toolchain and module list, Format:Go pclntab the functions.
See also
For compressed save files see Technique:Reverse engineering game save files.
Pitfalls
- One binary often holds several copies (a browser or Node.js build carries zlib twice); search for every table and compare versions.
- MSVC's linker drops unreferenced copyright banners; GCC and clang builds keep them. Tables survive in both.
- A static table match proves the format, not the library: confirm with a second marker (name, string, version constant) before naming a product.
- Obfuscated games change magics (a custom first byte, XOR over the header); check whether the stream after a few bytes of offset passes a real decoder.
Open questions
- Version strings of older bzip2 releases (0.9.5, 1.0.0-1.0.7) and of lzop headers were not collected.