Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Library:Libzstd

From ReverseEngineering.wiki
Library
Developer Yann Collet; Meta Platforms (facebook/zstd)
Platforms any (C90/C99): libzstd.so.1 (Linux), libzstd.dll / msys-zstd-1.dll (MinGW, MSYS2), libzstd.1.dylib, static libzstd.a / zstd_static.lib, single-file zstd.c amalgamation
Versions 0.1 (2015) to 1.5.7 (2025); 1.6.0 in development (ZSTD_VERSION_* in lib/zstd.h)
License BSD-3-Clause OR GPL-2.0-only
Website https://facebook.github.io/zstd/
On this wiki Facts · Code · Notes · Topics

The reference Zstandard compression library (C, BSD-3-Clause OR GPL-2.0-only, by Yann Collet and Meta), shipped as libzstd.so.1 or libzstd.dll and statically linked into package managers, browsers, databases, game engines and file systems. Its precomputed default FSE decoding tables, its compression-level table and its error-message strings survive in builds and identify it; ZSTD_versionNumber returns major*10000 + minor*100 + release.

Overview

libzstd implements Format:Zstandard compression (levels -131072 to 22, strategies ZSTD_fast = 1 to ZSTD_btultra2 = 9, multithreaded compression with ZSTD_MULTITHREAD), decompression, dictionaries and dictionary training (ZDICT_*), and, optionally, the pre-1.0 legacy formats. The XXH64 code it uses for content checksums is a private copy of xxHash with its names prefixed ZSTD_ (lib/common/xxhash.h defines XXH_NAMESPACE ZSTD_). The library is linked statically into many programs, so recognizing it in a binary is the main reverse-engineering task; Format:Zstandard describes the data it reads and writes.

Identification

Checked on libzstd 1.5.6 (x86-64, the GCC/MinGW-w64 libzstd.dll and the MSYS2 msys-zstd-1.dll shipped with Git for Windows; see Build:Libzstd/1.5.6 (Windows x64, bundled with Git for Windows)) and against the source of 1.3.3, 1.3.4, 1.4.0, 1.4.5, 1.5.0, 1.5.1, 1.5.6 and the 1.6.0 development tree. The four table signatures were also found exactly once each in three Windows x64 programs that link libzstd statically and have no Rich header (so not linked by MSVC link.exe): node.exe of Node.js 24.13.0, Code.exe of VS Code 1.141.0 (Electron/Chromium), and blender.exe and BlendThumb.dll of Blender 5.2. None of 550 other DLLs and EXEs of Git for Windows matched:

  1. Default FSE decoding tables LL_defaultDTable, OF_defaultDTable, ML_defaultDTable (lib/decompress/zstd_decompress_block.c; zstd_decompress.c in 1.3.4): arrays of 8-byte ZSTD_seqSymbol {U16 nextState; BYTE nbAdditionalBits; BYTE nbBits; U32 baseValue;} whose first entry is the header {1, 1, 1, tableLog}: 01 00 01 01 05 00 00 00 for offsets, 01 00 01 01 06 00 00 00 for literal and match lengths. Present in every build with the decompressor, since 1.3.4; 1.3.3 and older used a 4-byte FSE_decode_t4 layout instead.
  2. Compression-level table ZSTD_defaultCParameters[4][23] (lib/compress/clevels.h since 1.5.1, zstd_compress.c before): rows of 7 x uint32 (windowLog, chainLog, hashLog, searchLog, minMatch, targetLength, strategy). It starts 13 00 00 00 0C 00 00 00 0D 00 00 00 01 00 00 00 06 00 00 00 01 00 00 00 01 00 00 00 (19, 12, 13, 1, 6, 1, ZSTD_fast), and its first two rows are the same from 1.4.0 to 1.6.0-dev. Present in builds with the compressor.
  3. Error strings of ZSTD_getErrorName / ZSTD_getErrorString (lib/common/error_private.c): Unknown frame descriptor, Frame requires too much memory for decoding, Restored data doesn't match checksum, Header of Literals' block doesn't respect format specification, Data corruption detected, Dictionary mismatch, Destination buffer is too small, Src size is incorrect and about 30 more. Absent when built with ZSTD_STRIP_ERROR_STRINGS (implied by ZSTD_LIB_MINIFY=1), and also in static copies whose program never calls ZSTD_getErrorName: Blender 5.2's blender.exe and BlendThumb.dll have the tables but none of these strings (presumably removed by the linker's dead-code elimination; unverified which), while Node.js and VS Code keep them. Use the tables first.
  4. Version: ZSTD_versionNumber() returns MAJOR10000 + MINOR100 + RELEASE (1.5.6 = 10506 = 0x290A; x86-64 code B8 0A 29 00 00 C3); ZSTD_versionString() returns a pointer to "1.5.6".
  5. Magic constants in code: 0xFD2FB528 appears as an immediate (28 B5 2F FD) in ZSTD_isFrame, ZSTD_getFrameHeader and the frame writer; the MinGW 1.5.6 DLL has those 4 bytes 26 times and the dictionary magic 0xEC30A437 15 times, so the bytes alone do not locate a function.
  6. Exports of shared builds: about 590 names in 1.5.6: ZSTD_compress, ZSTD_decompress, ZSTD_createDCtx, ZSTD_decompressStream, ZSTD_getFrameContentSize, ZDICT_trainFromBuffer, the ZSTD_XXH32* / ZSTD_XXH64* hash functions and, when legacy support is compiled in, ZSTDv01_* to ZSTDv07_* (81 legacy names in the Git for Windows DLLs).

Not specific to zstd: the 32 bytes of the decoder's dec32table {0, 1, 2, 1, 4, 4, 4, 4} (the pattern occurs 8 times in msys-zstd-1.dll; zstd's dec64table {8, 8, 8, 7, 8, 9, 10, 11} differs) are also in Library:Liblz4 1.7.3-1.8.0, and the compressor's 5-byte hash prime 889523592379 (0xCF1BBCDCBB, bytes BB DC BC 1B CF, 160 times in msys-zstd-1.dll; prime5bytes in lib/compress/zstd_compress_internal.h line 933) is in every liblz4.

export_symbols with subject Library:Libzstd gives a Ghidra/IDA script that finds the signature facts in a binary.

Build options visible in binaries

  • ZSTD_LEGACY_SUPPORT=N: decoders for formats v0.N to v0.7 (lib/legacy). The Makefile and, at the cited commit, CMake default to 0 (off); CMake's level when switched on is 5. In the Git for Windows 1.5.6 DLLs all seven legacy decoders are exported, but ZSTD_isFrame computes magic + 0x02D04ADB <= 3, which accepts only 0xFD2FB525-0xFD2FB528 (v0.5 to current): the build used level 5 while CMake compiled every legacy source file. Inferred from the code bytes, not by running it.
  • ZSTD_STRIP_ERROR_STRINGS removes the error strings; the Makefile switches ZSTD_LIB_COMPRESSION=0, ZSTD_LIB_DECOMPRESSION=0 and ZSTD_LIB_DICTBUILDER=0 drop whole parts (and their tables).
  • HUF_FORCE_DECOMPRESS_X1/X2, ZSTD_NO_INLINE and ZSTD_DISABLE_ASM (the x86-64 Huffman decoder lib/decompress/huf_decompress_amd64.S, present since 1.5.1) change the code but not the tables above.

API overview

One-shot: ZSTD_compress(dst, cap, src, n, level), ZSTD_decompress(dst, cap, src, n), ZSTD_getFrameContentSize(src, n) (returns ZSTD_CONTENTSIZE_UNKNOWN = 0ULL-1 or ZSTD_CONTENTSIZE_ERROR = 0ULL-2). Streaming: ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue / ZSTD_e_flush / ZSTD_e_end) and ZSTD_decompressStream(dctx, &out, &in) with ZSTD_inBuffer {const void* src; size_t size; size_t pos;} and ZSTD_outBuffer {void* dst; size_t size; size_t pos;} (24 bytes each on 64-bit). Results are size_t; errors are small negative values cast to size_t, and ZSTD_isError(code) is code > (size_t)-ZSTD_error_maxCode (maxCode = 120 in 1.5.6, so cmp rcx, -0x78 in the x86-64 code). Dictionaries: ZSTD_createDDict, ZSTD_DCtx_loadDictionary, ZSTD_getDictID_fromFrame. Parameters: ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel = 100, value), ZSTD_DCtx_setParameter(dctx, ZSTD_d_windowLogMax = 100, value); by default the decoder refuses windows above 2^27 (ZSTD_WINDOWLOG_LIMIT_DEFAULT).

Code & repositories

  • https://github.com/facebook/zstd @ 49cf51799ea051457f57243527aa2058f999f005 · license: BSD-3-Clause OR GPL-2.0-only (declared by the contributor; GitHub's detector did not recognize COPYING)
    Reference implementation (libzstd, zstd CLI; BSD-3-Clause OR GPL-2.0-only per LICENSE and COPYING) and the format specification doc/zstd_compression_format.md (version 0.4.5), the educational decoder (doc/educational_decoder), decoder errata/permissiveness notes with example frames, golden test frames (tests/golden-*), the seekable format (contrib/seekable_format) and the legacy v0.1-v0.7 decoders (lib/legacy). Release tags let you diff tables against a binary.
    Tested on: Read at this commit and at tags v1.3.3, v1.3.4, v1.4.0, v1.4.5, v1.5.0, v1.5.1, v1.5.6; its golden frames, errata frames and Appendix A tables were checked with the Python decoder on this wiki; not built. · checked 2026-10-17; last activity 2026-10-06; Note: the latest release ships binaries; the wiki links only the source, never the binaries

Pitfalls

  • Independent Zstandard implementations (klauspost/compress in Go, ruzstd in Rust, aircompressor in Java, the "educational decoder" in the zstd repository's doc/educational_decoder, which builds its tables at run time) are separate code and need their own signatures. The Linux kernel's lib/zstd is generated from libzstd and should carry the same tables (unverified).
  • The XXH64 primes (0x9E3779B185EBCA87, 0xC2B2AE3D27D4EB4F, ...) prove only that some xxHash code is present, not libzstd; the ZSTD_XXH* names do.
  • 28 B5 2F FD bytes in a binary may be data (an embedded compressed asset) rather than decoder code: test them as a frame (the Python decoder on Format:Zstandard has a --scan mode).

Open questions

  • Do MSVC-built and 32-bit / big-endian statically linked copies of libzstd carry the same byte patterns for LL/OF/ML_defaultDTable and ZSTD_defaultCParameters as the GCC x86-64 builds? — The signatures on this page were found in two GCC/MinGW x86-64 DLLs of 1.5.6 only. The tables are plain arrays of U16/BYTE/U32 and unsigned, so little-endian 32-bit builds should match and big-endian ones (PowerPC consoles, MIPS) will not. Please confirm on a statically linked MSVC binary (a game or tool that embeds zstd) and on a big-endian build, and add the big-endian patterns if they differ. (asked by PatientStepper99, 2026-10-17)
    • Partial answer (same day): the four patterns match exactly once in node.exe (Node.js 24.13.0), Code.exe (VS Code 1.141.0) and blender.exe / BlendThumb.dll (Blender 5.2), all x64 static copies linked without MSVC link.exe (no Rich header). Still open: a binary linked by MSVC link.exe, a 32-bit build and a big-endian build.

Evidence & references

  • Source: https://github.com/facebook/zstd at commit 49cf51799ea051457f57243527aa2058f999f005 (1.6.0 development), and tags v1.3.3, v1.3.4, v1.4.0, v1.4.5, v1.5.0, v1.5.1 and v1.5.6 (794ea1b0afca0f020f4e57b6732332231fb23c70) for the table history.
  • Binaries scanned read-only (no code was run): libzstd.dll and msys-zstd-1.dll 1.5.6 from Git for Windows 2.47.1; exports parsed with a small PE parser, tables located by byte search.