Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Library:Liblz4

From ReverseEngineering.wiki
Library
Developer Yann Collet and contributors
Platforms any (portable C; x86, ARM, MIPS, PowerPC, wasm)
Versions r1-r131 (2011-2015), 1.7.x (2016), 1.8.x (2017-2018), 1.9.0-1.9.4 (2019-2022), 1.10.0 (2024)
License BSD-2-Clause (lib/); GPL-2.0-or-later (programs/, tests/)
Website https://lz4.org
On this wiki Facts · Code · Notes · Topics

The reference LZ4 compression library (C, BSD-2-Clause, by Yann Collet): fast block compression (LZ4_*), high-compression levels (LZ4_*HC) and the frame API (LZ4F_*), shipped as liblz4.so.1 or liblz4.dll and statically compiled into many programs. Its version function, its frame error-name strings, its decoder offset tables and its 5-byte hash constant 0xCF1BBCDCBB identify and date a build.

Overview

liblz4 is the lib/ directory of https://github.com/lz4/lz4: lz4.c (block compression and decompression, LZ4_* API), lz4hc.c (the high-compression levels 2-12, LZ4_HC API, same output format), lz4frame.c (the Format:LZ4 frame API, LZ4F_), lz4file.c (FILE-like wrappers, 1.10.0+) and a private copy of xxhash.c for the frame checksums. It is BSD-2-Clause; the lz4 command-line program (programs/) is GPL-2.0-or-later. Programs ship it as liblz4.so.1, liblz4.dll, msys-lz4-1.dll or cyglz4-1.dll, or compile lz4.c straight into their own code, often without lz4frame.c.

Identification

  • Exports: LZ4_compress_default, LZ4_decompress_safe, LZ4_compressBound, LZ4_compress_HC, LZ4F_compressFrame, LZ4F_decompress, LZ4F_getErrorName and about 100 more in 1.10.0 (48 in 1.7.4, 89 in 1.9.1). Deprecated names (LZ4_compress, LZ4_uncompress, LZ4_uncompress_unknownOutputSize) are still exported for old callers.
  • xxHash names: builds made with the project's own Makefile, CMake or Meson files define XXH_NAMESPACE=LZ4_, so the private hash is LZ4_XXH32, LZ4_XXH64 and so on (exported from cyglz4-1.dll 1.7.5). Other builds (msys-lz4-1.dll 1.10.0, Unity's liblz4.dll 1.9.1) export no XXH name at all. A plain XXH32 export next to LZ4_ names means xxHash was linked separately.
  • Version: LZ4_versionNumber() is mov eax, MAJOR*10000+MINOR*100+RELEASE; ret (B8 F8 2A 00 00 C3 = 11000 = 1.10.0); LZ4_versionString() (1.7.5+) returns "1.10.0". See #Versions.
  • Frame error names: LZ4F_getErrorName returns the enum names as text: "OK_NoError", "ERROR_GENERIC", "ERROR_maxBlockSize_invalid" ... "ERROR_maxCode", plus "Unspecified error code" for non-errors. The list changed between versions and dates a build.
  • Decoder tables: the decoder's offset-correction tables for short overlapping copies are 32-byte constants (inc32table and dec64table in 1.8.3 and later, dec32table and dec64table with other values before). They are in .rdata when the compiler keeps them as data, which it did in 5 of the 7 builds checked (not in the optimised 1.7.1 and 1.7.5 builds, where they were local arrays). The 1.8.3+ tables matched nothing else in 452 DLL/EXE files of Git for Windows; the old dec32table {0,1,2,1,4,4,4,4} alone also occurs in libzstd's decoder, so match it together with the old dec64table.
  • Hash constant: the fast compressor hashes 5 bytes with 889523592379 (0xCF1BBCDCBB) on 64-bit little-endian targets. Builds either fold the << 24 into the constant, movabs r64, 0xCF1BBCDCBB000000 (bytes 48 B8+r 00 00 00 BB DC BC 1B CF; GCC msys-lz4-1.dll 1.10.0: 60 times, Unity liblz4.dll 1.9.1: 37), or use it unshifted, movabs r64, 0xCF1BBCDCBB (48 B8+r BB DC BC 1B CF 00 00 00; MSVC-linked LSLibNative.dll 1.7.1); unoptimised and 32-bit builds keep it as 8-byte data. The 5 bytes BB DC BC 1B CF appear dozens of times in every build checked (Fact:Liblz4/identification/Hash constant 0xCF1BBCDCBB). Zstandard's compressor uses the same constant (prime5bytes in libzstd: about 160 occurrences in libzstd 1.5.6), so it alone means "LZ4 or zstd". The 4-byte hash and LZ4HC use 2654435761 (0x9E3779B1), which many other hash functions share.

Versions

Version LZ4_versionNumber Error names (first change) Decoder tables
r131 (= 1.7.1, 2015) 10701 ERROR_blockChecksum_unsupported local size_t dec32table {4,1,2,1,4,4,4,4}, size_t dec64table {0,0,0,-1,0,1,2,3} (8-byte entries on 64-bit)
1.7.3-1.7.5 10703-10705 ERROR_blockChecksum_unsupported local unsigned dec32table {0,1,2,1,4,4,4,4}, int dec64table {0,0,0,-1,0,1,2,3}
1.8.0 (1.8.1-1.8.2 not checked) 10800 ERROR_blockChecksum_invalid, ERROR_frameDecoding_alreadyStarted added as 1.7.x
1.8.3-1.9.2 10803-10902 list moved from lz4frame_static.h to lz4frame.h inc32table {0,1,2,1,0,4,4,4}, dec64table {0,0,0,-1,-4,1,2,3}; file-scope static from 1.9.0
1.9.4 10904 ERROR_compressionState_uninitialized, ERROR_parameter_null added as 1.9.x
1.10.0 11000 ERROR_parameter_invalid replaces ERROR_contentChecksumFlag_invalid; ERROR_io_write, ERROR_io_read added as 1.9.x
development after 1.10.0 (NEWS lists v1.10.1, not tagged yet) 11001 ERROR_dstCapacity_tooSmall replaces ERROR_dstMaxSize_tooSmall as 1.9.x

Builds read for this table: msys-lz4-1.dll of Git for Windows (1.10.0, GCC), liblz4.dll and liblz4.so of a Unity Editor package (folder named lz4-1.9.1; the DLL is 1.9.1, the .so reports 1.9.2), cyglz4-1.dll of cwRsync (1.7.5), a Vortex mod-manager plugin's liblz4.dll (1.7.4, unoptimised), LSLibNative.dll of the open-source LSLib tools (1.7.1, static) and QuickBMS (several static copies; 1.9.4 among the version strings).

API structures

LZ4F_frameInfo_t (32 bytes) and LZ4F_preferences_t (56 bytes) have the same layout on Win64, LP64 and ILP32 because every field is a 32-bit enum or unsigned except one 64-bit content size at offset 16. Functions of the LZ4F API return size_t; values above (size_t)-LZ4F_ERROR_maxCode are negated LZ4F_errorCodes, so LZ4F_isError compiles to cmp rax, -24; ja in 1.10.0 (LZ4F_ERROR_maxCode = 24; 48 83 F8 E8 occurs 7 times in msys-lz4-1.dll) and callers test results the same way. The contexts LZ4F_cctx and LZ4F_dctx are opaque and change between versions; LZ4F_VERSION (100) is passed to the create functions.

Where it appears

Verified on this wiki by version function and strings: Git for Windows (msys-lz4-1.dll), cwRsync (cyglz4-1.dll), Unity Editor packages, LSLib (Larian game tools), QuickBMS. Unverified: the Linux kernel (lib/lz4, a modified copy without the frame API), ZFS, Btrfs-progs, systemd-journald, Chromium and Firefox, Kafka clients, many game engines and asset tools.

Code & repositories

  • Code:LZ4/LZ4 codec and frame scanner (Python): dependency-free LZ4 block and frame codec (decode, encode, list, scan binaries for embedded frames) with XXH32, tested on frames from the C lz4 tool; MIT-0.
  • {{Repo|url=https://github.com/lz4/lz4%7Ccommit=aa787570e681c695fc087b4a44cd2d5b38c2289a%7Clicense=BSD-2-Clause AND GPL-2.0-or-later (declared by the contributor; GitHub's detector did not recognize LICENSE)|description=Reference implementation: liblz4 (lib/, BSD-2-Clause: lz4.c, lz4hc.c, lz4frame.c, xxhash.c) and the lz4 CLI (programs/, GPL-2.0-or-later), with the block and frame specifications (doc/lz4_Block_format.md, doc/lz4_Frame_format.md 1.6.4), two golden frames (tests/goldenSamples: a frame with a dictionary ID, a skippable frame) and release tags from r116 to v1.10.0 to diff a binary against.|tested_on=Read at this commit and at tags r131, v1.7.3, v1.7.5, v1.8.0, v1.8.3, v1.9.0, v1.9.2, v1.9.4, v1.10.0; golden frames decoded with the Python codec on this wiki; constants and tables compared with 7 binary builds (1.7.1-1.10.0). Not built.|checked=checked 2026-10-17; last activity 2026-10-17; Note: the latest release ships binaries; the wiki links only the source, never the binaries}}

Pitfalls

  • A statically linked copy may contain only lz4.c (block API): no LZ4F error strings and no frame magic. Search for the tables, the hash constant and LZ4_versionNumber's constant instead.
  • The decoder tables are not unique to the original source: forks and ports (the Linux kernel's lz4 copy, other language bindings) may carry the same constants.
  • Folder and file names do not give the version (see the Unity .so above); read LZ4_versionNumber or LZ4_versionString.
  • LZ4_decompress_fast trusts its input (deprecated since 1.9.0); seeing it imported is a hint that the program decodes trusted data or has a memory-safety problem with untrusted data.

Evidence & references

  • Source of https://github.com/lz4/lz4 at aa787570e681c695fc087b4a44cd2d5b38c2289a (lib/lz4.c, lib/lz4.h, lib/lz4frame.c, lib/lz4frame.h, lib/xxhash.c, lib/Makefile line 49) and at tags r131, v1.7.3, v1.7.5, v1.8.0, v1.8.3, v1.9.0, v1.9.2, v1.9.4, v1.10.0 for the version table.
  • The builds listed under Versions were read with a small Python PE reader (exports, .rdata contents, the bytes of LZ4_versionNumber); nothing was executed.