Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Library:Liblzma

From ReverseEngineering.wiki
Library
Developer Lasse Collin and the XZ Utils contributors (LZMA code derived from Igor Pavlov's LZMA SDK)
Platforms any (C99): liblzma.so.5 (Linux), liblzma-5.dll (MinGW/MSYS2: msys-lzma-5.dll), static libraries
Versions 5.0.0 (2010-10-23) to 5.8.4 (2026-09-09)
License 0BSD (5.6.0 and later); public domain up to 5.4.x
Website https://tukaani.org/xz/
On this wiki Facts · Code · Notes · Topics

The compression library of XZ Utils (C; 0BSD since 5.6.0, public domain up to 5.4.x): .xz, .lzma, raw LZMA/LZMA2, lzip decoding and the BCJ/Delta filters behind the lzma_* API, shipped as liblzma.so.5 or liblzma-5.dll and statically linked into many tools (Python's _lzma on Windows, for one). Its static CRC-64, price and fastpos tables and its filter features[] table survive in builds and bracket the version.

Overview

liblzma implements the Format:XZ container, the legacy Format:LZMA (.lzma) file, raw LZMA1/LZMA2 streams, decoding of Format:Lzip (5.4.0+), MicroLZMA (5.4.0+), the branch-call-jump converters for x86, PowerPC, IA-64, ARM, ARM Thumb, ARM64, SPARC and RISC-V, the Delta filter, CRC32/CRC64/SHA-256, and a threaded encoder (5.2.0+) and decoder (5.4.0+). Programs that handle .xz almost always use it: package managers, kmod, libarchive, Python's lzma module. The xz, xzdec, lzmadec and lzmainfo tools of XZ Utils are thin front ends. 7-Zip and the LZMA SDK are separate code by Igor Pavlov with different internals (see #Pitfalls).

Identification

What survives in a build, checked on liblzma 5.6.3 (MinGW-w64 and MSYS2 DLLs of Git for Windows) and a statically linked liblzma 5.4.x in CPython 3.11's Windows _lzma.pyd (MSVC):

  1. CRC-64 table lzma_crc64_table[4][256] (check/crc64_table_le.h): 00 x8 then 6F 5F A7 03 BE 4C 2E B3 5B A0 40 A8 57 36 84 F4. The big-endian header stores byte-swapped values, so the byte pattern is the same on every target. Absent in --enable-small builds (computed at run time) and in builds where CLMUL is enabled unconditionally by compiler flags.
  2. Range-coder price table lzma_rc_prices[128] (rangecoder/price_table.c): 80 67 5B 54 4E 49 45 42 3F 3D 3A 38 36 34 33 31 ... (readable as the string g[TNIEB?=:864310.-,+*)('&%$#""!). Present whenever the LZMA encoder is linked.
  3. fastpos table lzma_fastpos[8192] (lzma/fastpos_table.c): 00 01 02 03 04 04 05 05 06 06 06 06 07 07 07 07 08 x8 09 x8 .... Encoder only; absent in --enable-small builds.
  4. Filter features table features[] (common/filter_common.c): rows of (filter id u64, options size, 3 bools). On 64-bit targets it starts 01 00 00 00 00 00 00 40 70 00 00 00 00 00 00 00 00 01 01 (LZMA1 id 0x4000000000000001, sizeof(lzma_options_lzma) = 0x70) and its rows give the version range (see #Versions). The decoder and encoder tables (filter_decoder.c, filter_encoder.c) repeat the same ids next to function pointers.
  5. Magic arrays lzma_header_magic FD 37 7A 58 5A 00 and lzma_footer_magic 59 5A (common/stream_flags_common.c); in the GCC builds the footer magic directly precedes the header magic (59 5A FD 37 7A 58 5A 00), in the MSVC build it does not.
  6. Strings: liblzma has almost none. 5.4.0+ builds that link lzma_str_to_filters() carry its messages (Options must be 'name=value' pairs separated with commas, The sum of lc and lp must not exceed 4, Unsupported preset, filter names lzma2, armthumb, powerpc ...). The version string (5.6.3) is there only if lzma_version_string() is linked; the static copy in _lzma.pyd has neither.
  7. Exports of shared builds: about 110 lzma_* functions (lzma_code, lzma_end, lzma_stream_decoder, lzma_easy_encoder, lzma_alone_decoder, lzma_auto_decoder, lzma_crc64, lzma_index_*, ...). On Linux they carry the symbol versions XZ_5.0, XZ_5.2, XZ_5.4, XZ_5.6.0 and XZ_5.8.

export_symbols with subject Library:Liblzma gives a Ghidra/IDA script that finds these signature facts in a binary.

Versions

  • features[] rows (filter ids in order): 5.0.x-5.2.x: LZMA1, LZMA2 (0x21), x86, PowerPC, IA-64, ARM, ARM Thumb, SPARC, Delta (9 rows); 5.4.x adds LZMA1EXT (0x4000000000000002) after LZMA1 and ARM64 (0x0A) after ARM Thumb (11 rows); 5.6.0+ adds RISC-V (0x0B) after SPARC (12 rows). Each table ends with a row of id FF x8 (LZMA_VLI_UNKNOWN). CPython 3.11's _lzma.pyd has 11 rows, so it carries a 5.4.x liblzma.
  • Version number: lzma_version_number() returns the compile-time LZMA_VERSION = major * 10000000 + minor * 10000 + patch * 10 + stability (0 alpha, 1 beta, 2 stable); 5.6.3 is 50060032 (0x02FBDB00), and the function is B8 00 DB FB 02 C3 (mov eax, imm32; ret) in the MinGW build.
  • Symbol versions (Linux): XZ_5.2 (threaded encoder, lzma_cputhreads), XZ_5.4 (lzma_str_to_filters, lzma_lzip_decoder, lzma_microlzma_*, lzma_stream_decoder_mt, lzma_file_info_decoder), XZ_5.6.0 (lzma_mt_block_size), XZ_5.8 (lzma_bcj_x86_encode and the other lzma_bcj_* functions).
  • 5.6.0 (2024-02-24) and 5.6.1 (2024-03-09) are the releases whose tarballs carried the CVE-2024-3094 backdoor (NEWS: "enabled only in the release tarballs"); 5.6.2 removed it. A version string or number of 50060002 / 50060012 in a Linux build is worth a closer look.

API overview

All coders share one pattern: zero an lzma_stream (LZMA_STREAM_INIT), call an initialiser (lzma_easy_encoder(strm, preset, check), lzma_stream_encoder(strm, filters, check), lzma_stream_decoder(strm, memlimit, flags), lzma_alone_decoder(strm, memlimit), lzma_auto_decoder, lzma_raw_decoder(strm, filters), lzma_stream_encoder_mt(strm, &mt)), then loop lzma_code(strm, action) with next_in/avail_in/next_out/avail_out until it returns LZMA_STREAM_END (1), and finally lzma_end(strm). Actions: LZMA_RUN 0, LZMA_SYNC_FLUSH 1, LZMA_FULL_FLUSH 2, LZMA_FINISH 3, LZMA_FULL_BARRIER 4. One-shot helpers: lzma_easy_buffer_encode, lzma_stream_buffer_decode, lzma_raw_buffer_decode. Filter chains are arrays of lzma_filter {lzma_vli id; void *options;} ended by id LZMA_VLI_UNKNOWN (all FF). Decoder flags: LZMA_TELL_NO_CHECK 0x01, LZMA_TELL_UNSUPPORTED_CHECK 0x02, LZMA_TELL_ANY_CHECK 0x04, LZMA_IGNORE_CHECK 0x10, LZMA_CONCATENATED 0x08, LZMA_FAIL_FAST 0x20 (container.h lines 631-718). A build of this library with its exported names and table addresses is Build:Liblzma/5.6.3 (Windows x64, bundled with Git for Windows).

Core data structures

  • lzma_stream (base.h): next_in, avail_in, total_in, next_out, avail_out, total_out, allocator, internal, four reserved pointers, seek_pos, three reserved integers and two reserved enums: 136 bytes on 64-bit Windows and Linux.
  • lzma_options_lzma (lzma12.h): 112 bytes (0x70) on 64-bit, as recorded in features[]; lzma_options_bcj 4 bytes (start_offset), lzma_options_delta 40 bytes.

Code & repositories

  • https://github.com/tukaani-project/xz @ 299eb5c3f597f107fd6019bf156885ac8bfaa5ea · license: 0BSD AND LGPL-2.1-or-later AND GPL-2.0-or-later (declared by the contributor; GitHub's detector did not recognize COPYING)
    Official XZ Utils repository: liblzma (src/liblzma, 0BSD), the xz tools, the .xz and .lzma format specifications (doc/xz-file-format.txt, doc/lzma-file-format.txt, public domain) and the test files in tests/files. Release tags v5.0.0 onwards let you diff the tables of any version found in a binary. Per COPYING: liblzma and the tools are 0BSD, the getopt fallback LGPL-2.1+, the xzgrep/xzdiff scripts GPL-2.0+.
    Tested on: Read at this commit and at tags v5.0.0, v5.2.5, v5.4.0, v5.6.0 (filter_common.c, price_table.c, fastpos_table.c, crc64_table_*.h, lzma2_decoder.c, lzip_decoder.c, liblzma_linux.map); behaviour checked with Python 3.11 lzma and ctypes calls into a MinGW-w64 liblzma-5.dll 5.6.3; not built. · checked 2026-10-17; last activity 2026-10-16; Note: the latest release ships binaries; the wiki links only the source, never the binaries

Pitfalls

  • The CRC-64 polynomial is shared: 7-Zip/LZMA SDK (C/XzCrc64.c) and Go's hash/crc64 ECMA table compute the same table at run time, so the table in a file points to liblzma (or another static-table implementation), while the same bytes in a memory dump do not. 7-Zip's LZMA encoder also builds its price and fastpos tables at run time (LzmaEnc_InitPriceTables, LzmaEnc_FastPosInit).
  • Decoder-only builds (--disable-encoders, used for small loaders) have none of the encoder tables; use the CRC-64 table, the features table and the magic arrays.
  • liblzma's CRC32 table is the ordinary zlib one and proves nothing by itself.

Evidence & references

  • Source: https://github.com/tukaani-project/xz at commit 299eb5c3f597f107fd6019bf156885ac8bfaa5ea (5.8.4 plus fixes), and the tags v5.0.0, v5.2.5, v5.4.0 and v5.6.0 for the features[] history; 7-Zip sources (github.com/ip7z/7zip, C/XzCrc64.c and C/LzmaEnc.c) for the run-time tables.
  • Binaries scanned (read only): liblzma-5.dll and msys-lzma-5.dll 5.6.3 from Git for Windows, CPython 3.11 _lzma.pyd; lzma_version_number, lzma_version_string, lzma_crc64, lzma_check_size, the threaded encoder and the lzip decoder were called through ctypes.