Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Format:XZ

From ReverseEngineering.wiki
Format
Developer Lasse Collin and Igor Pavlov (spec 1.0.0 2009, 1.2.1 2024-04-08)
Uses Library:Liblzma
Magic FD 37 7A 58 5A 00 at 0; 59 5A at the end of each Stream
Extensions .xz, .txz (tar.xz)
Endianness little (fixed fields); sizes as base-128 variable-length integers
Website https://tukaani.org/xz/xz-file-format.txt
On this wiki Facts · Code · Notes · Topics

Container format of XZ Utils, 7-Zip and many Linux packages (.xz, .txz; specification 1.2.1, public domain): one or more Streams, each a 12-byte Stream Header (FD 37 7A 58 5A 00, flags, CRC32), Blocks of filtered data (LZMA2, optionally behind BCJ or Delta filters), an Index of block sizes and a 12-byte Stream Footer ending in "YZ". Sizes are 7-bit variable-length integers and every part is padded to a multiple of four bytes.

Overview

The .xz format is the container of XZ Utils' xz and of Library:Liblzma; 7-Zip and the LZMA SDK read and write it too. It replaced the legacy Format:LZMA (.lzma) file, which has no magic and no check: .xz adds magic bytes, a choice of integrity check, filter chains (branch converters for executables, Delta) in front of LZMA2, an Index that allows random access, and multiple Blocks so that compression and decompression can run in threads. Where it shows up: .tar.xz source and package archives, compressed kernel modules (.ko.xz), and Python's lzma module. Unverified here: also the .gnu_debugdata (MiniDebugInfo) section of ELF files, see Format:ELF (Executable and Linkable Format)#Debug information.

Identification

  • Every Stream starts with FD 37 7A 58 5A 00 ("\xFD7zXZ\0") and ends with 59 5A ("YZ"); a valid file is a multiple of 4 bytes long. 0xFD was chosen so that .xz is never taken for .lzma, whose first byte is at most 0xE0.
  • The full 12-byte Stream Header depends only on the check type: FD 37 7A 58 5A 00 00 00 FF 12 D9 41 (none), ... 00 01 69 22 DE 36 (CRC32), ... 00 04 E6 D6 B4 46 (CRC64, the default of xz and Python), ... 00 0A E1 FB 0C A1 (SHA-256).
  • A file from liblzma's single-threaded encoder continues with the 12-byte Block Header 02 00 21 01 PP 00 00 00 + CRC32, PP being the LZMA2 dictionary byte of the preset: 0x0C (preset 0, 256 KiB), 0x10 (1, 1 MiB), 0x12 (2), 0x14 (3 and 4), 0x16 (5 and 6, 8 MiB: 02 00 21 01 16 00 00 00 74 2F E5 A3), 0x18 (7), 0x1A (8), 0x1C (9, 64 MiB).
  • The empty input with default settings is exactly 32 bytes: FD 37 7A 58 5A 00 00 04 E6 D6 B4 46 00 00 00 00 1C DF 44 21 1F B6 F3 7D 01 00 00 00 00 04 59 5A (Stream Header, an Index with no records, Stream Footer).
  • Not to be confused with the 7z archive (37 7A BC AF 27 1C), which uses the same LZMA/LZMA2 and BCJ coders in its own container.

Layout

A file is one or more Streams, each optionally followed by Stream Padding (null bytes, a multiple of four). A Stream:

Part Size Content
Stream Header 12 magic FD 37 7A 58 5A 00, Stream Flags (2 bytes: 00, check id), CRC32 of the flags
Blocks each a multiple of 4 Block Header, Compressed Data, Block Padding (0-3 null bytes), Check of the uncompressed data
Index multiple of 4 Index Indicator 00, number of records, one (Unpadded Size, Uncompressed Size) pair per Block, null padding, CRC32
Stream Footer 12 CRC32 of the next 6 bytes, Backward Size (u32: Index size / 4 - 1), Stream Flags (copy of the header's), 59 5A

Block Header: byte 0 is the header size / 4 - 1 (1-255, so 8-1024 bytes; a 00 in this position is the Index Indicator, which is how a reader knows the Blocks have ended); byte 1 the Block Flags (bits 0-1 number of filters - 1, bit 6 Compressed Size present, bit 7 Uncompressed Size present, bits 2-5 zero); then the optional sizes, one Filter Flags entry per filter (filter id, size of properties, properties), null padding and a CRC32 of everything before it.

Unpadded Size in the Index is Block Header + Compressed Data + Check without the Block Padding; round it up to a multiple of four to step from one Block to the next. A reader can therefore parse a Stream backwards (Footer, Backward Size, Index, Block offsets) without decompressing anything, which is what xz --list does.

Integers and checks

  • Sizes, counts and filter ids are little-endian base-128 integers: 7 bits per byte, the high bit set on every byte but the last, at most 9 bytes (63 bits), and a 00 byte after the first is invalid. 0 is 00, 127 7F, 128 80 01, 300 AC 02, 1048576 80 80 40.
  • Fixed-size fields (CRC32s, Backward Size, BCJ start offsets) are little endian.
  • Check ids (low nibble of the second Stream Flags byte): 0 none, 1 CRC32, 4 CRC64, 10 SHA-256. The reserved ids have fixed sizes (1-3: 4 bytes, 4-6: 8, 7-9: 16, 10-12: 32, 13-15: 64) so a decoder can skip an unknown check; liblzma's lzma_check_size() returns exactly this table.
  • CRC32 is the zlib/gzip CRC-32 (polynomial 0xEDB88320). CRC64 uses the reflected ECMA-182 polynomial 0xC96C5795D7870F42 with all-ones initial value and final XOR ("CRC-64/XZ"): 123456789 gives 0x995DC9BBDF1939FA.

Filters

ID Filter Properties
0x21 LZMA2 (Format:LZMA#LZMA2) 1 byte: dictionary size = (2 + (b & 1)) << (b / 2 + 11); 0 = 4 KiB, 40 = 4 GiB - 1
0x03 Delta 1 byte: distance - 1 (1-256)
0x04 x86 BCJ 0 or 4 bytes: start offset
0x05 PowerPC (big endian) same
0x06 IA-64 same
0x07 ARM same
0x08 ARM Thumb same
0x09 SPARC same
0x0A ARM64 (spec 1.1.0; XZ Utils 5.4.0) same
0x0B RISC-V (spec 1.2.0; XZ Utils 5.6.0) same

At most four filters per Block; LZMA2 may only be the last, the others only non-last. Plain LZMA1 is not allowed in .xz. Ids from 2^62 up are reserved for implementations' internal use: liblzma's 0x4000000000000001 (LZMA1) and 0x4000000000000002 (LZMA1EXT) never appear in files, but they do appear in liblzma's own tables (see Library:Liblzma#Identification).

What encoders write

  • liblzma's single-threaded encoder (Python's lzma, xz -T1) writes one Block per Stream (more only when the application calls LZMA_FULL_FLUSH) and puts neither size in the Block Header (flags 00, or 01 with a BCJ or Delta filter); the sizes are only in the Index.
  • The multi-threaded encoder (lzma_stream_encoder_mt) cuts the input into Blocks of block_size and writes both sizes in each Block Header (flags C0, 16-byte headers such as 03 C0 DE 33 80 80 40 21 01 10 00 00 + CRC32 for 6622 compressed / 1048576 uncompressed bytes). Only such files can be decompressed in parallel by lzma_stream_decoder_mt.
  • Within a Block, liblzma's LZMA2 encoder emits uncompressed chunks for data that does not shrink (see Format:LZMA#LZMA2).

Tools

  • xz --list -vv prints Streams, Blocks, filters and sizes from the Index.
  • Code:XZ/xzinfo (Python) walks Streams, Blocks, Index and LZMA2 chunk headers in plain Python, checks every CRC32 and optionally decodes each Block and compares its Check.

Code & repositories

  • https://github.com/tukaani-project/xz @ 299eb5c3f597f107fd6019bf156885ac8bfaa5ea · license: 0BSD AND LGPL-2.1-or-later AND GPL-2.0-or-later (declared by the contributor; GitHub's detector did not recognize COPYING)
    Official XZ Utils repository: liblzma (src/liblzma, 0BSD), the xz tools, the .xz and .lzma format specifications (doc/xz-file-format.txt, doc/lzma-file-format.txt, public domain) and the test files in tests/files. Release tags v5.0.0 onwards let you diff the tables of any version found in a binary. Per COPYING: liblzma and the tools are 0BSD, the getopt fallback LGPL-2.1+, the xzgrep/xzdiff scripts GPL-2.0+.
    Tested on: Read at this commit and at tags v5.0.0, v5.2.5, v5.4.0, v5.6.0 (filter_common.c, price_table.c, fastpos_table.c, crc64_table_*.h, lzma2_decoder.c, lzip_decoder.c, liblzma_linux.map); behaviour checked with Python 3.11 lzma and ctypes calls into a MinGW-w64 liblzma-5.dll 5.6.3; not built. · checked 2026-10-17; last activity 2026-10-16; Note: the latest release ships binaries; the wiki links only the source, never the binaries

Pitfalls

  • A Block Header without Compressed Size can only be skipped through the Index or by walking the LZMA2 chunk headers (each gives its compressed size; 00 ends the stream).
  • Standalone .xz files may hold several concatenated Streams with null padding between them; when .xz is embedded in another format, decoders are expected to stop after the first Stream.

Evidence & references

  • Specification doc/xz-file-format.txt version 1.2.1 in https://github.com/tukaani-project/xz at commit 299eb5c3f597f107fd6019bf156885ac8bfaa5ea (sections 1.2, 2.1, 3.1, 4, 5.3, 6).
  • All byte sequences above were produced with Python 3.11's lzma module and with liblzma 5.6.3 (MinGW-w64 build) through ctypes for the multi-threaded encoder, then parsed with xzinfo.