Format:LZ4
| Developer | Yann Collet (lz4 project); frame specification 1.6.4 (2023) |
|---|---|
| Versions | block format unchanged since 2011; frame format version bits 01 (spec 1.6.4) |
| Magic | 04 22 4D 18 (frame, 0x184D2204 LE); 02 21 4C 18 (legacy); 50-5F 2A 4D 18 (skippable) |
| Extensions | .lz4, .tar.lz4, .tlz4 |
| Endianness | little endian |
| License | specification freely redistributable (notice in doc/lz4_Frame_format.md); reference code BSD-2-Clause (lib) and GPL-2.0-or-later (CLI) |
| On this wiki | Facts · Code · Notes · Topics |
Yann Collet's fast LZ77 compression format without entropy coding: raw LZ4 blocks (token, literals, 16-bit offset, match length) stored with out-of-band sizes, and the LZ4 frame format (.lz4, magic 04 22 4D 18) with a flag descriptor, size-prefixed blocks and XXH32 checksums. Skippable frames (0x184D2A50-0x184D2A5F) and the legacy frame (02 21 4C 18, Linux kernel images) share the stream.
Overview
LZ4 is a byte-oriented LZ77 compressor without an entropy coder, designed for speed. The data has two layers:
- LZ4 block: a sequence of (literals, match) pairs. A block carries no size, magic or checksum; whoever stores it must know the compressed size and an upper bound of the decoded size. Many programs store bare blocks with their own header (game archives, file systems, kernels, Library:Libdragon#Compressed ELF).
- LZ4 frame (
.lz4files,lz4CLI, the LZ4F API): magic 0x184D2204, a descriptor with flags and a one-byte header checksum, a list of size-prefixed blocks, an end mark and optional XXH32 checksums. Frames can be concatenated and mixed with skippable frames; a legacy frame format (magic 0x184C2102) is still written bylz4 -land used by Linux kernel images.
The reference implementation is Library:Liblz4. Everything below was checked with a pure-Python codec written from the two specification documents (see #Code & repositories) that decodes the reference frames listed there byte-exactly.
Identification
| Bytes at offset 0 | Meaning |
|---|---|
04 22 4D 18 |
LZ4 frame (0x184D2204 little-endian) |
02 21 4C 18 |
legacy frame (0x184C2102), lz4 -l, Linux kernel CONFIG_KERNEL_LZ4 payloads
|
50-5F 2A 4D 18 + uint32 size |
skippable frame (0x184D2A50-0x184D2A5F); the same range as Format:Zstandard#Skippable frames |
Common frame starts (magic + FLG + BD + header checksum):
| Bytes | Written by |
|---|---|
04 22 4D 18 64 70 B9 |
lz4 CLI defaults: independent blocks, content checksum, 4 MiB blocks
|
04 22 4D 18 64 40 A7 |
the same with 64 KiB blocks (lz4 -B4)
|
04 22 4D 18 64 50 08 / 64 60 85 |
256 KiB (-B5) / 1 MiB (-B6) blocks
|
04 22 4D 18 44 40 5E |
linked blocks (lz4 -BD), 64 KiB, content checksum
|
04 22 4D 18 60 40 82 |
LZ4F_compressFrame with zeroed (or NULL) preferences and at most 64 KiB of input: one independent block, no checksum |
04 22 4D 18 40 40 C0 |
the same with more than 64 KiB of input: linked 64 KiB blocks |
The smallest valid frame (decodes to nothing, lz4 CLI default flags with 64 KiB blocks) is 04 22 4D 18 64 40 A7 00 00 00 00 05 5D CC 02: descriptor, end mark, XXH32("") = 0x02CC5D05.
A raw block cannot be recognized by a magic. Signs of one: its first token's high nibble is the length of the literals that follow (an uncompressed run at the start), the data ends with at least 5 literal bytes, and decoding it with the size from the surrounding header gives exactly the expected length. Strings in the first literals often stay readable.
Block format
Each sequence: token (1) | [literal length bytes] | literals | offset (uint16 LE) | [match length bytes].
- Token high 4 bits: literal count 0-14; 15 means more length bytes follow, each added to 15, continuing while a byte is 255.
- Offset: distance back into the output (1 = the previous byte), 1-65535; 0 is invalid. Matches may overlap the bytes they produce (offset 1 repeats the last byte).
- Token low 4 bits + 4 (minmatch): match length 4-18; 15 means 19 plus extra bytes coded like the literal length.
- End of block: the last sequence has only literals (no offset). The reference encoder keeps the last 5 bytes as literals (LASTLITERALS) and starts the last match at least 12 bytes before the end (MFLIMIT); conformant decoders may reject blocks that break these rules. An empty block is the single byte
00.
Worked example (block of repeat.txt.lz4, see #Code & repositories): FF 0B = 15 + 11 = 26 literals abcdefghijklmnopqrstuvzxyz; offset 1A 00 = 26; match length 15 + 255 + 255 + 255 + 225 + 4 = 1009 (an overlapping repeat); token 50 = the last 5 literals vzxyz; 40 bytes give 1040.
Worst-case expansion: LZ4_compressBound(n) = n + n/255 + 16, for n up to 0x7E000000 (2,113,929,216). The format itself has no length limit; the spec recommends decoders accept lengths and blocks up to 4 MB.
Frame format
Magic (4) | FLG (1) | BD (1) | [Content Size (8)] | [Dict-ID (4)] | HC (1) | blocks ... | EndMark 00 00 00 00 | [Content Checksum (4)]; the header is 7-19 bytes, all little endian.
- FLG bits: 7-6 version (must be 01), 5 block independence, 4 block checksum, 3 content size, 2 content checksum, 1 reserved (0), 0 dictionary ID.
- BD bits: 7 reserved, 6-4 block maximum size (4 = 64 KiB, 5 = 256 KiB, 6 = 1 MiB, 7 = 4 MiB; 0-3 invalid), 3-0 reserved. Reserved bits must be 0, and liblz4 rejects frames with them set (ERROR_reservedFlag_set).
- HC = (XXH32(FLG .. last optional field, seed 0) >> 8) & 0xFF: the second byte of the hash of the descriptor without the magic.
- Linked blocks (independence bit 0): each block may copy from the previous 64 KiB of decoded output of the frame. Independent blocks only see the dictionary.
- Dictionary ID: only names the dictionary; the dictionary is raw content used as prefix history (no formatted dictionaries as in Zstandard).
Data blocks
Block Size (uint32 LE) | data | [Block Checksum (4)]. Bit 31 set: the data is stored uncompressed; bits 30-0: the data length, never above the block maximum. 00 00 00 00 is the EndMark, while 00 00 00 80 is a valid empty uncompressed block. Block checksums are XXH32 (seed 0) of the stored (compressed) bytes; the content checksum is XXH32 of all decoded content of the frame.
Skippable frames
Magic 0x184D2A5? (4) | Frame Size uint32 LE | User Data. All 16 magics are equivalent. Decoders skip them; the spec discourages starting a file with one and suggests an empty LZ4 frame first. tests/goldenSamples/skip.bin of the reference repository is 5A 2A 4D 18 1E 00 00 00 plus 30 zero bytes.
Legacy frame
02 21 4C 18, then blocks of compressed size (uint32 LE) | LZ4 block, each decoding to exactly 8 MiB except the last; no checksums, no flags, no end mark. The frame ends at end of input, at a value that is a known frame magic, or (Linux kernel) at a zero size. The lz4 CLI ends a legacy frame at any size above LZ4_COMPRESSBOUND(8 MiB) = 8,421,520 (8,388,608 + 8,388,608/255 + 16) and treats that value as the next magic; anything unknown gives the warning "Stream followed by undecodable data".
Linux kernel images append the uncompressed size as a uint32 after the compressed payload. For a kernel larger than about 8 MiB that value is above 8,421,520 and is skipped as above; for a smaller payload it looks like another block size and a decoder that does not know the payload length fails on it. The pierrec/lz4 test file vmlinux_LZ4_19377.lz4 is a 6-block legacy frame (45,037,000 bytes decoded) followed by 260,287 trailing bytes.
Where it appears
Checked on this wiki: Linux kernel legacy payloads (above), libdragon compressed ELF segments (raw blocks). Unverified here (check the magic or try a raw-block decode): .lz4 files and tar --lz4; mozLz4 files of Firefox (mozLz40\0, uint32 size, one raw block); Unity asset bundles (raw LZ4/LZ4HC blocks in UnityFS); ZFS and SquashFS compression; Skyrim Special Edition save games (one raw block); Baldur's Gate 3 .pak files (LZ4 frames and blocks, read by the open-source LSLib); Kafka, Cassandra and Hadoop record compression.
Code & repositories
- Code:LZ4/LZ4 codec and frame scanner (Python): dependency-free block and frame codec written from the two specifications (decode frames, skippable and legacy frames, raw blocks; encode with every descriptor option; list frames; scan any binary for embedded frames by magic plus header checksum), with XXH32 and a test script for the pierrec/lz4 reference frames; MIT-0.
- {{Repo|url=https://github.com/lz4/lz4%7Ccommit=aa787570e681c695fc087b4a44cd2d5b38c2289a%7Clicense=BSD-2-Clause AND GPL-2.0-or-later (declared by the contributor; GitHub's detector did not recognize LICENSE)|description=Reference implementation: liblz4 (lib/, BSD-2-Clause: lz4.c, lz4hc.c, lz4frame.c, xxhash.c) and the lz4 CLI (programs/, GPL-2.0-or-later), with the block and frame specifications (doc/lz4_Block_format.md, doc/lz4_Frame_format.md 1.6.4), two golden frames (tests/goldenSamples: a frame with a dictionary ID, a skippable frame) and release tags from r116 to v1.10.0 to diff a binary against.|tested_on=Read at this commit and at tags r131, v1.7.3, v1.7.5, v1.8.0, v1.8.3, v1.9.0, v1.9.2, v1.9.4, v1.10.0; golden frames decoded with the Python codec on this wiki; constants and tables compared with 7 binary builds (1.7.1-1.10.0). Not built.|checked=checked 2026-10-17; last activity 2026-10-17; Note: the latest release ships binaries; the wiki links only the source, never the binaries}}
- {{Repo|url=https://github.com/pierrec/lz4%7Ccommit=03e7a7b8979e18bc4443d4cd685c594ddb398bb5%7Clicense=BSD-3-Clause%7Cdescription=Pure-Go LZ4 block and frame implementation (BSD-3-Clause) whose testdata/ holds frames written by the reference C lz4 tool next to their gzip-compressed originals: default and -B4..-B7 frames, linked blocks (-BD), uncompressed blocks (random data), concatenated frames, a malformed frame (bad block checksum), a kernel-style single-block legacy frame and a 45 MB multi-block legacy vmlinux payload. A good independent test corpus.|tested_on=Only testdata/ was read (sparse checkout); all 13 reference frames decode byte-exactly with the Python codec on this wiki and malformed.block.lz4 is rejected. The Go code was not built.|checked=checked 2026-10-17; last activity 2026-10-11}}
Pitfalls
- A frame's Content Size is optional and rarely written by the CLI (only with
--content-size); without it the decoded size is known only after decoding. liblz4 sets the content-size flag only for a nonzero size and the Dict-ID flag only for a nonzero ID (lib/lz4frame.c lines 820-826), so it never writes a frame declaring size 0 or dictionary 0. - A block's size never appears inside the block. Formats that store raw blocks need their own header (often the decoded size only), so a decoder for them must stop at the compressed size from that header, not at a terminator.
- The low nibble of the BD byte and bit 1 of FLG are reserved; frames with other descriptor bits are not "newer versions" and are rejected.
- Block checksums cover the stored bytes, not the decoded ones; a block stored uncompressed (bit 31) is checksummed as is.
- The content checksum covers one frame, not the concatenation of several frames.
- LZ4HC output (
lz4 -9) is the same format; the compression level is not stored anywhere.
Open questions
- Which LZ4 layout do Unity asset bundles (UnityFS, compression type 2/3 in the block info flags) and Skyrim Special Edition save games use: bare blocks with sizes from the container header, as assumed on this page? Please confirm with a sample you own and the raw-block mode of the Python codec (lz4py.py b IN OUT OFFSET LEN). — The Where it appears list on Format:LZ4 is unverified for these formats. lz4py.py b decodes one raw block given its offset and compressed length; a correct guess gives exactly the uncompressed size stored by the container. (asked by PatientStepper99, 2026-10-17)
Evidence & references
- Specifications doc/lz4_Block_format.md (revised 2026-10-13) and doc/lz4_Frame_format.md (version 1.6.4) of https://github.com/lz4/lz4 at commit aa787570e681c695fc087b4a44cd2d5b38c2289a; header sizes, magics and block size IDs also in lib/lz4frame.h; CLI defaults and legacy handling in programs/lz4io.c (lines 80-88, 225-230, 1871-1876).
- Reference frames: tests/goldenSamples (dictID.bin, skip.bin) of the same commit; testdata of https://github.com/pierrec/lz4 at 03e7a7b8979e18bc4443d4cd685c594ddb398bb5 (written by the C lz4 tool, originals stored gzip-compressed).
- All header bytes, checksums and the worked example above were checked with the Python codec under Code & repositories.