Format:Zstandard
| Developer | Yann Collet, Meta (Facebook); IETF RFC 8478 (2018), replaced by RFC 8878 (2021) |
|---|---|
| Versions | format stable since zstd 0.8.0 (2016); legacy v0.1-v0.7 frames have other magics; spec document 0.4.5 (2026-05-14) |
| Magic | 28 B5 2F FD at 0 (0xFD2FB528 little-endian); skippable frames 50-5F 2A 4D 18; dictionaries 37 A4 30 EC |
| Extensions | .zst, .zstd, .tzst / .tar.zst, .pkg.tar.zst |
| Endianness | little endian; bitstreams read backwards from an end mark |
| License | specification freely redistributable (Meta notice); reference code BSD-3-Clause OR GPL-2.0-only |
| On this wiki | Facts · Code · Notes · Topics |
Yann Collet's LZ77 + entropy-coded compression format (RFC 8878; .zst files, zstd tool, libzstd): a sequence of frames, each the magic 28 B5 2F FD, a 2-14-byte frame header, blocks with a 3-byte header (raw, RLE or compressed: Huffman-coded literals plus FSE-coded sequences) and an optional 4-byte XXH64 checksum. Skippable frames (0x184D2A50-0x184D2A5F) carry metadata, and formatted dictionaries start with 0xEC30A437.
Overview
A Zstandard stream is one or more frames back to back; the decoded output is the concatenation of every frame's content. A frame is either a Zstandard frame (compressed data) or a skippable frame (user metadata, ignored by decoders). Inside a frame, data is cut into blocks of at most 128 KiB decoded. A compressed block holds a literals section (bytes copied as-is, optionally Huffman-coded) and a sequences section: (literal length, offset, match length) triples coded with FSE (tANS) tables. Decoders keep a window of earlier output (and optionally a dictionary in front of it) that matches copy from, three "repeat offsets", and the last Huffman and FSE tables, which later blocks can reuse.
The reference implementation is Library:Libzstd. Everything below was checked with a pure-Python decoder written from the specification (see #Code & repositories) that decodes the reference test frames byte-exactly.
Identification
| Bytes at offset 0 | Meaning |
|---|---|
28 B5 2F FD |
Zstandard frame (magic 0xFD2FB528, little-endian, valid since zstd 0.8.0) |
50-5F 2A 4D 18 + uint32 size |
skippable frame (0x184D2A50-0x184D2A5F); the same range as Format:LZ4#Skippable frames |
37 A4 30 EC + uint32 dictionary id |
formatted dictionary file (zstd --train, ZDICT)
|
22-27 B5 2F FD |
legacy frames of zstd 0.2-0.7 (see #Legacy formats) |
FD 2F B5 1E |
legacy zstd 0.1 frame (written big-endian) |
- The byte after the magic (Frame_Header_Descriptor) must have bit 3 (Reserved_bit) clear. A plausible embedded frame then has block headers whose type is not 3 and whose 21-bit size is at most 131072 (and at most the window size for raw and RLE blocks).
- A file whose last 4 bytes are
B1 EA 92 8F(0x8F92EAB1) carries a seek table (#Seekable format). - The content checksum is the low 32 bits of XXH64 (seed 0) of the decoded frame content, stored little-endian. Test vectors: XXH64("") = 0xEF46DB3751D8E999 (stored as
99 E9 D8 51), XXH64("abc") = 0x44BC2CF5AD770999. - Smallest valid frame (decodes to nothing):
28 B5 2F FD 20 00 15 00 00 00 00: descriptor 0x20 (single segment), content size 0, one last compressed block of 2 bytes (raw literals of size 0, 0 sequences).
Frame header
Magic(4) | Frame_Header_Descriptor(1) | [Window_Descriptor(1)] | [Dictionary_ID(0/1/2/4)] | [Frame_Content_Size(0/1/2/4/8)], then the blocks, then the optional 4-byte checksum.
Frame_Header_Descriptor bits: 7-6 Frame_Content_Size_flag, 5 Single_Segment_flag, 4 unused, 3 reserved (must be 0), 2 Content_Checksum_flag, 1-0 Dictionary_ID_flag.
- Frame_Content_Size field size by flag 0/1/2/3: 0 (or 1 if Single_Segment is set) / 2 / 4 / 8 bytes. The 2-byte form stores the size minus 256 (range 256-65791). Absent means unknown (streamed data).
- Single_Segment: no Window_Descriptor; the window is the whole content (Window_Size = Frame_Content_Size).
- Window_Descriptor:
Window_Size = (1 << (10 + Exponent)) + ((1 << (10 + Exponent)) / 8) * Mantissa, Exponent = bits 7-3, Mantissa = bits 2-0. 0x00 = 1 KiB, 0x58 = 2 MiB, 0x5D = 3.25 MiB, 0xFF = 3.75 TiB (the maximum). - Dictionary_ID size by flag 0/1/2/3: 0/1/2/4 bytes. 0 means "not specified". IDs below 32768 and from 2^31 are reserved for a future registrar.
Blocks
Each block starts with a 3-byte little-endian header: bit 0 Last_Block, bits 1-2 Block_Type (0 Raw, 1 RLE, 2 Compressed, 3 Reserved = corrupt), bits 3-23 Block_Size. For Raw and Compressed blocks Block_Size is the content length; an RLE block's content is one byte repeated Block_Size times. Block_Maximum_Size is min(Window_Size, 128 KiB) and bounds both content and decoded size.
A compressed block is Literals_Section | Sequences_Section.
Literals section
Header of 1-5 bytes, bits read from the least significant end: Literals_Block_Type (2 bits: 0 Raw, 1 RLE, 2 Compressed, 3 Treeless = reuse the previous Huffman table), then Size_Format (bits 2-3):
- Raw/RLE: Size_Format 00 or 10: 1-byte header, size = byte0 >> 3 (0-31); 01: 2 bytes, 12-bit size; 11: 3 bytes, 20-bit size.
- Compressed/Treeless: 00: 1 stream, 3-byte header, 10-bit regenerated and 10-bit compressed size; 01: 4 streams, same sizes; 10: 4 streams, 4-byte header, 14+14 bits; 11: 4 streams, 5-byte header, 18+18 bits. The compressed size includes the Huffman tree description. Example:
42 86 0C= compressed, 1 stream, 100 bytes regenerated from 50. - Four streams: a 6-byte jump table of three uint16 stream sizes (the fourth is the rest); each of the first three streams decodes (Regenerated_Size + 3) / 4 bytes.
Sequences section
Number_of_Sequences: byte0 < 128: byte0; byte0 < 255: ((byte0 - 128) << 8) + byte1; byte0 = 255: byte1 + (byte2 << 8) + 0x7F00. Zero sequences ends the section (libzstd treats further bytes as corruption). Otherwise one mode byte follows: bits 7-6 literal lengths, 5-4 offsets, 3-2 match lengths, 1-0 reserved (0); each mode is 0 Predefined (default distribution), 1 RLE (one symbol byte follows), 2 FSE_Compressed (a table description follows; max accuracy log 9 for lengths, 8 for offsets), 3 Repeat (previous table, or the dictionary's). Tables follow in the order literal lengths, offsets, match lengths, then the bitstream.
The bitstream is read backwards: its last byte holds an end-mark 1-bit above 0-7 zero padding bits. Initial states are read in the order LL, OF, ML; each sequence reads offset extra bits, then match-length extra bits, then literal-length extra bits; states are then updated in the order LL, ML, OF (not after the last sequence). The stream must be consumed exactly.
Codes: literal length codes 0-35 (0-15 are the length itself; 16-35 have baselines 16, 18, 20, 22, 24, 28, 32, 40, 48, 64, 128 ... 65536 with 1-16 extra bits), match length codes 0-52 (code + 3 for 0-31; then baselines 35 ... 65539), offset code N: Offset_Value = (1 << N) + N extra bits. Default distributions use accuracy log 6 (LL, ML) and 5 (OF, codes up to 28).
Sequence execution and repeat offsets
For each sequence: copy literal_length literals, then copy match_length bytes from offset bytes back (overlap allowed). Offset_Value > 3 means offset = Offset_Value - 3. Values 1-3 select a repeat offset: Repeated_Offset1..3, initially 1, 4, 8 (or the dictionary's). When literal_length is 0 the selection shifts by one: 1 means Repeated_Offset2, 2 means Repeated_Offset3, 3 means Repeated_Offset1 - 1 (0 is corruption). Using a new offset (or the "minus one" case) pushes it to the front; using repeat 2 or 3 moves that one to the front. Literals left after the last sequence are appended. Raw and RLE blocks do not touch the repeat offsets or tables.
Entropy coding
- FSE table description (read forwards, little-endian bits): Accuracy_Log = low 4 bits + 5, then one variable-width value per symbol: probability = value - 1, where -1 means "less than 1" (one cell at the end of the table, full state reset); a zero probability is followed by 2-bit repeat counts of further zeros (3 = read another). Cells are spread with
position = (position + (size >> 1) + (size >> 3) + 3) & (size - 1), skipping the cells taken by -1 symbols. - Huffman tree description: header byte < 128: that many bytes of FSE-compressed weights (two interleaved states, accuracy log at most 6); >= 128: (header - 127) weights stored as 4-bit nibbles, high nibble first. The last symbol's weight is implied (completes the sum of 2^(weight-1) to a power of two). Number_of_Bits = Max_Number_of_Bits + 1 - Weight, maximum code length 11 bits. Codes are canonical: sorted by weight, then symbol value, lowest weight gets the lowest codes.
- Huffman streams are read backwards from an end-mark bit like the sequence bitstream.
Skippable frames
Magic 0x184D2A5? (4) | Frame_Size uint32 LE | User_Data. All 16 magic values are valid and equivalent. Decoders skip them; they can carry any metadata (the seek table below is one), including tracking data that survives concatenation.
Dictionaries
A raw-content dictionary is any buffer of at least 8 bytes, used as history in front of the frame. A formatted dictionary is 37 A4 30 EC | Dictionary_ID uint32 | Huffman table | FSE table for offsets | FSE table for match lengths | FSE table for literal lengths | 3 x uint32 repeat offsets | content. The entropy tables seed Treeless literals and Repeat modes in the first block. A frame names the dictionary it needs only by the 32-bit ID; matches may reach into dictionary content while the decoded size is at most Window_Size.
Seekable format
contrib/seekable_format of the reference repository: ordinary frames followed by a final skippable frame (magic 0x184D2A5E) holding Number_Of_Frames entries of (compressed size uint32, decompressed size uint32[, XXH64 low 32 bits]) and a 9-byte footer: Number_Of_Frames uint32, Seek_Table_Descriptor (bit 7 = per-entry checksum, bits 6-2 reserved = 0), magic 0x8F92EAB1 (B1 EA 92 8F) as the last 4 bytes of the file.
Legacy formats
Pre-0.8 releases used other frame layouts and magics: v0.1 0xFD2FB51E (stored big-endian, file bytes FD 2F B5 1E), v0.2-v0.7 0xFD2FB522-0xFD2FB527 (little-endian, 22 B5 2F FD ... 27 B5 2F FD; there is no 0xFD2FB51F-0xFD2FB521). Dictionaries have carried 0xEC30A437 since 0.7.0. The reference library decodes them only when built with ZSTD_LEGACY_SUPPORT (lib/legacy/zstd_v0*.c).
Edge cases
From doc/decompressor_errata.md and doc/decompressor_permissive.md of the reference repository (hex frames given there, all checked with the Python decoder):
- Valid but once rejected by libzstd: a compressed block of exactly 128 KiB (until 1.5.2; tests/golden-decompression/block-128k.zst), a compressed block with 0 literals and 0 sequences (until 1.5.2; the smallest frame above), 0 sequences written in the 2-byte form
80 00(until 1.5.5; zeroSeq_2B.zst), a first RLE block of 131072 bytes followed by more blocks (zstd CLI until 1.4.3), a last FSE table less than 4 bytes from the block end (until 1.3.4). - Invalid but once accepted: offset 0 from "Repeated_Offset1 - 1" (treated as 1 until 1.5.5; golden-decompression-errors/off0.bin.zst), truncated initial states of FSE-compressed Huffman weights (read as zero until 1.5.6; truncated_huff_state.zst).
- The spec's own Huffman stream example (section "Huffman-coded Streams") is inconsistent: with its code table (E = 0000, F = 0001) the bitstream
00010000 00001101decodes to A, B, F, E, not "ABEF"; its encoding row swaps the E and F labels. Decoders following the code table are right.
Where it appears
Unverified on this wiki unless linked; check the magic at the payload start: ELF compressed sections with ch_type 2 (ELFCOMPRESS_ZSTD, see Format:ELF (Executable and Linkable Format)#Debug information), Arch Linux packages (.pkg.tar.zst), Debian/Ubuntu .deb members (data.tar.zst), RPM payloads, Linux kernel and initramfs images, Btrfs/SquashFS/EROFS compressed data, ZIP method 93, HTTP Content-Encoding: zstd, game and engine asset archives. Inside archives a frame may also be stored without its magic ("magicless" format, ZSTD_f_zstd1_magicless in libzstd), in which case the data starts directly with the frame header descriptor.
Code & repositories
- Code:Zstandard/Zstandard decoder (Python): dependency-free decoder, frame lister (--list) and embedded-frame scanner (--scan) written from the specification, with its test script; MIT-0.
- {{Repo|url=https://github.com/facebook/zstd%7Ccommit=49cf51799ea051457f57243527aa2058f999f005%7Clicense=BSD-3-Clause OR GPL-2.0-only (declared by the contributor; GitHub's detector did not recognize COPYING)|description=Reference implementation (libzstd, zstd CLI; BSD-3-Clause OR GPL-2.0-only per LICENSE and COPYING) and the format specification doc/zstd_compression_format.md (version 0.4.5), the educational decoder (doc/educational_decoder), decoder errata/permissiveness notes with example frames, golden test frames (tests/golden-*), the seekable format (contrib/seekable_format) and the legacy v0.1-v0.7 decoders (lib/legacy). Release tags let you diff tables against a binary.|tested_on=Read at this commit and at tags v1.3.3, v1.3.4, v1.4.0, v1.4.5, v1.5.0, v1.5.1, v1.5.6; its golden frames, errata frames and Appendix A tables were checked with the Python decoder on this wiki; not built.|checked=checked 2026-10-17; last activity 2026-10-06; Note: the latest release ships binaries; the wiki links only the source, never the binaries}}
- {{Repo|url=https://github.com/klauspost/compress%7Ccommit=920bb4bcc0188e341ce7655cd2bc96b164cd6c56%7Clicense=BSD-3-Clause AND Apache-2.0 AND MIT (declared by the contributor; GitHub's detector did not recognize LICENSE)|description=Pure-Go Zstandard encoder/decoder (zstd/ package, BSD-3-Clause; the LICENSE file adds Apache-2.0 for gzhttp/ and MIT for two s2 helpers) and a large public Zstandard test corpus in zstd/testdata: good.zip and decoder.zip (frame + expected output pairs), bad.zip (invalid frames), dict-tests-small.zip (frames with four trained dictionaries), headers.zip, seqs.zip. An independent implementation, useful to cross-check behaviour without libzstd.|tested_on=Only zstd/testdata was read (sparse checkout); good.zip 11/11, decoder.zip 94/94, bad.zip 44/44 rejected and dict-tests-small.zip 40/40 checked with the Python decoder on this wiki. The Go code was not built.|checked=checked 2026-10-17; last activity 2026-10-16; Note: the latest release ships binaries; the wiki links only the source, never the binaries}}
Pitfalls
- Windows can exceed the 8 MiB that decoders are recommended to support; libzstd accepts up to 2^27 (128 MiB, ZSTD_WINDOWLOG_LIMIT_DEFAULT) by default and refuses larger windows (
zstd --long=31output) unless the caller raises ZSTD_d_windowLogMax. - Single_Segment frames have Window_Size = Frame_Content_Size, which can be below the 1 KiB minimum of the window descriptor (0 for an empty frame).
- Frame_Content_Size in a 2-byte field is stored minus 256.
- A frame without content size, or several concatenated frames, cannot be sized from the first header; ZSTD_decompressBound or a full decode is needed.
- The checksum covers only the decoded content of its own frame, not skippable frames or the header.
Evidence & references
- Specification doc/zstd_compression_format.md (version 0.4.5) and lib/zstd.h of https://github.com/facebook/zstd at commit 49cf51799ea051457f57243527aa2058f999f005; RFC 8878.
- Test frames: tests/golden-decompression*, doc/decompressor_errata.md, doc/decompressor_permissive.md of the same commit; zstd/testdata of https://github.com/klauspost/compress at 920bb4bcc0188e341ce7655cd2bc96b164cd6c56 (good.zip, decoder.zip, bad.zip, dict-tests-small.zip).
- Checked by decoding all of these with the Python decoder listed under Code & repositories (frame contents compared byte for byte or by their XXH64 checksum; invalid frames rejected) and by comparing its predefined FSE tables with Appendix A of the specification.