Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Technique:Identifying checksums in file formats

From ReverseEngineering.wiki
Technique
On this wiki Facts · Code · Notes · Topics

How to find out what a checksum or hash field in a save file, firmware image or network packet covers and which algorithm computes it: locate the field, vary the input, try the common CRC, Adler, Fletcher and Internet checksums, and run a CRC parameter search when none fits.

Overview

Save files, firmware images, game archives and packets usually carry a small integrity field. Editing the data without fixing the field makes the loader reject the file, so reversers need two answers: what bytes does the field cover, and which algorithm is it. The algorithm is rarely custom: most are a CRC (Technique:CRC variants and parameters), Adler-32, Fletcher, the Internet checksum, a simple sum, or a non-cryptographic or cryptographic hash (Technique:Non-cryptographic hashes in file formats, Technique:Identifying cryptographic constants).

Steps

  1. Find the field. Take two saves that differ by one tiny change (one item count). Bytes that change besides the edited value are the checksum. A field of width 1, 2, 4 or 8 bytes that changes in a pseudo-random way when one bit of data flips is a CRC or hash; a field that changes by the same amount as the edit is a plain sum. Fields are usually in the header, or the last 2 to 8 bytes of the file or block.
  2. Find the covered range. Candidates in order: whole file with the field zeroed or skipped, everything after the header, each section separately, payload without the trailing field. Flip a byte in various places and see which flips make the loader complain (or the checksum change in a re-saving program): the covered range is where flips matter. Footers often cover everything before them; headers often cover the bytes after them.
  3. Try the common algorithms on each candidate range with both byte orders for the stored value (see below). A 32-bit stored value matches a CRC-32/ISO-HDLC check on one range in most formats; test that first.
  4. If the field is 8, 16 or 32 bits and nothing fits, use a CRC parameter search (below).
  5. Check the program: search the binary for the CRC-32 table (0xEDB88320 reflected or 0x04C11DB7 non-reflected, table starts 00 00 00 00 96 30 07 77 or 00 00 00 00 B7 1D C1 04, see Technique:Identifying cryptographic constants), for Adler-32's modulus 65521 (0xFFF1), for FNV, murmur or xxHash primes, and for the SSE4.2 crc32 instruction (CRC-32C). A cross-reference from the table to the code tells the covered range directly.

Quick tests with known algorithms

  • Compute the CRC of the ASCII string 123456789 with your implementation and compare with the check values in Technique:CRC variants and parameters; this proves the implementation before it is blamed on the file.
  • Zlib-family: CRC-32 is zlib.crc32 in Python; Adler-32 is zlib.adler32 (starts at 1, modulus 65521). The zlib stream trailer is big-endian Adler-32, gzip trailer little-endian CRC-32 then size, see Library:Zlib.
  • Seeds and chaining: formats may start the CRC from a non-default init (a previous block's CRC, or the file length). Try init 0, 0xFFFFFFFF, and continuing from the header CRC.
  • Result post-processing: stored value may be inverted, byte-swapped, or masked (Snappy masks CRC-32C by rotating 15 and adding 0xA282EAD8, see Format:Snappy).
  • Reversed polynomial test: if your tool wants a polynomial and you only know 0xEDB88320, the normal form is 0x04C11DB7.

CRC parameter search

When the checksum is a CRC but the parameters are unknown, the RevEng tool (reveng, by Gregory Cook) can search width, poly, init, xorout and reflection from several samples of data plus checksum (its -s search with -w width and -l/-b options and hex samples). Roughly: give it at least as many samples as the width needs, ideally differing in length, with the stored checksum written in the same byte order. Its catalogue is the source of the standard names. madler's crcany (Tool:crcany) generates C code for any catalogued CRC. The Python crcmod package, pycrc and CRC RevEng cover the same models. Unverified: command line details of reveng were not tested here; read its manual for your version.

For a very short field, or when the algorithm is not a CRC, vary one input byte at a time and solve linear structure: a CRC is linear over GF(2) (crc(a xor b) xor crc(0) is crc(a) xor crc(b) xor crc(0) for equal-length inputs), a sum is additive, a hash is neither. Two files that differ in one bit with the checksum difference depending only on the bit position, not on the other data, are a CRC.

Pitfalls

  • The CRC may include the checksum field itself, set to zero or a placeholder.
  • Per-block checksums plus a master checksum over the block checksums.
  • Encrypted or compressed containers checksum the plaintext (decrypt first) or the compressed bytes (do not decompress first); check both.
  • A checksum of zero or an all-ones value may be accepted as "not set" by the loader.
  • Tamper checks that compare a stored value to a table of known hashes are not checksums in this sense; treat them as signatures.

See also

Save-file workflow around it: Technique:Reverse engineering game save files.