Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Technique:Reversing network protocols

From ReverseEngineering.wiki
Technique
On this wiki Facts · Code · Notes · Topics

How to work out an application's own network protocol from captures of your own traffic: capture, strip the transport, find message boundaries, then length fields, type ids, checksums, compression and serialization, and write a dissector to test the model.

Scope

For analysing your own application, a server you run or a documented protocol. Capture only traffic you own. Encrypted transports (TLS) need the client's own key log (Format:SSLKEYLOGFILE, which can be embedded in a Format:Pcapng Decryption Secrets Block) or a debug build; do not intercept other people's traffic.

Workflow

  1. Capture with tcpdump or Wireshark into Format:pcap (libpcap savefile) or Format:pcapng. Use the loopback or a dedicated interface, fixed ports, and short scripted sessions (one action per capture) so differences are attributable. Note snaplen: truncated packets break length analysis.
  2. Separate the layers. Check the link type (see the pcap page), then IP/TCP or UDP. Find the stream with "Follow TCP stream" and look at each direction as raw bytes. Check first whether it is a known protocol on a nonstandard port (HTTP, TLS, DNS, MQTT, WebSocket): if the first bytes are 16 03 0x it is a TLS handshake; see Technique:TLS client fingerprinting for what the Client Hello reveals about the client library.
  3. Find message boundaries. TCP is a byte stream, so one packet is not one message. Look for (a) a fixed-size header, (b) a length prefix, (c) a delimiter such as CRLF or a NUL, (d) one message per UDP datagram. Reassemble per stream before judging.
  4. Length fields. In a hexdump of several messages of different sizes, look for a 1, 2, 3 or 4 byte integer at a fixed offset that equals the message size, size minus header, or size minus a constant; try both byte orders. Varints (7 bits per byte, high bit as continuation) are typical of Format:Protocol Buffers wire format and many custom protocols.
  5. Type / opcode ids. A small field near the start that takes few distinct values and correlates with the action you triggered. Make a table: value, direction, size range, what you did. Request and response often share the id or differ by one bit.
  6. Sequence and session fields. A counter that increments per message, a constant session id after login, or an echoed request id. Change one action and diff.
  7. Checksums. A trailing 1, 2 or 4 byte field that changes with the content: use the method of Technique:Identifying checksums in file formats (CRC-16/32 parameter search, sums, XOR). Remember transport checksums (TCP/UDP) are offloaded and show as wrong in captures taken on the sending host; this is normal.
  8. Compression. High-entropy payload with no readable strings after the header may be compressed or encrypted. Test for magics: zlib 78 9C / 78 01 / 78 DA (Format:Zlib stream, Format:DEFLATE), gzip 1F 8B, zstd 28 B5 2F FD, LZ4 frame 04 22 4D 18 (Unverified: magic values from general knowledge). See Technique:Identifying compression libraries in binaries. If the client has a compression library imported, trace the call that sits next to the send.
  9. Serialization. Quick tells by first bytes: protobuf tags like 08 0A 12 (Format:Protocol Buffers wire format); a Thrift compact message starts 82 21 or 82 41, a Thrift struct uses field-delta nibbles and ends in 00 (Format:Thrift compact protocol); a MessagePack map starts 8N or DE followed by fixstr keys A0-BF (Format:MessagePack); CBOR maps start A0-BF with keys 60-7B, and may begin D9 D9 F7 (Format:CBOR); BSON starts with an int32 equal to the total length and ends in 00 (Format:BSON); a FlatBuffers buffer starts with a small forward u32 offset and often a 4-letter identifier at offset 4 (Format:FlatBuffers). Readable field names mean JSON/BSON/MessagePack-like text-ish formats; tag bytes 0x08 0x0A 0x12 and length-delimited strings point to Format:Protocol Buffers wire format; fixed-offset structs point to C structs sent raw (check padding and endianness).
  10. Cross-check with the binary. Search the client for the type ids as constants, the strings that appear in the packets, and the functions that call send/WSASend/SSL_write; names found go on the software page.
  11. Write a dissector (Tool:Wireshark Lua dissectors) so every new capture is parsed automatically; unparsed trailing bytes are your to-do list.

Pitfalls

  • Fields change with the version of the client: record it.
  • Padding and alignment make lengths look odd; look at several sizes.
  • Nagle and delayed ACK merge or split messages; never rely on packet boundaries for TCP.
  • Data may be obfuscated with a fixed XOR key: XOR of two ciphertexts of the same plaintext is zero.
  • UDP game protocols often have reliability layers (sequence, ack bitmaps) before the real message.