Jump to content
Setting up an AI agent? ReverseEngineering.wiki skill (works from the command line, no configuration): https://reverseengineering.wiki/skill/rewiki/SKILL.md · setup for every client, including MCP: https://reverseengineering.wiki/llms.txt · REWiki:Connect

Library:Protocol Buffers

From ReverseEngineering.wiki
Library
On this wiki Facts · Code · Notes · Topics

Google's schema-based serialization library and the protoc code generator (BSD-3-Clause). Programs that use it contain generated message classes plus the full serialized .proto file as an embedded string, so the message definitions can usually be recovered from the binary without any decompilation.

Overview

protoc compiles .proto files into per-language code (C++, Java, Python, C#, Go, Rust, ...). Messages are encoded with Format:Protocol Buffers wire format. In the C++, Python and Java generators the whole FileDescriptorProto of each .proto file is serialized and embedded in the output as a byte string, so that reflection, text format, JSON and Any work at runtime. The embedded descriptor contains message, field, enum, service and method names, field numbers, types and labels, and the file name and package, which is nearly the original .proto. Only comments are missing (SourceCodeInfo is not embedded) and, in newer versions, options with source retention are stripped. Details and the recovery recipe: Library:Protocol Buffers/Embedded descriptors.

Identifying protobuf in a binary

  • C++: symbols/RTTI containing google::protobuf::Message, MessageLite, internal::DescriptorTable, .?AVMessage@protobuf@google@@ (MSVC) or _ZTVN6google8protobuf7MessageE (Itanium). Generated classes derive from ::google::protobuf::Message (full runtime) or MessageLite (optimize_for = LITE_RUNTIME).
  • C++ generated data symbols: descriptor_table_<file>_2eproto (the DescriptorTable), descriptor_table_protodef_<file>_2eproto (the serialized descriptor), where non-alphanumeric characters of the .proto path become _ plus two hex digits (foo/bar.proto -> foo_2fbar_2eproto). Present only if not stripped; the strings below survive stripping.
  • The embedded blob begins with tag 0x0A (field 1 = file name) then the length-prefixed path ending in .proto, so search for the bytes 0A xx ... 2E 70 72 6F 74 6F (.proto); google/protobuf/descriptor.proto, google/protobuf/any.proto, google/protobuf/timestamp.proto are in nearly every full-runtime binary.
  • Strings: type.googleapis.com/ (Any type URL prefix), libprotobuf ERROR, CHECK failed: (absl logging), proto3 (syntax field, field 12 of the descriptor), google.protobuf. prefixes.
  • Generated C++ headers contain the comment // Generated by the protocol buffer compiler. DO NOT EDIT! and // Protobuf C++ Version: <ver> (not compiled into binaries); the version is checked at compile time against PROTOBUF_VERSION (7038000-style: major1000000 + minor1000 + patch, with the language-runtime version, e.g. C++ 7.x belongs to protoc 3x).
  • Python: _descriptor_pool.Default().AddSerializedFile(b'...') in *_pb2.py; a # Protobuf Python Version: header and _runtime_version.ValidateProtobufRuntimeVersion( call in recent generators; error texts Detected incompatible Protobuf Gencode/Runtime versions when loading.
  • Java: descriptorData = { ... } string array passed to internalBuildGeneratedFileFrom.
  • gRPC services show up as ServiceDescriptorProto entries; method full names /pkg.Service/Method are used as HTTP/2 paths.

Generated C++ class shape

A message class has FullMessageName() returning the full proto name (a plain string in the binary), GetClassData(), _InternalSerialize / table-driven parse (TcParseTable) in new versions, _internal_metadata_, a _has_bits_ word array for optional fields and _cached_size_. Field numbers appear as k<Name>FieldNumber constants in headers (not in the binary); in the binary the field numbers are in the parse tables and, as data, in the embedded descriptor.

Topics

  • Embedded descriptors — How to find and decode the FileDescriptorProto that protoc embeds in generated C++, Python and Java code, how the DescriptorTable that points to it is laid…

Pitfalls

  • optimize_for = LITE_RUNTIME and strip_nonfunctional_codegen builds have no (or an empty) descriptor: field names are gone; use the wire format and the parse tables.
  • Descriptors above 65535 bytes are emitted as a char array on MSVC instead of a string literal (no difference in the binary).
  • Obfuscators and some mobile SDKs rename messages but the field numbers/types are unchanged.
  • Third-party runtimes (nanopb, protobuf-c, Go protobuf-go, prost, Swift) have different layouts; Go embeds the descriptor as file_<path>_proto_rawDesc (unverified here, from the protobuf-go project, not checked).

Code & repositories

Evidence & references

  • Embedding of the serialized descriptor and naming: src/google/protobuf/compiler/cpp/file.cc lines 1152-1285, compiler/cpp/helpers.cc lines 989-1008, compiler/python/generator.cc lines 340-370 and 571-617, compiler/java/shared_code_generator.cc lines 142-188 (commit 522b8d41d5d5cdc7ef581c913277aa309f9b7c6a).