Technique:Recognizing code obfuscation/Native obfuscation patterns
What compiler-level obfuscation of native code looks like in a disassembler: Obfuscator-LLVM control-flow flattening with a switchVar dispatcher, bogus control flow with always-true predicates, instruction substitution, and string decryption stubs.
Obfuscator-LLVM (O-LLVM)
Obfuscator-LLVM is an LLVM fork (LLVM license) that adds IR passes run at compile time; source is on the llvm-4.0 branch (the master branch holds only a README). Passes live in lib/Transforms/Obfuscation/ (Flattening.cpp, BogusControlFlow.cpp, Substitution.cpp, SplitBasicBlocks.cpp). Because the passes run on IR, the result is ordinary machine code with an odd shape; there is no runtime library to find.
Control-flow flattening (-fla, pass name flattening)
Flattening.cpp splits the function into basic blocks, then builds this skeleton: an entry block that stores an initial value into an i32 alloca named switchVar; a loopEntry block that loads switchVar and runs a switch on it; a switchDefault block; and a loopEnd block that jumps back to loopEntry. Every original block is a case of the switch; at its end it stores the number of its successor into switchVar and jumps to loopEnd. A conditional branch becomes a select between two case numbers. In a decompiler this shows as while (1) { switch (state) { ... } } where state is a stack variable, every case ends with state = constant, and conditional exits assign one of two constants (a cmov/select). Recognition tips: a function whose CFG is a star around one block with a big jump table; very little data flow between cases except through memory; the case order is shuffled relative to the original code. Per-function control uses source annotations fla and nofla; -fla on the command line applies it to all functions.
Bogus control flow (-bcf)
BogusControlFlow.cpp clones a basic block, puts a conditional branch in front of it and makes the clone loop back to the original. Options: -bcf_prob (percent of blocks, default 30) and -bcf_loop (times the pass repeats on a function, default 1). The first condition is built as a floating-point compare of 1.0 with 1.0 (always true), and doFinalization later rewrites it into an opaque predicate written in the source comments as y < 10 || x * (x + 1) % 2 == 0, where x and y are module-level globals (x*(x+1) is always even, so the predicate is always true). Recognition: a conditional on two global variables with a multiply, % 2 (an AND with 1), and a signed compare with 10; one outcome leads to a block that falls back into the original block and is never executed. Annotation names: bcf and nobcf.
Instruction substitution (-sub)
Substitution.cpp randomly picks one of the rewrites for integer add, sub, and, or, xor. Examples named in the source: add as a - (-b), with random-constant forms (addRand), and equivalents for and/or/xor built from the other operators. Recognition: sequences such as neg + sub where an add is expected, or xor/and/or combined with constants that cancel out. Annotation names: sub and nosub. All variants read in lib/Transforms/Obfuscation/Substitution.cpp (obfuscator-llvm/obfuscator commit 42f732c274e7cf9d7ddf2097bfc181de7587147e), from the comments above each function: add has four rewrites, a = b - (-c) (addNeg, line 196), a = -(-b + (-c)) (addDoubleNeg, 218) and two with a random constant r (addRand, addRand2: add r, add c, subtract r); sub has three, a = b + (-c) (subNeg, 301) and two with a random constant (subRand, subRand2); and, or and xor each choose between two rewrites (get_range(2)): b & c as (b ^ ~c) & b, or as the De Morgan form with an always-true (r | !r) term; b | c as (b & c) | (b ^ c); a ^ b as (!a & b) | (a & !b). Mul, div, rem and shifts are visited but left unchanged (lines 148-170). Pass name substitution (line 104).
Basic block splitting (-split)
SplitBasicBlocks.cpp splits each block -split_num times (default 2, allowed up to 10), which multiplies the cases a flattened function has.
Forks
Hikari and other O-LLVM descendants add string encryption, function wrapping, indirect branches. Unverified: this wiki has not read their sources; check the repository of the fork before trusting a pass list.
String encryption with decrypt-at-runtime stubs
Generic shape (not tied to one tool): the string literals are no longer readable in the data section; each use becomes a call to a small function that takes an index or a pointer to an encrypted blob and a key, decodes it (xor, add, rc4-like loop) into a buffer or a heap string, and returns it. Often the decoder runs once per string and caches the result, or all strings are decoded in a static initializer at startup. Analyst approach: find the decoder by its many callers with constant arguments, then emulate it or read the decoded strings at runtime in a debugger; document the algorithm as a fact on the tool.
obfus.h
DIE detects the header-only C/C++ obfuscation library obfus.h by section names matching .obfh plus optional two digits, or the export WhatSoundDoesACowMake (rule protection_obfus_h.5.sg, requires compiler version 6.0 as DIE reports it).
See also Technique:Recognizing code obfuscation.