VMProtect has three protection modes you can apply to any function: Mutation, Virtualization, and Ultra. Most write-ups focus on the VM because it is the most visible. Mutation gets skipped because on the surface it just looks like junk code. But it is worth understanding how the mutation engine works and what each mode actually does to the binary, because the differences matter a lot when you are trying to analyze a protected function.
Mutation Mode
In Mutation mode, VMProtect does not replace your code with bytecode. The function stays as native x86. What changes is the structure of the code itself. The engine reads the original function, disassembles it, and applies a series of source-to-destination transforms. Each transform takes a pattern of one or more instructions and replaces it with a semantically equivalent but structurally different sequence.
A single add can become a sequence that computes the same result through three intermediate values. A compare-and-jump can become a computed branch using an arithmetic result routed through a dispatch table. Constants can be split into multiple parts that are combined at use time instead of appearing as a single immediate. The original function is gone and what runs in its place is a cloud of mutated code that produces the same outputs but looks completely different.
The xor key is randomized per build. The net effect on EAX is identical to add eax, 5 but there is no add with an immediate 5 anywhere in the output. Pattern matching against the original will not find anything.
What Gets Mutated
Mutation applies to the body of the function but also to the entry and exit sequences, any jump target alignment stubs, and in Ultra mode (covered below) the VM handler bodies themselves. The engine works off a pattern library. Each entry in the library is a before pattern and one or more after patterns. The engine picks randomly from the valid after patterns each time it processes a match, so two builds protecting the same function will produce different output.
Not everything can be mutated. Instructions that depend on flags in complex ways are harder to transform without breaking correctness. The engine is conservative in those cases and either leaves them alone or applies lighter transforms that preserve the exact flag semantics.
Junk Code Insertion
In addition to transforming real instructions, the engine inserts dead instructions that look like real operations but have no effect on the output. These are called opaque predicates in the literature. A branch that always takes the same path, based on a computation that always evaluates the same way, is inserted to confuse control flow analysis. Dead writes to scratch registers, arithmetic on values that are never read, and conditional moves where both paths produce the same result are all common forms.
Control Flow Flattening
Mutation also flattens control flow. A function with a normal if-else structure becomes a single dispatch loop where a state variable determines which block runs next. The original edges between blocks are replaced with computed transitions. The decompiler sees a switch statement instead of the original control flow, and the switch arms themselves are mutated.
The state constants are randomized and do not correspond to the original block ordering. To recover the original control flow you have to evaluate which blocks transfer to which and rebuild the graph from the transitions.
Virtualization Mode
Virtualization mode is what I described in Part 4. The function body is translated to VM bytecode and the original instructions are replaced with a stub that enters the VM. The mutation engine is not involved in transforming the bytecode. The bytecode is a faithful translation of the original operations. What is obfuscated is the representation: custom opcodes, encrypted stream, handler dispatch.
The key point here is that in plain Virtualization mode the handler code itself is not mutated. Each handler is the original, unmodified implementation. This means a skilled analyst who identifies the handlers can recover a clean mapping of opcode to operation and lift the bytecode back to something readable.
Ultra Mode
Ultra mode combines both. The function is first virtualized and then the handler code is also run through the mutation engine. The bytecode itself is not changed, but every handler body gets the full mutation treatment: transformed instruction sequences, junk code, and control flow flattening.
This breaks the shortcut that Virtualization mode allows. In Virtualization mode you can find the add handler, read it, and verify it does add. In Ultra mode the add handler is a cloud of mutated instructions that eventually produce an add result. You cannot read the handler directly. You have to either symbolically execute it to derive what it does or use dynamic tracing.
Constant Unfolding
One specific transform that shows up everywhere, in all three modes, is constant unfolding. Any time there is an immediate value in the original code, the mutation engine can replace it with a runtime computation that produces the same value. The computation itself can be another mutated sequence.
A single 32-bit immediate can be replaced with a seed value XORed through several intermediate steps, combined with a rotation, and masked back down. The final result is the original constant but it never appears as a literal anywhere. Tools that look for specific constant values in binary searches will not find them.
Register Renaming
Mutation also renames registers. A sequence that originally uses EAX and EBX might be rewritten to use ESI and EDI with equivalent logic. Since the function is entirely rewritten, the choice of registers is unconstrained. Two protected copies of the same function will use different scratch registers and may move live values through completely different temporaries.
This makes signature matching against the mutated code unreliable. You cannot write a byte pattern for a function and expect it to match a different build even if the source code is identical.
Build-to-Build Variance
Everything I described here is randomized per protection run. The xor keys used in constant unfolding, the opaque predicate conditions, the state variable values in the dispatch table, the register assignments, and the selection of which transform variant to apply are all drawn from a random pool each time. Two builds protecting the same function will produce code that does the same thing but shares no recognizable byte sequences.
This is the actual threat model. It is not just that the code is obfuscated today. It is that every copy shipped to every user is uniquely obfuscated, so a bypass developed against one copy will not work on another without significant rework.
Next
In part 6 I will cover devirtualization. What the current tooling looks like, how symbolic execution approaches the problem, and where the analysis actually breaks down when Ultra mode is involved.