Software Protection and Reverse Engineering: Techniques, Tools, and Defense

February 9, 2025
Software SecurityEXE PackersDebuggingReverse Engineering

🛡️ Software Protection and Reverse Engineering

Protecting software against piracy, overuse, and reverse engineering is a critical aspect of cybersecurity. This blog post walks through software protection mechanisms, the EXE packing process, and how reverse engineers unpack and analyze protected binaries using tools like x64dbg, Scylla, and more.

🚨 What is Software Protection?

Software protection involves preventing unauthorized use or analysis of applications. Common methods include

  • Anti debugging: Preventing analysis by debuggers.
  • EXE packing: Compressing and obfuscating executables.
  • Runtime encryption and code virtualization.

📦 What is EXE Packing?

EXE packing refers to compressing or protecting executables by

  • Reducing file size
  • Preventing reverse engineering
  • Obfuscating original code

In reversing, both packers and protectors are commonly called packers.

Popular Packers

  • UPX
  • Armadillo
  • VMProtect
  • AsProtect

🎯 Purpose of EXE Packing

  • Prevent software cracking (license key bypass, serial patching)
  • Defeat static and dynamic analysis
  • Reduce distribution file size

💥 How Reverse Engineers Defeat Software Protection

🔓 Unpacking

Let the software decompress in memory, then dump the original EXE from RAM and patch it.

🛠️ Using Loaders

A loader attaches to the packed EXE, waits for unpacking in memory, then patches the running process without modifying the original binary.

🧩 What is Unpacking?

Unpacking = Extracting the original binary from a packed EXE.

Techniques

  • Live debugging using x64dbg or OllyDbg
  • Automatic unpackers (for simple packers only)

🕵️‍♂️ Detecting Packers

🔍 Tools

  • PEiD, DIE automatic packer detection
  • PEview, PE Editor analyze PE header manually

📁 Structure of a Packed EXE

Before PackingAfter Packing
Original codeCompressed/encrypted shell
Normal headersObfuscated headers
Full import tableStub loader with hidden IAT

🔢 Types of Packers

TypeDescriptionExample
ISingle unpacking routine, simpleUPX
IIMulti layer unpackingASPack
IIINon linear unpacking with loopsASProtect
IVInterleaved unpacking & codeACProtect
VFragmented frame based unpackingBeria
VIFragment unpacking on demandArmadillo 8.0
VIIInstruction virtualizationVMProtect, Themida

🧠 Execution Flow of a Packed EXE

  1. Starts at new OEP
  2. Saves registers: PUSHAD or PUSH EBP
  3. Unpacks sections in memory
  4. Resolves Import Address Table (IAT)
  5. Restores registers: POPAD or POP EBP
  6. Jumps to original entry point

Instruction Breakdown

1PUSHAD ; pushes EAX, ECX, EDX, EBX, ESP, EBP, ESI, EDI
2POPAD  ; pops them back

🪛 Unpacking Standard Process

  1. Debug to find the real OEP
  2. Dump the binary at OEP
  3. Fix the Import Table (IAT)
  4. Patch the PE Header

🐞 Unpacking with x64dbg

  1. Load EXE in x64dbg
  2. Trace until PUSHAD or PUSH EBP
  3. Set hardware breakpoint on EBP in stack
  4. Press F9 to continue
  5. Break after POPAD or POP EBP
  6. Trace until JMP OEP
  7. Dump EXE with Scylla plugin
  8. Fix IAT

🔧 Anti Debugging Plugins

Use plugins like:

  • ScyllaHide
  • SharpOD

These bypass anti debugging checks in software and allow reverse engineering even when protections are in place.

🎓 Conclusion

Software protection is a cat and mouse game between developers and attackers. Understanding how EXE packing works, and how it can be reversed using tools like x64dbg, Scylla, and PEview, equips you with valuable insight into defensive software engineering and reverse engineering.

← Previous: DevSecOps