Reverse Engineering Malware: What It Is and Where It Stops

Reverse engineering malware explained: the static and dynamic flow, what changes with packed and VM-protected samples, and whether it is legal.

Reverse engineering malware is the analysis of a malicious program's compiled code to work out what it does, how it persists and how to detect it, without access to its source. Analysts combine static analysis, which reads the binary in a disassembler or decompiler, with dynamic analysis, which runs the sample in an isolated lab and records its behavior.

The method works well until the code is protected. MITRE ATT&CK tracks protection as sub-technique T1027.002, Software Packing, and lists VMProtect and Themida among the tools tracked groups have used. A study ANY.RUN published on 18 June 2024 found that the protected samples it examined were easier to analyze than they looked. This post covers the usual flow, the point where it breaks, and the legal question analysts get asked most.

What is reverse engineering malware?

Reverse engineering malware means working backwards from a compiled sample (usually a Windows PE file, an x86 or x64 executable or DLL) to the logic its author wrote. The output is not the original source code. Names, and structure are gone at compile time. What an analyst recovers is behavior: what the sample reads and writes, which processes it touches, how it talks to its command-and-control server, how it survives a reboot, and which strings, hashes and network patterns can detect it.

Defenders do this work in incident response, threat intelligence, antivirus and EDR research, and detection engineering. The shared goal is a defense: a detection rule, a containment step or an accurate account of what an intrusion did.

Why do analysts reverse engineer malware?

Analysts reverse engineer malware when a sandbox report is not enough. An automated sandbox shows what a sample did during one run. Reverse engineering explains why, including branches that never fired:

Dormant behavior. A sample may wait for a date, a domain or a keyboard layout before it acts.

Configuration. C2 addresses, encryption keys and campaign IDs often sit encrypted inside the binary.

Capability. An incident lead needs to know whether a sample can exfiltrate data or move laterally, not just whether it did in the lab.

Detection. Durable detections key on code and behavior, not on a hash that changes every build.

The usual flow: triage, static analysis, dynamic analysis

Malware reverse engineering follows a loop rather than a straight line, but most work starts in the same order.

Triage decides how much effort a sample deserves. Analysts hash the file, check it against threat-intel sources, read the PE headers, list imports and strings, and check whether the file is packed. High entropy, a tiny import table and section names from a known protector are the usual tells.

Static analysis reads the code without running it. A disassembler or decompiler such as IDA or Ghidra turns machine code into assembly and C-like pseudocode, and the analyst follows the functions that matter: the entry point, the network routines, the persistence setup and the decryption of strings and configuration.

Dynamic analysis runs the sample in an isolated virtual machine with no route to production networks, while a debugger, process monitor and network capture record what happens. It answers questions static analysis cannot reach quickly, such as what a decrypted string says.

What changes when a sample is packed or VM-protected?

A packed or VM-protected malware sample hides its code from static analysis until it runs. MITRE ATT&CK tracks this as sub-technique T1027.002, Software Packing, which covers both compression packers such as UPX and virtual machine software protection, where the original code is translated into bytecode that only an embedded virtual machine can run. Packing usually falls to unpacking; virtualization does not.

The two cases look different in a disassembler:

A packed sample shows a small stub that decompresses or decrypts the real code into memory. Dump the process after the stub finishes and the original instructions are back.

A virtualized function never turns back into x86 or x64 instructions. The protector converted it to its own bytecode before the file shipped, and an interpreter (a dispatcher loop and dozens of small handlers) runs that bytecode at runtime. A decompiler shows the interpreter, not the malware.

That second case is where most beginner tutorials stop. The analyst has three options: trace the virtual machine dynamically and accept partial coverage, translate the bytecode back into readable code (devirtualization), or bound the work and report which functions stayed unread. Devirtualization vs deobfuscation vs unpacking compares the three kinds of protection and what removes each one.

Does every protected sample need devirtualization?

Not every protected malware sample needs devirtualization. In a study published on 18 June 2024, ANY.RUN examined Themida- and VMProtect-protected samples from five malware families and found that none of the samples used the protectors' code virtualization. The protectors were used much like basic packers, so unpacking and dynamic analysis reached the readable code.

The ANY.RUN write-up covers Arkei, RisePro, PrivateLoader, Amadey and Lumma. Its conclusion is that the hard parts were string obfuscation and C2 concealment inside the malware itself, not the protector. The practical lesson for triage is to check whether the functions you need are virtualized before planning a devirtualization job.

When they are, the job changes. Automated devirtualization services exist for exactly this case. Devirt, for example, returns the recovered functions of an authorized sample as an IDA database or annotated pseudocode, with a coverage ledger of which functions were proven and which stayed open. Its page for malware analysts explains the scoping, and an unfamiliar build can start with a free assessment.

What should the analysis produce?

A reverse engineering pass is finished when it hands the people defending the network something they can act on: a short summary, the observed and possible behavior, persistence, indicators of compromise, the configuration recovered, and a clear statement of what was proven and what was not. That last item is the one most reports leave out, and it matters more on protected samples, where some functions may stay unreadable.

Is it legal to reverse engineer malware?

Reverse engineering malware for defense is a standard part of security work, done by incident responders, threat-intelligence teams and antivirus vendors. The legal questions usually concern the samples you hold and the terms you agreed, not the analysis itself, so analysts work only on samples they are authorized to handle and keep them in an isolated environment.

Services that handle samples draw the same line in their terms. Devirt's acceptable use policy, for example, accepts malware analysis and defensive threat intelligence and prohibits deploying or improving malware. This is general information, not legal advice. If your situation involves third-party software licenses or a specific jurisdiction, ask counsel.

For what a finished analysis should contain, see our SOC deliverable checklist on DEV.

To scope a protected sample, start on Devirt's malware analyst page .

Written by Devirt, an automated devirtualization service for authorized analysis of protected software.