Study guide

11.3 Malware Analysis Techniques for SOC Responders

EC-Council Certified SOC Analyst CSA 312-39

Track your progress as you read

Finished sections get a check mark.

1 Introduction: Current Exam Facts and Blueprint

0.1 Current Exam Facts, Blueprint, and Study Plan

2 Chapter 1: Security Operations and Management Fundamentals

1.1 SOC Overview, Capabilities, and Operational Workflow 1.2 SOC Architectures, Sourcing Models, and Organizational Components 1.3 SOC Maturity, Generations, Metrics, and KPIs

3 Chapter 2: Cyber Threat Vectors, TTPs, and Attack Techniques

2.1 Network, Host, and Password Attack Techniques 2.2 Application, Web, and Session Attack Vectors 2.3 Social Engineering, Email Threats, and Insider Attacks

4 Chapter 3: Threat Frameworks, Methodologies, and Indicators

3.1 Indicators of Compromise (IoCs), Indicators of Attack (IoAs), and Pyramid of Pain 3.2 Cyber Kill Chain and Diamond Model of Intrusion Analysis 3.3 MITRE ATT&CK and MITRE D3FEND Frameworks for SOC Defense

5 Chapter 4: Log Management, Formats, and Centralized Architecture

4.1 Log Fundamentals, Sources, and Host Logging 4.2 Network Device, Web Server, and Database Logs 4.3 Centralized Logging Architectures and Pipeline 4.4 Log Parsing, Normalization, and Field Extraction

6 Chapter 5: SIEM Architecture, Deployment, and Use Case Engineering

5.1 SIEM Architecture, Components, and Deployment Models 5.2 SIEM Use Case Development Lifecycle 5.3 Detection Rule Engineering and Sigma Rules

7 Chapter 6: SIEM Alert Triage, Detection Scenarios, and SOC Operations

6.1 Host and Application-Level Incident Detection Use Cases 6.2 Network-Level and Behavioral Anomaly Detection Use Cases 6.3 Alert Triage Methodology and False Positive Elimination 6.4 SOC Dashboards, Visualizations, and Reporting

8 Chapter 7: Cyber Threat Intelligence (CTI) and Threat-Informed Defense

7.1 Cyber Threat Intelligence (CTI) Lifecycle and Intelligence Types 7.2 Threat Intelligence Sources, Formats, and Platforms 7.3 Integrating Threat Intelligence into SOC and SIEM Operations

9 Chapter 8: Threat Hunting Methodologies and Proactive Detection

8.1 Threat Hunting Principles, Frameworks, and Process 8.2 Threat Hunting Techniques with PowerShell and Scripting 8.3 YARA Rule Crafting and Threat Hunting Tooling

10 Chapter 9: Incident Response Lifecycle, Governance, and Handling Procedures

9.1 Incident Response Frameworks and the SOC-IRT Interface 9.2 Incident Triage, Ticketing, and Prioritization 9.3 Containment, Eradication, Recovery, and Root Cause Analysis

11 Chapter 10: Incident Response Playbooks and SOC Automation

10.1 Specialized Incident Response Playbooks 10.2 Endpoint Detection and Response (EDR) and Extended Detection and Response (XDR) 10.3 Security Orchestration, Automation, and Response (SOAR) Playbooks

12 Chapter 11: Digital Forensics and Evidence Handling for SOC Analysts

11.1 Digital Forensics Fundamentals and Evidence Preservation 11.2 Host, Network, and Memory Forensics Triage 11.3 Malware Analysis Techniques for SOC Responders

13 Chapter 12: Cloud SOC Architectures and Monitoring

12.1 Cloud SOC Foundations and Shared Responsibility 12.2 Azure Security Operations with Microsoft Sentinel 12.3 AWS and Google Cloud Platform SOC Architectures

11.3 Malware Analysis Techniques for SOC Responders

Key Takeaways Explain This Topic AI

Malware analysis proceeds through a tiered maturity model from automated sandboxing to static triage, behavioral dynamic execution, and advanced disassembly/debugging.

Static triage inspects cryptographic hashes, PE headers, imports, exports, and string tables without code execution, identifying capabilities through suspicious Windows APIs.

Dynamic behavioral analysis runs specimens in isolated environments equipped with fake network responders (INetSim) and telemetry monitors (Procmon, Regshot, Wireshark).

Operational handling procedures should defang observables and use an approved encrypted archive, access control, and an out-of-band password when a live sample must be transferred; no single archive password is universally required.

Last updated: September 2026

The Five Stages of Malware Analysis

In a Security Operations Center, malware analysis is not a monolithic activity; it is a structured, risk-managed discipline structured into progressive tiers of analytical depth. As an investigation escalates from initial alert validation to in-depth threat intelligence profiling, the analyst applies increasingly complex and time-intensive techniques. Malware Analysis Maturity Pyramid:

  • [Stage 5: Advanced Dynamic] ── Manual Kernel Debugging (WinDbg, x64dbg)
  • [Stage 4: Advanced Static] ── Disassembly & Decompilation (Ghidra, IDA Pro)
  • [Stage 3: Dynamic Analysis] ── Behavioral Sandboxing (Procmon, Regshot, INetSim)
  • [Stage 2: Static Analysis] ── PE Headers, Strings, Imports, Hashing, Entropy
  • [Stage 1: Automated Analysis] ── VirusTotal, Hybrid-Analysis, Any.Run, Joe Sandbox
  • The Five Analysis Stages

Stage 1: Automated Analysis : Rapid, automated submission of suspicious binaries to online or on-premises sandbox appliances (such as VirusTotal, Hybrid-Analysis, Any.Run, or Joe Sandbox). The system executes the binary automatically and generates a high-level report containing multi-antivirus detection ratios, observed network connections, dropped files, and MITRE ATT&CK technique tags. Automated analysis provides immediate triage within minutes, though sophisticated malware may detect the automated hypervisor environment and suppress its malicious behavior.

Stage 2: Static Analysis (Static Triage) : Examining the file's properties, headers, metadata, strings, and structure without executing any code. Static triage identifies file hashes, confirms true file formats via magic bytes, extracts human-readable text strings, analyzes Portable Executable (PE) headers, inspects imported Windows API functions, and calculates entropy to detect packing. Static analysis reduces execution risk because the analyst does not intentionally run the binary, but malformed files, vulnerable parsers, accidental launches, and active content still justify an isolated analysis environment.

Stage 3: Dynamic Analysis (Behavioral Analysis) : Executing the malware specimen inside a tightly controlled, instrumented, and isolated virtual machine sandbox. The analyst monitors the real-time behavioral impact of the binary on the operating system: newly spawned child processes, injected threads, modified registry autostart keys, dropped filesystem payloads, and generated network socket traffic.

Stage 4: Advanced Static Analysis (Reverse Engineering) : Loading the binary into disassemblers and decompilers (such as NSA Ghidra or IDA Pro) to convert machine bytecode into human-readable assembly instructions (x86/x64) or pseudo-C code. The reverse engineer analyzes cryptographic routines, proprietary network protocol encoding, internal algorithm logic, and dormant backdoor capabilities that never trigger during dynamic sandbox execution.

Stage 5: Advanced Dynamic Analysis (Debugging) : Attaching a user-mode or kernel-mode debugger (such as x64dbg, x32dbg, or WinDbg) to the running malware process. The analyst sets memory breakpoints, steps through execution instructions single-cycle at a time, manipulates register values, bypasses anti-debugging checks, and captures unpacked code directly from memory buffers as the malware self-decrypts.

Malware Analysis Stages Comparison Table

Analysis Stage Primary Tools Employed Analyst Skill Level Operational Risk Actionable Output / Deliverables

1. Automated Analysis VirusTotal, Joe Sandbox, Any.Run, Hybrid-Analysis Tier 1 SOC Analyst Low (Risk of data leakage if confidential files uploaded to public repos) Rapid risk score, community reputation, automated IoC list, sandbox video.

2. Static Analysis strings , FLOSS, PEview, PE-bear, Detect It Easy, CFF Explorer Tier 1 / Tier 2 Analyst None (Binary remains completely dormant and unexecuted) Cryptographic hashes, true file format, suspicious API import list, entropy evaluation, embedded strings.

3. Dynamic Analysis Procmon, Process Explorer, Regshot, Wireshark, INetSim Tier 2 Incident Handler Moderate (Containment failure if sandbox network isolation leaks) Behavioral execution tree, dropped files, registry persistence keys, C2 network signatures, IoCs.

4. Advanced Static Ghidra, IDA Pro, Binary Ninja, Cutter Tier 3 Malware Specialist None (Offline code inspection in disassembler) Decompiled C code logic, C2 encryption algorithm decryption, unexecuted hidden features.

5. Advanced Dynamic x64dbg, WinDbg, OllyDbg, ScyllaHide Tier 3 Reverse Engineer High (Live code execution under debugger; anti-analysis exploits) Unpacked raw executable dumped from memory, memory-decrypted payloads, zero-day analysis.

Static Analysis Tools and Techniques

Static triage provides immediate insight into a specimen's intended functionality while preserving absolute host safety. Analysts follow a structured static analysis checklist.

Cryptographic Hashing and Fuzzy Matching

MD5 and SHA-256 : Generates unique cryptographic fingerprints for exact-match IoC queries across SIEM, EDR, and threat intelligence platforms.

SSDEEP (Context Triggered Piecewise Hashing) : Unlike standard cryptographic hashes where altering a single bit changes the entire hash, SSDEEP generates a fuzzy hash based on rolling byte sequences. It computes percentage similarity between two files, allowing analysts to instantly recognize when a new sample is a minor recompile or variant of a known malware family.

Imphash (Import Hash) : Calculates an MD5 hash of the imported functions and their specific sequence within the binary's Import Address Table (IAT). Malware developers frequently modify strings, icons, and variable names to evade file hash detection, but reuse standard software frameworks and import structures. Binaries exhibiting identical Imphash values almost certainly belong to the same threat group or development pipeline.

File Type Identification & Magic Numbers

Adversaries deliberately disguise executable files by appending benign file extensions (e.g., naming a malicious PE executable invoice.pdf or photo.jpg ). Operating systems rely on file extensions, but file parsers rely on magic numbers —specific hexadecimal byte sequences located at offset zero (the very start) of the file.

True File Format Magic Bytes (Hexadecimal) ASCII Representation Architectural Context

Windows Portable Executable (PE) 4D 5A MZ Executables ( .exe ), dynamic link libraries ( .dll ), drivers ( .sys ). Initialized by Mark Zbikowski's initials.

Linux Executable and Linkable Format 7F 45 4C 46 .ELF Linux native binaries, shared libraries ( .so ), kernel modules.

ZIP Archive / MS Office Open XML 50 4B 03 04 PK.. Standard ZIP archives, modern Office files ( .docx , .xlsx , .pptx ), Android APKs, Java JARs.

PDF Document 25 50 44 46 %PDF Adobe Portable Document Format.

Legacy MS Office Compound File D0 CF 11 E0 .... Legacy Microsoft Office documents ( .doc , .xls , .ppt ).

Java Class File CA FE BA BE .... Compiled Java bytecode.

Strings Extraction: strings vs. FLOSS

Extracting human-readable character sequences embedded within a binary reveals IP addresses, URLs, command-line arguments, registry keys, error messages, and file paths. However, standard utilities like GNU strings only extract plaintext ASCII and 16-bit Unicode sequences.

Threat actors routinely obfuscate strings using XOR loops, stack strings (constructed byte-by-byte in CPU registers right before use), or custom encoding to defeat standard strings extraction. To solve this, Mandiant developed FLOSS (FireEye Labs Obfuscated String Solver) . FLOSS uses advanced static heuristics and automated emulation to detect decoding tight-loops, deobfuscate stack strings, and extract concealed ASCII/Unicode strings automatically without requiring manual reverse engineering.

Portable Executable (PE) Header Analysis

Windows executables conform to the PE format, structured into headers and sections:

.text : Contains the executable machine code instructions executed by the CPU. Configured with Read and Execute ( RX ) permissions.

.data : Contains initialized global and static variables. Configured with Read and Write ( RW ) permissions.

.rdata : Contains read-only data, string literals, and critical tables: the Import Table (APIs the binary imports from the OS) and the Export Table (APIs the binary exposes to external callers).

.rsrc : Contains system resources: icons, dialog boxes, version information, language manifests, and frequently, hidden embedded secondary binaries or scripts.

Suspicious Windows API Imports Matrix

By reviewing the Import Address Table using tools like PEview , PE-bear , or CFF Explorer , an analyst infers the binary's underlying architectural intent.

Windows API Function Exporting DLL Targeted Adversary Capability / Malware Intent

VirtualAlloc / VirtualAllocEx kernel32.dll Allocates memory pages within the local or a remote process. Frequently configured with RWX permissions for shellcode staging.

WriteProcessMemory kernel32.dll Writes data (such as injected shellcode or DLL paths) into the allocated virtual memory space of a remote process.

CreateRemoteThread kernel32.dll Spawns a new thread of execution inside another running process (classic DLL injection or process hollowing).

NtCreateSection / NtMapViewOfSection ntdll.dll Low-level kernel native APIs used for process hollowing and memory sharing, bypassing user-mode API hooking.

SetWindowsHookEx user32.dll Installs an application-defined hook procedure into a hook chain; heavily utilized by keyloggers to intercept global keystrokes.

InternetOpen / InternetReadFile wininet.dll Establishes HTTP/HTTPS network communication, used for C2 beaconing and retrieving secondary payloads.

URLDownloadToFile urlmon.dll Downloads a payload directly from a remote web server and writes it directly to disk in a single command.

IsDebuggerPresent / CheckRemoteDebuggerPresent kernel32.dll Anti-analysis checks querying the Process Environment Block ( PEB.BeingDebugged ) to detect if a security analyst is debugging the process.

RegCreateKeyEx / RegSetValueEx advapi32.dll Creates or modifies Windows Registry keys, frequently targeted at Run and RunOnce keys to achieve persistence.

AdjustTokenPrivileges advapi32.dll Enables administrative security privileges (such as SeDebugPrivilege ), allowing the process to inspect and inject into lsass.exe .

Packing Detection and Shannon Entropy

To defeat antivirus scanners and static analysis, malware authors utilize packers (such as UPX, Themida, ASPack, or VMProtect). A packer compresses or encrypts the original executable and bundles it with a lightweight decompression stub. When executed, the stub unpacks the payload directly into memory and transfers CPU execution to the unpacked entry point.

Analysts detect packed binaries using two key indicators:

Import Address Table Scarcity : A standard Windows program imports dozens or hundreds of functions across multiple DLLs. A packed binary typically imports only two to four functions—specifically LoadLibraryA and GetProcAddress from kernel32.dll —which the stub needs to dynamically resolve imports after unpacking.

Shannon Entropy Analysis : In information theory, entropy measures the degree of randomness in a dataset on a scale from 0.0 (completely uniform) to 8.0 (completely random, maximum uncertainty). Plaintext source code and standard compiled machine instructions typically measure between 5.5 and 6.8 . Compressed or strongly encrypted data exhibits an entropy score approaching 8.0 . A very high entropy score in an executable section—especially alongside a sparse import table—is a packing, compression, or encryption indicator, not definitive proof. Legitimate compressed data and specialized binaries can also have high entropy.

Dynamic Analysis Tools and Techniques

Dynamic analysis reveals the true behavioral actions of a specimen as it executes in real time.

Sandbox Architecture and INetSim Network Simulation

A dynamic analysis sandbox should use layered isolation and controlled simulation to reduce the risk that malware reaches production networks or the public internet. No lab design makes escape risk literally zero, so teams also use access controls, monitoring, disposable hosts, validated snapshots, and documented procedures.

Host-Only Virtual Network : The guest analysis virtual machine is attached to an internal, host-only virtual switch with no default gateway leading to the physical network card.

INetSim (Internet Network Services Simulation) : Because many malware specimens immediately terminate or refuse to unpack if internet connectivity is missing, analysts pair the guest detonation machine with a secondary Linux VM hosting INetSim . INetSim emulates standard internet application protocols: DNS, HTTP, HTTPS, SMTP, FTP, NTP, TFTP, and IRC. When the malware attempts to resolve a C2 domain (e.g., evil-c2.com ), INetSim's DNS service resolves the query to its own local IP. When the malware issues an HTTP GET request to download a secondary payload, INetSim returns an HTTP 200 OK response with a benign dummy executable. This convinces the malware that it is online, prompting it to proceed with subsequent operational stages while safely trapping all network telemetry locally.

Endpoint Behavioral Monitoring Tools

Sysinternals Process Monitor (Procmon) : Captures real-time operating system calls across four core subsystems: Process/Thread activity, File System activity, Registry activity, and Network connections. Because Procmon captures tens of thousands of benign events per minute, analysts apply strict display filters: Process Name is malware.exe , Operation is RegSetValue , or Operation is CreateFile .

Sysinternals Process Explorer : Provides an interactive view of running processes, parent-child lineages, loaded DLL modules, and operating system handles. It highlights newly spawned processes in green and terminating processes in red, enabling rapid visual identification of process hollowing.

Regshot : An open-source utility that takes an initial snapshot (Shot 1) of the entire Windows Registry before malware execution, and a second snapshot (Shot 2) post-execution. Regshot computes a text-based differential report listing every key added, deleted, or modified, immediately pinpointing persistence mechanisms and system configuration tampering.

Wireshark : Running on the host network interface, Wireshark captures all raw socket communications between the detonation VM and INetSim, recording full HTTP headers, user-agent strings, and raw payload requests.

Safe Handling, Defanging, and Sample Storage

Handling live malware carries inherent operational risk. SOC procedures enforce strict containment controls to prevent accidental detonation:

Defanging Observables : Before sharing malicious domains, URLs, or IP addresses in tickets, emails, or reports, analysts must defang them to prevent recipients or email clients from accidentally hyperlinking or triggering network calls:

  • Replace http:// with hxxp:// and https:// with hxxps://
  • Bracket dots in domains and IPs: malicious[.]com , 198[.]51[.]100[.]24
  • Example: http://bad.org/worm.exe becomes hxxp://bad[.]org/worm[.]exe

Protected Sample Handling : Do not place live malware unencrypted on ordinary shared drives or send it through standard messaging channels. Security controls may quarantine it, but the larger risks are accidental execution, unauthorized access, uncontrolled replication, and policy violation. When policy permits transferring a live specimen, store it in an approved encrypted container with least-privilege access and communicate the password out of band. Values such as infected are common conventions, not universal requirements; follow the receiving laboratory’s procedure. File extensions should be altered (e.g., renaming malware.exe to malware.bin or malware.sample ) to prevent accidental execution by double-clicking.

Loading diagram... Isolated Dynamic Malware Analysis Sandbox Architecture