4.2 Malware Analysis and Detection

EC-Council Certified Ethical Hacker CEH

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 CEH Exam Format, Cost & Eligibility 1.2 The Five Phases of Ethical Hacking 1.3 Information Security Fundamentals 1.4 Ethical Hacking Concepts, Scope & Legal Framework 1.5 The CEH Domains & Your Study Priority Map

2 Reconnaissance, Footprinting, Scanning & Enumeration

2.1 Footprinting & Reconnaissance Concepts 2.2 Footprinting Sources & Techniques 2.3 Network Scanning Concepts 2.4 Banner Grabbing, OS Fingerprinting & Scanning Beyond IDS/Firewall 2.5 Enumeration 2.6 Vulnerability Scanning & Countermeasures

3 Vulnerability Analysis & System Hacking Phases

3.1 Vulnerability Analysis Concepts 3.2 System Hacking Methodology Overview 3.3 Gaining Access: Password Attack Concepts 3.4 Privilege Escalation Concepts 3.5 Maintaining Access: Backdoors, Rootkits & Persistence 3.6 Covering Tracks, Anti-Forensics & Blue-Team Detection

4 Malware Threats, Sniffing & Social Engineering

4.1 Malware Concepts and Taxonomy 4.2 Malware Analysis and Detection 4.3 Sniffing Concepts and Layer 2 Attacks 4.4 Sniffing Countermeasures and Detection 4.5 Social Engineering Types and Human Psychology 4.6 Social Engineering and Insider Threat Defenses

5 Network & Perimeter Hacking

5.1 Denial of Service and DDoS Attacks 5.2 Session Hijacking 5.3 Evading IDS and IPS (Defensive Focus) 5.4 Evading Firewalls 5.5 Evading Honeypots and Defensive Deception 5.6 Network Security Controls and Architecture 5.7 Packet Analysis and Secure Protocols for Defenders

6 Web Server & Web Application Hacking

6.1 Web Server Attacks & Hardening 6.2 OWASP Top 10 Overview 6.3 Injection: SQL Injection & Command Injection 6.4 Cross-Site Scripting (XSS) & CSRF 6.5 Authentication, Session & Access-Control Flaws 6.6 API & Web App Security Testing Concepts

7 Wireless Network Hacking

7.1 Wireless Concepts and Standards 7.2 Wireless Encryption 7.3 Wireless Threats and Attack Concepts 7.4 Wireless Security Countermeasures

8 Mobile Platform, IoT & OT Hacking

8.1 Mobile Platform Threats & Security Models 8.2 Mobile Attack Vectors & Risks 8.3 Mobile Device Management & Enterprise Defense 8.4 IoT Hacking Concepts & Defense 8.5 OT, ICS & SCADA Security

9 Cloud Computing & Cryptography

9.1 Cloud Computing Concepts 9.2 Cloud Threats & Attack Surfaces 9.3 Cloud Security Controls 9.4 Cryptography Fundamentals 9.5 PKI & Cryptographic Applications 9.6 Cryptanalysis, Crypto Attacks & Best Practices

4.2 Malware Analysis and Detection

Key Takeaways Explain This Topic AI

Static analysis inspects a sample without running it (hashes, strings, headers, imports, disassembly); dynamic analysis observes behavior while it runs in an isolated sandbox.

Sandboxing must isolate the sample and assume modern malware tries to detect virtual/analysis environments and stay dormant (anti-VM, anti-debugging, sleep timers).

Indicators of Compromise (IoCs) are forensic artifacts — file hashes, C2 domains/IPs, registry keys, mutexes — used to detect and hunt the same threat elsewhere.

Signature detection catches known malware fast but fails on packed, polymorphic, and fileless threats; heuristic and behavioral detection cover that gap.

Defense in depth combines anti-malware, application allowlisting, least privilege, patching, egress filtering, and tested offline backups — no single control suffices.

Last updated: June 2026

Static Analysis: Examine Without Executing

CEH frames malware analysis as a defensive skill: understanding a sample so you can detect, contain, and hunt it. The two pillars are static and dynamic analysis.

Static analysis inspects the file without running it , which is safe but can be defeated by obfuscation. Core static techniques the exam expects:

File fingerprinting — compute a cryptographic hash ( MD5, SHA-1, SHA-256 ) to uniquely identify the sample and look it up on threat-intel services like VirusTotal .

Strings analysis — extract printable strings to reveal URLs, IP addresses, registry keys, error messages, or commands embedded in the binary.

PE header / import analysis — examine the Portable Executable header and the imported APIs (Import Address Table); calls to CreateRemoteThread , URLDownloadToFile , or crypto APIs hint at intent. Tools: PEiD, PEview, Dependency Walker .

Packer detection — identify whether the binary is packed/encrypted (e.g., UPX). A packed file shows few imports and high entropy, signaling the real code is hidden until runtime.

Disassembly — convert machine code to assembly with IDA Pro, Ghidra , or a debugger to study logic.

Static analysis is fast and low-risk, but packers, crypters, and polymorphism mutate the binary so its hash and signature change every infection, blunting static detection. That limitation is exactly why dynamic analysis exists.

Dynamic Analysis, Sandboxing, and IoCs

Dynamic (behavioral) analysis runs the sample in a controlled, isolated environment and watches what it actually does: files written, processes spawned, registry keys set, and — critically — network connections to command-and-control servers. Key monitoring tools CEH references include Process Monitor and Process Explorer (Sysinternals) for process/registry/file activity, Regshot for before/after registry diffs, Wireshark for captured C2 traffic, and automated sandboxes (Cuckoo, Any.Run, Joe Sandbox).

The sandbox must be isolated from production (no route to the real network, snapshot/rollback enabled) because executing live malware is dangerous. Sophisticated malware practices sandbox evasion : it checks for VM artifacts (MAC address ranges, VMware tools, low CPU/RAM), looks for debuggers ( anti-debugging ), or simply sleeps for minutes to outlast automated analysis. A behaving-too-nicely sample may simply be detecting your lab.

The output of analysis is a set of Indicators of Compromise (IoCs) — concrete artifacts you feed into detection systems to find the threat elsewhere:

IoC type Example
File hash SHA-256 of the dropper
Network C2 domain, IP, URL pattern
Host Registry key, scheduled task, mutex name
Behavioral Spawning PowerShell from Office

IoCs are often shared in STIX/TAXII format and matched in tools like YARA rules. Behavioral IoCs map to attacker techniques in the MITRE ATT&CK framework, which CEH increasingly references.

Detection Methods and Defense in Depth

Anti-malware uses several detection strategies, each with a blind spot:

Signature-based — matches files against a database of known-malware hashes/byte patterns. Fast and accurate for known threats, but blind to new, packed, polymorphic, and fileless malware.

Heuristic — flags suspicious code constructs or behavior patterns that resemble malware, catching some unknown variants at the cost of false positives.

Behavioral / EDR — monitors runtime actions (process injection, mass file encryption, C2 beaconing) and is the only reliable way to catch fileless attacks. Endpoint Detection and Response (EDR) records and correlates this telemetry.

Sandbox detonation — runs unknown attachments/files in a sandbox before delivery.

No single control stops everything, so CEH emphasizes defense in depth for malware:

Patch management to close the exploits worms and droppers use.

Application allowlisting so only approved binaries run (blocks unknown executables outright).

Least privilege so a compromise can't trivially install kernel rootkits.

Email/web filtering and macro disabling to stop the most common delivery path.

Network segmentation and egress filtering to break lateral movement and C2 callbacks.

Tested, offline (immutable) backups — the definitive ransomware countermeasure, because paying is never guaranteed to restore data.

User awareness training , since most malware arrives through a person.

On the exam, when a question asks for the best defense against ransomware specifically, tested offline backups outrank antivirus; when it asks about fileless malware, behavioral/EDR detection beats signatures.

The Triage Workflow and Threat Hunting

A practical malware-analysis workflow ties the techniques together in order:

Isolate the sample and the infected host (pull from the network, snapshot for evidence).

Fingerprint — hash the file (SHA-256) and check threat-intel/VirusTotal for prior sightings.

Static triage — strings, PE header, imports, and packer detection to form a hypothesis cheaply.

Dynamic detonation — run it in an isolated sandbox while capturing process, file, registry, and network activity.

Extract IoCs — record hashes, C2 domains/IPs, dropped files, registry keys, mutexes, and scheduled tasks.

Hunt and remediate — push IoCs and YARA rules across the fleet to find other infected hosts, then rebuild, patch, and rotate credentials.

Threat hunting uses the resulting IoCs and ATT&CK techniques to proactively search telemetry for the same adversary elsewhere — for example, searching EDR logs for the C2 domain or for Office spawning PowerShell. CEH frames anti-malware controls as layers in a broader program, not a silver bullet.

Common evasion the analyst must anticipate:

Evasion technique Defeats
Packing / crypting Static signatures
Polymorphism / metamorphism Hash and pattern matching
Anti-VM / anti-sandbox Automated dynamic analysis
Anti-debugging Manual debugging
Process injection / hollowing Naive process listing
Timestomping / log clearing Forensic timelines

The takeaway for the exam: a single control fails to a determined adversary, so detection must combine signatures (known threats) + heuristics (suspicious constructs) + behavioral/EDR (runtime actions) + sandboxing (pre-delivery) , all backed by patching, least privilege, allowlisting, segmentation, and tested backups . When a scenario describes malware that changed its appearance to slip past antivirus, the analyst should pivot from static signatures to behavioral analysis and IoC-driven hunting .