5.3 Evading IDS and IPS (Defensive Focus)

EC-Council Certified Ethical Hacker CEH

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 CEH Exam Format, Cost & Eligibility 1.2 The Five Phases of Ethical Hacking 1.3 Information Security Fundamentals 1.4 Ethical Hacking Concepts, Scope & Legal Framework 1.5 The CEH Domains & Your Study Priority Map

2 Reconnaissance, Footprinting, Scanning & Enumeration

2.1 Footprinting & Reconnaissance Concepts 2.2 Footprinting Sources & Techniques 2.3 Network Scanning Concepts 2.4 Banner Grabbing, OS Fingerprinting & Scanning Beyond IDS/Firewall 2.5 Enumeration 2.6 Vulnerability Scanning & Countermeasures

3 Vulnerability Analysis & System Hacking Phases

3.1 Vulnerability Analysis Concepts 3.2 System Hacking Methodology Overview 3.3 Gaining Access: Password Attack Concepts 3.4 Privilege Escalation Concepts 3.5 Maintaining Access: Backdoors, Rootkits & Persistence 3.6 Covering Tracks, Anti-Forensics & Blue-Team Detection

4 Malware Threats, Sniffing & Social Engineering

4.1 Malware Concepts and Taxonomy 4.2 Malware Analysis and Detection 4.3 Sniffing Concepts and Layer 2 Attacks 4.4 Sniffing Countermeasures and Detection 4.5 Social Engineering Types and Human Psychology 4.6 Social Engineering and Insider Threat Defenses

5 Network & Perimeter Hacking

5.1 Denial of Service and DDoS Attacks 5.2 Session Hijacking 5.3 Evading IDS and IPS (Defensive Focus) 5.4 Evading Firewalls 5.5 Evading Honeypots and Defensive Deception 5.6 Network Security Controls and Architecture 5.7 Packet Analysis and Secure Protocols for Defenders

6 Web Server & Web Application Hacking

6.1 Web Server Attacks & Hardening 6.2 OWASP Top 10 Overview 6.3 Injection: SQL Injection & Command Injection 6.4 Cross-Site Scripting (XSS) & CSRF 6.5 Authentication, Session & Access-Control Flaws 6.6 API & Web App Security Testing Concepts

7 Wireless Network Hacking

7.1 Wireless Concepts and Standards 7.2 Wireless Encryption 7.3 Wireless Threats and Attack Concepts 7.4 Wireless Security Countermeasures

8 Mobile Platform, IoT & OT Hacking

8.1 Mobile Platform Threats & Security Models 8.2 Mobile Attack Vectors & Risks 8.3 Mobile Device Management & Enterprise Defense 8.4 IoT Hacking Concepts & Defense 8.5 OT, ICS & SCADA Security

9 Cloud Computing & Cryptography

9.1 Cloud Computing Concepts 9.2 Cloud Threats & Attack Surfaces 9.3 Cloud Security Controls 9.4 Cryptography Fundamentals 9.5 PKI & Cryptographic Applications 9.6 Cryptanalysis, Crypto Attacks & Best Practices

5.3 Evading IDS and IPS (Defensive Focus)

Key Takeaways Explain This Topic AI

An IDS detects and alerts on malicious traffic (passive, out-of-band); an IPS sits inline and can actively block/drop it. Detection methods are signature-based, anomaly-based, and stateful protocol analysis.

Insertion forces the IDS to accept a packet the end host rejects; evasion forces the IDS to reject a packet the end host accepts — both desynchronize what the sensor sees from what the target processes.

Fragmentation, TTL manipulation, and overlapping/out-of-order segments exploit differing reassembly between the IDS and the target to slip attacks past signatures.

Encoding/obfuscation (Unicode, hex, URL encoding, polymorphic shellcode) hides payloads from pattern matching; understanding it lets defenders normalize traffic before inspection.

Snort is the reference open-source IDS/IPS; defenders counter evasion with full traffic normalization, target-based reassembly, decryption/TLS inspection, and tuned, updated signatures.

Last updated: June 2026

IDS vs. IPS and Detection Methods

An Intrusion Detection System (IDS) monitors traffic and raises alerts when it observes malicious activity, but it sits passively out-of-band (often on a SPAN/mirror port) and does not stop the packet. An Intrusion Prevention System (IPS) sits inline in the traffic path and can actively drop, reset, or quarantine offending flows. IDS/IPS are further classified by placement: a NIDS/NIPS is network-based (monitors a segment), while a HIDS is host-based (watches a single system's logs, files, and processes).

Detection methods CEH tests:

Method How It Works Strength / Weakness

Signature-based Matches traffic against a database of known attack patterns Accurate for known threats; blind to zero-days

Anomaly-based Flags deviations from a learned baseline of "normal" Can catch novel attacks; high false-positive rate

Stateful protocol analysis Compares observed protocol behavior to vendor profiles of legitimate use Catches protocol abuse; resource-intensive

Snort, Suricata, and Zeek are common network sensors; Snort is the canonical CEH example and uses a rule language ( alert tcp any any -> $HOME_NET 80 ... ).

Insertion and Evasion: The Core Concept

Ptacek and Newsham's foundational work defined two complementary attacks on a sensor:

Insertion — the attacker sends a packet the IDS accepts but the target rejects . The sensor's reconstructed stream now contains extra bytes the victim never processes, so a signature can be "diluted" or broken (e.g., inserting characters with a bad checksum or low TTL that die before reaching the target).

Evasion — the attacker sends a packet the IDS rejects but the target accepts . The sensor misses bytes that the victim does process, so the malicious payload never matches a signature.

Both attacks exploit the same root cause: the IDS and the end host disagree on how to interpret/reassemble the same packet stream. Whenever an IDS makes a different decision than the actual target about TTL, fragment overlap, checksum validity, or segmentation, an attacker can hide an exploit in the gap. This is why modern defenders favor target-based reassembly — the sensor models how the specific destination OS reassembles traffic — rather than a generic guess.

Fragmentation, TTL, and Encoding Techniques

Fragmentation attacks split the payload across many small IP fragments or TCP segments so no single packet matches a signature; only after reassembly is the attack visible, and if the IDS reassembles differently from the host, it misses the match. Variants include tiny fragments , overlapping fragments (offsets crafted so different OSes keep older vs. newer data), and out-of-order delivery . Nmap's -f flag and fragroute automate fragmentation.

TTL manipulation sets a Time-To-Live just high enough to reach the IDS but expire before the target (insertion) or vice-versa, desynchronizing the two views. Bad checksums are another insertion trick: the IDS may process a packet that the host silently drops.

Encoding and obfuscation defeat signature matching by transforming the payload into a form the target decodes but the IDS does not:

URL/hex/Unicode encoding of HTTP requests (e.g., %2e%2e%2f for ../ ).

Polymorphic shellcode — each instance is XOR-encoded with a unique key plus a small decoder stub, so the byte pattern is never the same.

Case manipulation, double-encoding, and tabs/whitespace in protocol commands.

Encryption/tunneling (HTTPS, SSH) so the IDS sees only ciphertext.

Defensive Countermeasures

As CEH frames it, understanding evasion is what lets a defender close the gap. Countermeasures:

• Evasion Technique Defender Countermeasure
• Fragmentation / overlap Full target-based reassembly before inspection; reject ambiguous overlaps
• TTL / bad-checksum insertion Traffic normalization (a normalizer rewrites/drops ambiguous packets)
• Encoding / Unicode obfuscation Canonicalization/normalization of payloads prior to signature match
• Encryption/tunneling TLS inspection/decryption at the perimeter; inspect decrypted traffic
• Signature gaps / zero-days Combine signature + anomaly detection ; keep rules updated
• Low-and-slow evasion Stateful tracking with long-lived flow timeouts

Additional defenses: deploy the IPS inline so it can drop rather than merely alert; tune signatures to reduce false positives (alert fatigue is itself an evasion enabler); place sensors at multiple choke points; and feed alerts into a SIEM for correlation. In Snort , defenders enable preprocessors such as frag3 (target-based IP defragmentation) and stream5 (stateful TCP reassembly) precisely to defeat fragmentation and insertion/evasion.

A further evasion class is denial-of-service against the sensor itself : flooding the IDS with crafted alerts (false positives) or with more traffic than it can inspect at line rate causes fail-open behavior, where packets pass uninspected. Defenders size sensors for peak throughput, deploy them fail-closed on critical segments, and watch for inspection-drop counters. Attackers also try session splicing — delivering the payload one byte per packet with long delays so a sensor with a short reassembly timeout discards state before the full signature forms; the fix is long-lived stateful reassembly windows.

A final point: because signature-based sensors only catch known patterns, defenders pair them with anomaly-based detection and threat-intelligence feeds, and they regularly update rule sets so newly disclosed exploits are recognized. The key exam takeaway: evasion works by desynchronizing the sensor from the target (or overwhelming it), so normalization, target-aware reassembly, updated signatures, and adequate capacity are the structural fixes.