ExamFinch.com My Study

Study guide

2.2 TCP and UDP Transport Protocols & Handshake Mechanics

Practice this exam

CREST Practitioner Security Analyst CPSA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Soft Skills, Legal Framework & Assessment Management

1.1 The Penetration Testing Engagement Lifecycle & Scoping 1.2 Legal Framework: Computer Misuse Act & Compliance 1.3 Understanding, Explaining & Managing Technical Risk 1.4 Record Keeping, Interim Escalation & Reporting Standards

2 Chapter 2: Core Networking: IP Protocols & Network Architectures

2.1 IPv4 and IPv6 Addressing, Subnetting & Header Architecture 2.2 TCP and UDP Transport Protocols & Handshake Mechanics 2.3 ICMP Protocol, Network Diagnostics & Error Messages 2.4 Network Topologies, Switched Media & VLAN Segmentation 2.5 Network Media, Ethernet Standards & Physical Topologies

3 Chapter 3: Cryptography & Secure Communication Protocols

3.1 Symmetric & Asymmetric Cryptographic Algorithms 3.2 Cryptographic Hash Functions, MACs & Digital Signatures 3.3 Secure Protocols: SSL/TLS, IPsec & SSH

4 Chapter 4: Background Information Gathering & OSINT Reconnaissance

4.1 WHOIS Records, Domain Registrars & Autonomous Systems 4.2 DNS Infrastructure, Record Types & Zone Transfers 4.3 Search Engine Reconnaissance, Google Dorking & Information Leakage 4.4 Newsgroups, Mailing Lists & Public Archive Reconnaissance

5 Chapter 5: Networking Equipment & Infrastructure Protocols

5.1 Router & Switch Management: SNMP, SSH & Authentication 5.2 Layer 2 Protocols: ARP, CDP, STP, VTP & Redundancy 5.3 Core Network Services: DHCP, NTP & TFTP 5.4 Wireless Security (802.11) & VoIP Assessment 5.5 Out-of-Band Management, Console Access & Cisco Reverse Telnet

6 Chapter 6: Network Mapping, Nmap Techniques & Target Identification

6.1 Host Discovery Techniques & Ping Sweep Methodologies 6.2 Port Scanning Mechanics: TCP, UDP & Stealth Scans 6.3 OS Fingerprinting, Service Detection & Timing Templates 6.4 Firewall and IDS/IPS Evasion Techniques 6.5 Passive OS & Service Fingerprinting 6.6 Network Access Control Analysis & Firewall Rule Base Review

7 Chapter 7: Vulnerability Identification & Tool Output Analysis

7.1 Vulnerability Scanning Principles & Methodology 7.2 CVE Referencing, CVSS Scoring & Risk Prioritisation 7.3 Banner Grabbing, Service Verification & Manual Triage 7.4 Host Audit Techniques: Processes, Sockets, Patch Levels & Interesting Files

8 Chapter 8: Microsoft Windows Security Assessment & Active Directory

8.1 Active Directory Architecture & Domain Reconnaissance 8.2 Windows User & Group Enumeration via SMB & RPC 8.3 Windows Authentication: NTLM, Kerberos & Password Security 8.4 Windows Patch Management, Local Security & Common Services 8.5 Windows File System, Share & Registry Permissions 8.6 Desktop Lockdown, Citrix Breakout & Common Windows Applications

9 Chapter 9: Unix & Linux Security Assessment & Legacy Services

9.1 Linux/Unix Security Architecture, Permissions & Filesystem 9.2 Unix User Enumeration & Service Discovery 9.3 Network File System (NFS) & RPC Services Assessment 9.4 Legacy Unix Services: R-Services, X11, FTP & Modern SSH 9.5 Unix Remote Information Services: rusers, rwho, finger & systat 9.6 Unix Vulnerability Exploitation & Post-Exploitation Workflow

10 Chapter 10: Web Architectures, Technologies & Testing Methodologies

10.1 Web Server Architecture & the HTTP/HTTPS Protocol 10.2 Web Application Components, Frameworks & State Management 10.3 Web Reconnaissance, Content Discovery & Threat Modelling 10.4 Testing Authentication, Authorization & Session Management 10.5 Web Service Protocols: SOAP, REST & XML Messaging 10.6 Web APIs, Server Extensions & Client-Side Sub-Components

11 Chapter 11: Web Testing Techniques & OWASP Top Vulnerabilities

11.1 Input Validation Vulnerabilities & Parameter Manipulation 11.2 Cross-Site Scripting (XSS): Types, Impact & Defenses 11.3 Cross-Site Request Forgery (CSRF) & Client-Side Attacks 11.4 Command Injection, File Upload Vulnerabilities & API Flaws 11.5 Information Disclosure: Error Messages & Mark-up Leakage 11.6 Injection Beyond SQL: LDAP, XPath, Code & Template Injection 11.7 Source Code Review & Encoded/Cryptographic Value Analysis

12 Chapter 12: Database Security Assessment & SQL Injection

12.1 Database Architectures, Default Ports & Authentication 12.2 SQL Injection (SQLi) Foundations & In-Band Techniques 12.3 Blind SQL Injection & Database Enumeration 12.4 Database Attack Vectors, Stored Procedures & Hardening

2.2 TCP and UDP Transport Protocols & Handshake Mechanics

Key Takeaways Explain This Topic AI

TCP is a connection-oriented, full-duplex protocol providing reliable stream transport via sequence numbers, positive acknowledgments, and sliding window flow control.

The TCP 3-way handshake (SYN, SYN-ACK, ACK) synchronizes Initial Sequence Numbers (ISNs) and establishes Transmission Control Blocks (TCBs) on participating endpoints.

The TCP header contains 9 control flags (URG, ACK, PSH, RST, SYN, FIN, ECE, CWR, NS) that govern state transitions and provide the underlying mechanics for port scanning techniques.

UDP provides lightweight, connectionless datagram transport with a minimal 8-byte fixed header, lacking retransmission and flow control, which exposes it to reflection and amplification attacks.

Last updated: September 2026

2.2 TCP and UDP Transport Protocols & Handshake Mechanics

At Layer 4 (Transport Layer) of the OSI model, protocols facilitate end-to-end communication services for applications. While Layer 3 (IP) routes datagrams from host to host, Layer 4 is responsible for process-to-process delivery through port multiplexing . For the CREST CPSA examination, candidates must master the internal mechanics, flag behaviors, state transitions, and diagnostic signatures of the Transmission Control Protocol (TCP) and User Datagram Protocol (UDP).

1. Transport Layer Port Architecture

Both TCP and UDP utilize 16-bit port numbers (values 0 to 65,535) to identify specific application processes on an endpoint. The combination of an IP address and a port number defines a socket (e.g., 192.168.1.50:443 ).

Port ranges are divided by IANA into three distinct tiers:

Well-Known Ports (0 – 1023): Assigned to core system services and protocols (e.g., HTTP 80, HTTPS 443, SSH 22, DNS 53). On Unix-like systems, binding to a well-known port historically requires superuser ( root ) privileges.

Registered Ports (1024 – 49151): Assigned to user processes, third-party applications, and databases (e.g., Microsoft SQL Server 1433, MySQL 3306, RDP 3389).

Dynamic / Private / Ephemeral Ports (49152 – 65535): Automatically assigned by the operating system kernel to client-side outbound connections for the duration of a session.

2. Transmission Control Protocol (TCP) Architecture

Defined in RFC 793 (and updated by RFC 9293 ), TCP is a connection-oriented, reliable, full-duplex byte-stream protocol . Before data can be transmitted, a formal logical connection must be established between client and server.

Reliability Mechanics

Ordered Delivery: TCP numbers every byte sent using 32-bit Sequence Numbers, allowing the receiver to reassemble out-of-order packets into a contiguous stream.

Positive Acknowledgment with Retransmission (PAR): The receiver returns Acknowledgment Numbers confirming receipt. If the sender does not receive an acknowledgment before a Retransmission Timeout (RTO) expires, the data is retransmitted.

Flow Control: TCP uses a Sliding Window mechanism where the receiving host advertises its available buffer space (Window Size) to prevent buffer overrun.

TCP Header Structure

The TCP header has a minimum length of 20 bytes and can expand up to 60 bytes when options are included: 0 1 2 3

  • 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Source Port | Destination Port |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Sequence Number |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Acknowledgment Number |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Data | |C|E|U|A|P|R|S|F| |
  • | Offset| Rsrvd |W|C|R|C|S|S|Y|I| Window Size |
  • | | |R|E|G|K|H|T|N|N| |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Checksum | Urgent Pointer |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Options | Padding |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • Field Breakdown & Flag Analysis

Source Port & Destination Port (16 bits each): Sockets for sender and receiver.

Sequence Number (32 bits): In a SYN packet, this represents the Initial Sequence Number (ISN) . In regular data transfer, it represents the sequence number of the first data byte in the segment.

Acknowledgment Number (32 bits): If the ACK flag is set, this field contains the next sequence number the receiver expects to receive. Acknowledgment is cumulative ( A c k = S e q r e c e i v e d + L e n g t h d a t a + 1 Ack = Seq_{received} + Length_{data} + 1 A c k = S e q rece i v e d ​ + L e n g t h d a t a ​ + 1 ).

Data Offset (4 bits): Header length expressed in 32-bit (4-byte) words . Minimum value is 5 ( 5 × 4 = 20 5 \times 4 = 20 5 × 4 = 20 bytes); maximum is 15 ( 15 × 4 = 60 15 \times 4 = 60 15 × 4 = 60 bytes).

Reserved (3 bits): Reserved for future use (must be 0).

Control Flags (9 bits):

CWR (Congestion Window Reduced): Sender reduced its transmission rate (RFC 3168).

ECE (ECN-Echo): Indicates network congestion encountered (RFC 3168).

URG (Urgent): Indicates the Urgent Pointer field is significant.

ACK (Acknowledgment): Indicates the Acknowledgment Number field is valid. Once established, all TCP segments carry ACK.

PSH (Push): Instructs the receiver to push buffered data immediately up to the application layer rather than waiting for buffers to fill.

RST (Reset): Aborts an existing connection or rejects an incoming connection request to a closed port.

SYN (Synchronize): Synchronizes sequence numbers to initiate a connection.

FIN (Finish): Signals that the sender has finished transmitting data and requests graceful teardown.

NS (Nonce Sum): Protects against accidental or malicious concealment of marked packets (RFC 3540).

Window Size (16 bits): Number of data octets the sender is willing to accept without further acknowledgment.

Checksum (16 bits): Mandatory error-checking field covering the TCP header, TCP payload, and a 12-byte IP Pseudo-Header (Source IP, Destination IP, Reserved, Protocol 6, TCP Segment Length).

Urgent Pointer (16 bits): Byte offset from the sequence number pointing to the end of urgent data.

TCP Options (Variable): Common options negotiated during the handshake:

Maximum Segment Size (MSS, Kind 2): Maximum data payload a host can receive in a single segment (typically 1460 bytes for a 1500-byte Ethernet MTU).

Window Scale (Kind 3, RFC 7323): Multiplier allowing the 16-bit Window field to scale up to 1 GB 1\text{ GB} 1 GB for high-bandwidth/high-latency networks.

Selective Acknowledgment (SACK Permitted, Kind 4, RFC 2018): Enables receiver to acknowledge discontinuous blocks of received data.

Timestamps (Kind 8, RFC 7323): Used for Round Trip Time Measurement (RTTM) and Protection Against Wrapped Sequences (PAWS). Can leak remote system uptime during reconnaissance.

Initial Sequence Number (ISN) Generation & Security

In early TCP/IP implementations, the ISN was incremented predictably using a fixed counter. In 1995, Kevin Mitnick famously exploited this via TCP Blind Spoofing / Sequence Number Prediction : by flooding a trusted server with SYN packets to silence it, an attacker could spoof the trusted server's IP address against a target host, blindly predict the target's ISN response, and transmit a forged ACK to establish a one-way authenticated rsh/rlogin session.

Modern operating systems implement RFC 1948 and cryptographically secure pseudorandom number generators (PRNGs) to randomize ISNs, preventing blind sequence number prediction.

  • 3. TCP Handshake Mechanics & State Machine Client (Active Open) Server (Passive Open)
  • [CLOSED] [LISTEN]
  • | |
  • | -------------- 1. SYN (Seq=x) ------------------> | Server creates TCB
  • [SYN_SENT] [SYN_RECEIVED]
  • | |
  • | <--------- 2. SYN-ACK (Seq=y, Ack=x+1) ---------- |
  • [ESTABLISHED] |
  • | |
  • | ------------- 3. ACK (Seq=x+1, Ack=y+1) --------> |
  • | [ESTABLISHED]
  • | <=========== Full-Duplex Data Flow =============> |
  • The 3-Way Handshake Step-by-Step

SYN (Step 1): Client sends a segment with SYN=1 , a randomly generated client sequence number Seq = x , and proposed TCP Options (MSS, SACK, Window Scale). Client enters SYN_SENT .

SYN-ACK (Step 2): Server allocates a Transmission Control Block (TCB) memory buffer, selects its own random sequence number Seq = y , acknowledges the client's sequence number with Ack = x + 1 , and sets SYN=1, ACK=1 . Server enters SYN_RECEIVED .

ACK (Step 3): Client sends ACK=1 with Seq = x + 1 and Ack = y + 1 . Both endpoints enter ESTABLISHED .

SYN Flood Attack & SYN Cookies

Because the server allocates memory for the TCB upon receiving Step 1 (SYN) before the client confirms Step 3 (ACK), an attacker can flood thousands of SYN packets with spoofed source IPs. The server fills its connection backlog queue with half-open connections in the SYN_RECEIVED state, exhausting system memory and causing a Denial of Service (DoS).

Defense - SYN Cookies (RFC 4987): When the backlog queue fills, the server stops allocating TCB memory. Instead, it encodes the connection state, MSS, and timestamp cryptographically into the server's ISN ( y y y ). Only when the client returns a valid ACK ( y + 1 y + 1 y + 1 ) does the server reconstruct the TCB from the acknowledgment number.

Connection Teardown: 4-Way Handshake & RST Teardown

TCP connections terminate via a graceful 4-way handshake because TCP is full-duplex and each direction must close independently: Active Closer Passive Closer

  • [ESTABLISHED] [ESTABLISHED]
  • | |
  • | --------------- 1. FIN (Seq=u) -----------------> | Enters [CLOSE_WAIT]
  • [FIN_WAIT_1] |
  • | <-------------- 2. ACK (Ack=u+1) ---------------- | Application notified
  • [FIN_WAIT_2] |
  • | | Sends remaining data
  • | <-------------- 3. FIN (Seq=v) ------------------ | Enters [LAST_ACK]
  • | |
  • | --------------- 4. ACK (Ack=v+1) ---------------> |
  • [TIME_WAIT] [CLOSED]
  • | (Wait 2 * MSL)
  • [CLOSED]

TIME_WAIT State: The active closer remains in TIME_WAIT for 2 Maximum Segment Lifetimes (2 * MSL) , typically 60 to 120 seconds. This guarantees that the final ACK was delivered (if lost, the peer retransmits FIN) and ensures that lingering duplicate packets from this connection expire before the socket is reused.

RST Abortive Teardown: A connection can be instantaneously aborted by transmitting a segment with RST=1 . Any buffered in-flight data is discarded immediately without waiting for acknowledgments.

4. User Datagram Protocol (UDP) Architecture

Defined in RFC 768 , UDP is a connectionless, lightweight, unreliable transport protocol . It operates on a best-effort delivery model with no handshakes, no acknowledgments, no sequence ordering, and no flow or congestion control.

  • Fixed 8-Byte Header Structure 0 1 2 3
  • 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Source Port | Destination Port |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Length | Checksum |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • | Payload |
  • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Source Port (16 bits): Identifies sending process; optional in IPv4 (set to 0 if unused).

Destination Port (16 bits): Target application port.

Length (16 bits): Total length of UDP header and payload in bytes (minimum value is 8 ).

Checksum (16 bits): Covers UDP header, payload, and the IP pseudo-header. Optional in IPv4, but strictly mandatory in IPv6 .

Common UDP Services Tested on CPSA

Port Service Penetration Testing / Vulnerability Context

53 DNS UDP queries under 512 bytes; cache poisoning, DNS tunneling, amplification.

67 / 68 DHCP BootP/DHCP broadcast; rogue DHCP servers, DHCP exhaustion attacks.

69 TFTP Trivial FTP; no authentication, cleartext config file downloads (Cisco configs).

123 NTP Network Time Protocol; vulnerable to monlist reflection attacks, clock skewing.

161 / 162 SNMP Simple Network Management Protocol; default community strings ( public / private ).

514 Syslog Unauthenticated, unencrypted log transmission; trivial log spoofing.

UDP Amplification & Reflection Attacks

Because UDP is connectionless and does not perform a handshake, an attacker can forge (spoof) the packet's Source IP address to match a victim's IP. The attacker transmits small requests to publicly exposed third-party UDP servers (reflectors) that generate massive responses directed at the victim. Amplification factors:

DNS: 20x to 50x amplification using ANY or large EDNS0 queries.

NTP: Over 500x amplification using the legacy monlist diagnostic command.

5. Transport Layer Port Scanning Mechanics

Understanding how transport layer stacks respond to unexpected or malformed probes is the foundation of network reconnaissance with tools like Nmap.

TCP Port Scanning Methodologies

TCP Connect Scan ( nmap -sT ):

Uses the OS connect() API to complete the full 3-way handshake (SYN -> SYN-ACK -> ACK).

Advantage: Does not require raw socket / root privileges.

Disadvantage: Easily detected; leaves explicit connection log entries in application event logs.

TCP SYN Scan / Stealth Scan ( nmap -sS ):

The scanner sends a SYN packet.

Open Port: Target responds with SYN-ACK . Scanner immediately returns a RST to tear down the connection before the handshake completes. The application layer never sees an established connection.

Closed Port: Target responds with RST-ACK .

Filtered Port: No response received (packet dropped by firewall) or target returns ICMP Type 3 error.

Advantage: Fast, stealthy; requires root/raw socket access.

TCP Inverse & Malformed Scans (RFC 793 Compliant Scans):

NULL Scan ( -sN ): No flags set ( 0x00 ).

FIN Scan ( -sF ): Only FIN set.

XMAS Scan ( -sX ): FIN , PSH , and URG set simultaneously.

Rule: According to RFC 793, if a closed port receives any packet lacking SYN, RST, or ACK, it must respond with a RST . If the port is open, the packet is silently discarded.

OS Fingerprint Limitation: Microsoft Windows, Cisco, and BSDi operating systems violate RFC 793 and send RST for all ports regardless of state, rendering inverse scans ineffective against Windows environments.

UDP Port Scanning Mechanics ( nmap -sU )

Scanning UDP is notoriously slow and difficult due to the absence of handshakes:

Probe: Scanner sends an empty or protocol-specific UDP datagram to the target port.

Response Signatures:

If the target returns a UDP data response → \rightarrow → Port is OPEN .

If the target returns an ICMP Type 3 Code 3 (Port Unreachable) message → \rightarrow → Port is CLOSED .

If no response is received (or ICMP Type 3 Codes 1, 2, 9, 10, or 13) → \rightarrow → Port is classified as OPEN|FILTERED (the packet could be open and silently accepted, or dropped by a firewall).

Kernel Rate Limiting: Operating systems enforce RFC 1812 rate limiting on ICMP error generation (e.g., Linux limits ICMP Destination Unreachable responses to 1 per second). Consequently, a comprehensive 65,535-port UDP scan can require hours to complete.