10.3 Web Reconnaissance, Content Discovery & Threat Modelling

CREST Practitioner Security Analyst CPSA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Soft Skills, Legal Framework & Assessment Management

1.1 The Penetration Testing Engagement Lifecycle & Scoping 1.2 Legal Framework: Computer Misuse Act & Compliance 1.3 Understanding, Explaining & Managing Technical Risk 1.4 Record Keeping, Interim Escalation & Reporting Standards

2 Chapter 2: Core Networking: IP Protocols & Network Architectures

2.1 IPv4 and IPv6 Addressing, Subnetting & Header Architecture 2.2 TCP and UDP Transport Protocols & Handshake Mechanics 2.3 ICMP Protocol, Network Diagnostics & Error Messages 2.4 Network Topologies, Switched Media & VLAN Segmentation 2.5 Network Media, Ethernet Standards & Physical Topologies

3 Chapter 3: Cryptography & Secure Communication Protocols

3.1 Symmetric & Asymmetric Cryptographic Algorithms 3.2 Cryptographic Hash Functions, MACs & Digital Signatures 3.3 Secure Protocols: SSL/TLS, IPsec & SSH

4 Chapter 4: Background Information Gathering & OSINT Reconnaissance

4.1 WHOIS Records, Domain Registrars & Autonomous Systems 4.2 DNS Infrastructure, Record Types & Zone Transfers 4.3 Search Engine Reconnaissance, Google Dorking & Information Leakage 4.4 Newsgroups, Mailing Lists & Public Archive Reconnaissance

5 Chapter 5: Networking Equipment & Infrastructure Protocols

5.1 Router & Switch Management: SNMP, SSH & Authentication 5.2 Layer 2 Protocols: ARP, CDP, STP, VTP & Redundancy 5.3 Core Network Services: DHCP, NTP & TFTP 5.4 Wireless Security (802.11) & VoIP Assessment 5.5 Out-of-Band Management, Console Access & Cisco Reverse Telnet

6 Chapter 6: Network Mapping, Nmap Techniques & Target Identification

6.1 Host Discovery Techniques & Ping Sweep Methodologies 6.2 Port Scanning Mechanics: TCP, UDP & Stealth Scans 6.3 OS Fingerprinting, Service Detection & Timing Templates 6.4 Firewall and IDS/IPS Evasion Techniques 6.5 Passive OS & Service Fingerprinting 6.6 Network Access Control Analysis & Firewall Rule Base Review

7 Chapter 7: Vulnerability Identification & Tool Output Analysis

7.1 Vulnerability Scanning Principles & Methodology 7.2 CVE Referencing, CVSS Scoring & Risk Prioritisation 7.3 Banner Grabbing, Service Verification & Manual Triage 7.4 Host Audit Techniques: Processes, Sockets, Patch Levels & Interesting Files

8 Chapter 8: Microsoft Windows Security Assessment & Active Directory

8.1 Active Directory Architecture & Domain Reconnaissance 8.2 Windows User & Group Enumeration via SMB & RPC 8.3 Windows Authentication: NTLM, Kerberos & Password Security 8.4 Windows Patch Management, Local Security & Common Services 8.5 Windows File System, Share & Registry Permissions 8.6 Desktop Lockdown, Citrix Breakout & Common Windows Applications

9 Chapter 9: Unix & Linux Security Assessment & Legacy Services

9.1 Linux/Unix Security Architecture, Permissions & Filesystem 9.2 Unix User Enumeration & Service Discovery 9.3 Network File System (NFS) & RPC Services Assessment 9.4 Legacy Unix Services: R-Services, X11, FTP & Modern SSH 9.5 Unix Remote Information Services: rusers, rwho, finger & systat 9.6 Unix Vulnerability Exploitation & Post-Exploitation Workflow

10 Chapter 10: Web Architectures, Technologies & Testing Methodologies

10.1 Web Server Architecture & the HTTP/HTTPS Protocol 10.2 Web Application Components, Frameworks & State Management 10.3 Web Reconnaissance, Content Discovery & Threat Modelling 10.4 Testing Authentication, Authorization & Session Management 10.5 Web Service Protocols: SOAP, REST & XML Messaging 10.6 Web APIs, Server Extensions & Client-Side Sub-Components

11 Chapter 11: Web Testing Techniques & OWASP Top Vulnerabilities

11.1 Input Validation Vulnerabilities & Parameter Manipulation 11.2 Cross-Site Scripting (XSS): Types, Impact & Defenses 11.3 Cross-Site Request Forgery (CSRF) & Client-Side Attacks 11.4 Command Injection, File Upload Vulnerabilities & API Flaws 11.5 Information Disclosure: Error Messages & Mark-up Leakage 11.6 Injection Beyond SQL: LDAP, XPath, Code & Template Injection 11.7 Source Code Review & Encoded/Cryptographic Value Analysis

12 Chapter 12: Database Security Assessment & SQL Injection

12.1 Database Architectures, Default Ports & Authentication 12.2 SQL Injection (SQLi) Foundations & In-Band Techniques 12.3 Blind SQL Injection & Database Enumeration 12.4 Database Attack Vectors, Stored Procedures & Hardening

10.3 Web Reconnaissance, Content Discovery & Threat Modelling

Key Takeaways Explain This Topic AI

Web application reconnaissance combines passive crawling of DOM elements, hyperlinks, and client-side JavaScript bundles with active forced browsing to discover hidden endpoints and administrative interfaces.

Analyzing JavaScript source maps (.js.map) frequently permits penetration testers to reconstruct original unminified development code, exposing private backend API routes, internal comments, and embedded credentials.

Extension fuzzing against web roots targets temporary, backup, and environment artifacts (.bak, .old, .swp, .env, web.config) that inadvertently leak sensitive source code and database connection strings.

Public informational files such as robots.txt, sitemap.xml, and security.txt disclose sensitive directory paths that administrators intend to hide from search engine indexing.

The STRIDE threat model categorizes web vulnerabilities into Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege across Data Flow Diagram trust boundaries.

Last updated: September 2026

10.3 Web Reconnaissance, Content Discovery & Threat Modelling

A comprehensive web application penetration test begins with structured reconnaissance. Before attempting targeted exploitation, an analyst must map the target's attack surface, enumerate hidden content, identify the underlying technology stack, and construct an architectural threat model. Flaws in web applications frequently originate not only from coding errors in published pages, but from unlinked administrative interfaces, forgotten backup files, exposed configuration files, and unvalidated trust boundaries.

Web Application Reconnaissance & Spidering

Reconnaissance against web applications combines passive discovery (analyzing publicly accessible and linked resources) with active forced browsing (probing for unlinked assets using dictionary brute-forcing). +-----------------------------------------------------------------------------+

| WEB APPLICATION DISCOVERY METHODOLOGY |
+-----------------------------------------------------------------------------+
| 1. PASSIVE SPIDERING / CRAWLING |
| - Traverse visible HTML DOM links (<a href>, <form action>, <iframe>) |
| - Extract endpoints from client-side JavaScript files (.js) |
| - Retrieve JavaScript Source Maps (.js.map) to recover original source |
| - Inspect robots.txt, sitemap.xml, /.well-known/, HTML comments |
+-----------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------+
| 2. ACTIVE FORCED BROWSING & FUZZING |
| - Dictionary brute-forcing of directories and filenames |
| - Tools: ffuf, gobuster, dirsearch, dirb, wfuzz |
| - Targeted extension fuzzing (.php, .aspx, .json, .bak, .old, .env) |
| - Analyze response status codes (200, 301, 302, 401, 403 vs 404) |
+-----------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------+
| 3. TECHNOLOGY FINGERPRINTING & THREAT MODELLING |
| - HTTP Response Headers (Server, X-Powered-By, X-AspNet-Version) |
| - Cookie naming conventions (PHPSESSID, JSESSIONID, ASP.NET_SessionId) |
| - CMS indicators (WordPress /wp-admin/, Drupal core, Joomla paths) |
| - Map components to STRIDE categories across trust boundaries |
+-----------------------------------------------------------------------------+
1. Passive Spidering and JavaScript Analysis

Automated Crawling: Web crawlers (such as the Burp Suite Spider/Crawler or OWASP ZAP Spider) parse HTML responses, recursively populating the target site tree by extracting URLs from <a href="..."> , <form action="..."> , <script src="..."> , <link href="..."> , and <img src="..."> tags.

Analyzing Single Page Applications (SPAs): Modern front-end applications built with frameworks like React, Angular, or Vue bundle their routing logic into client-side JavaScript packages. Analysts inspect these .js files using regular expressions to extract unlinked API routes, development endpoints, and internal microservice addresses.

JavaScript Source Maps ( .map files): During production builds, developers often minify and bundle code. If they inadvertently deploy .map files (e.g., main.bundle.js.map ) to the web server, an analyst can use browser developer tools or utilities like sourcemapper to completely de-minify and reconstruct the developer's original raw source code—including developer comments, internal variable names, and unreleased feature endpoints.

2. Informational Files and Metadata

Web administrators deploy standard files intended for crawlers, search engines, or security researchers that inadvertently leak sensitive operational paths:

Informational Asset Standard Location Purpose Reconnaissance Value

robots.txt /robots.txt Directs search engine crawlers (Googlebot) on which paths to avoid indexing ( Disallow: ). Frequently discloses high-value sensitive directories (e.g., /admin/ , /portal-backup/ , /staging/ , /api/private/ ).

sitemap.xml /sitemap.xml XML map of site structure for search indexing. Lists complete inventories of public URLs, including recently added or obscure application endpoints.

security.txt /.well-known/security.txt or /security.txt RFC 9116 security contact information and disclosure policy. Discloses authorized contact emails, public PGP encryption keys, hiring domains, and testing policy scopes.

/.well-known/ /.well-known/ (RFC 8615) Standardized directory for site metadata. Exposes OAuth/OpenID discovery endpoints ( openid-configuration ), ACME challenges ( acme-challenge ), and asset links.

HTML Comments Throughout page source Developer notes (  ). Leaks developer usernames, internal IP addresses, testing notes, disabled input parameters, and backend system names.

Forced Browsing & Content Discovery

Because crawlers only find resources linked within HTML and JavaScript, security analysts use forced browsing (content discovery) to identify unlinked files, administrative interfaces, and backup copies. # Directory and endpoint fuzzing using ffuf

ffuf -u https://target.lan/FUZZ -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt \
mc 200,301,302,401,403 -c -v

# Targeted extension fuzzing with Gobuster
gobuster dir -u https://target.lan -w /usr/share/seclists/Discovery/Web-Content/common.txt \
x php,aspx,jsp,html,txt,json,env,bak,old,zip -t 40
Interpreting Response Codes During Discovery

200 OK : File or directory exists and is publicly accessible.

301 / 302 Redirect : Resource exists; check the Location response header. If it redirects to /login or appends a trailing slash ( /admin -> /admin/ ), the directory is verified.

401 Unauthorized : Resource exists and is protected by HTTP Basic/Digest/Bearer authentication.

403 Forbidden : Resource exists, but access is restricted by server configuration or IP whitelisting. This confirms the presence of an administrative portal or restricted endpoint.

404 Not Found : Resource does not exist (ensure the server does not return custom "soft 404" pages with HTTP 200 codes).

Targeted Extension Fuzzing & Backup File Enumeration

When developers edit or back up files directly on production servers, text editors and scripts often create residual artifacts accessible to any user: +-----------------------------------------------------------------------------+

| HIGH-VALUE BACKUP & CONFIG ARTIFACTS |
+-----------------------------------------------------------------------------+
| Category | File Extensions / Filenames |
+-------------------+---------------------------------------------------------+
| Web Frameworks | .php, .asp, .aspx, .jsp, .do, .action, .py, .js |
| Backup Files | .bak, .old, .orig, .temp, .backup, .save, ~ |
| Editor Swaps | .swp, .swo (Vim swap files), .#filename (Emacs) |
| Configuration | .env, web.config, httpd.conf, settings.py, database.yml |
| Archives & Dumps | .zip, .tar.gz, .tar.bz2, .sql, .dump, .7z |
| Version Control | /.git/, /.git/HEAD, /.svn/, /.svn/entries, /.hg/ |
+-----------------------------------------------------------------------------+

Critical Vulnerability Pattern: If an Apache web server is configured to execute .php files through its PHP interpreter, requesting index.php executes the code on the server. However, if a developer leaves a backup copy named index.php.bak , index.php~ , or index.php.old , Apache does not match the .php extension handler and instead serves the raw file as plaintext , exposing all underlying source code, database passwords, and API keys.

Identifying Web Technologies & Frameworks

Accurate fingerprinting allows testers to focus on known Common Vulnerabilities and Exposures (CVEs) and architecture-specific weaknesses. +-----------------------------------------------------------------------------+

| TECHNOLOGY FINGERPRINTING MATRIX |
+-----------------------------------------------------------------------------+
| Indicator Category | Disclosed Value / Name | Implied Technology |
+--------------------+-------------------------------+------------------------+
| Server Header | Apache/2.4.52 (Ubuntu) | Apache on Linux |
| | nginx/1.18.0 | Nginx Web Server |
| | Microsoft-IIS/10.0 | Windows Server IIS |
+--------------------+-------------------------------+------------------------+
| Application Header | X-Powered-By: Express | Node.js Express |
| | X-Powered-By: PHP/7.4.30 | PHP Runtime |
| | X-AspNet-Version: 4.0.30319 | ASP.NET Framework |
+--------------------+-------------------------------+------------------------+
| Cookie Identifier | PHPSESSID | PHP Application |
| | JSESSIONID | Java Servlet Container |
| | ASP.NET_SessionId | Microsoft ASP.NET |
| | connect.sid | Node.js Express |
| | csrftoken / sessionid | Python Django |
+--------------------+-------------------------------+------------------------+
| Distinct CMS Paths | /wp-admin/, /wp-includes/ | WordPress CMS |
| | /administrator/ | Joomla CMS |
| | /core/, CHANGELOG.txt | Drupal CMS |
+-----------------------------------------------------------------------------+
Automated Fingerprinting Tools

whatweb : Command-line reconnaissance utility that identifies content management systems, blogging platforms, JavaScript libraries, web servers, and embedded devices. whatweb https://target.lan -v --color=never

Wappalyzer : Browser extension and CLI package that inspects HTML markup, meta tags, script URLs, JavaScript global variables ( window.React , window.jQuery ), and HTTP headers to generate a detailed technology profile.

CMS-Specific Scanners: Utilities like wpscan (WordPress) audit plugins, themes, and users via REST API endpoints ( /wp-json/wp/v2/users ).

Threat Modelling for Web Applications: The STRIDE Model

Threat modelling provides a systematic methodology for evaluating application attack surfaces, mapping data flows, and identifying security weaknesses before testing or development. +-----------------------------------------------------------------------------+

| STRIDE THREAT MODELLING MODEL |
+-----------------------------------------------------------------------------+
| Threat Category | Violated Security Property | Web Attack Example |
+---------------------------+----------------------------+--------------------+
| S - Spoofing | Authenticity | Session Hijacking, |
| | | Credential Replay |
| T - Tampering | Integrity | Parameter Tampering|
| | | SQL Injection |
| R - Repudiation | Non-repudiation | Action Denied; |
| | | Insufficient Logs |
| I - Information Disclosure| Confidentiality | Verbose Error Dumps|
| | | Directory Listing |
| D - Denial of Service | Availability | ReDoS Regex Bomb, |
| | | Resource Exhaustion|
| E - Elevation of Privilege| Authorization | IDOR / BOLA, |
| | | Admin Role Bypass |
+-----------------------------------------------------------------------------+
Data Flow Diagrams (DFDs) and Trust Boundaries

Threat modelling relies on Data Flow Diagrams (DFDs) to map how data moves through an application:

External Entities: Users, third-party payment gateways, external APIs.

Processes: Code modules that compute, transform, or route data (e.g., authentication controller, payment handler).

Data Stores: Locations where data resides (e.g., SQL databases, log files, session caches).

Data Flows: Directed arrows showing the movement of data between entities, processes, and stores.

Trust Boundaries: The most critical element in threat modelling. A trust boundary represents the perimeter where data transitions from one trust level to another (e.g., from an untrusted web browser to a DMZ reverse proxy, or from an application server to an internal database). Any data crossing a trust boundary must be treated as untrusted and subjected to strict authentication, authorization, and input validation .