Study guide
6.3 Extracting Personal Data from Publicly Available Sources
Certified Information Privacy Technologist CIPT
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CIPT Exam Overview and Study Plan
1.1 CIPT Exam Facts, Blueprint, and Study Plan
2 Chapter 2: The Privacy Technologist's Legal and Procedural Role
2.1 Privacy Function Roles: DPO, Data Owner, Steward, Custodian, Legal, and Security 2.2 Translating Legal and Regulatory Requirements into Technical Controls 2.3 Notices, Policies, and Oversight of Privacy Operations, Audits, and Third Parties
3 Chapter 3: Technical Responsibilities: Risk Concepts and Incident Response
3.1 Risk Concepts: Threat, Vulnerability, Attack, Exploit, and Privacy Risk 3.2 Technical Support for Privacy Incident and Breach Response
4 Chapter 4: Privacy Risk Models, Frameworks, and Threat Models
4.1 Privacy Risk Models and Frameworks: Contextual Integrity, Calo, Solove, FIPPs, OECD, NIST, and FAIR 4.2 Privacy Threat Models: LINDDUN and MITRE PANOPTIC
5 Chapter 5: Data Ethics and Automated Decision-Making
5.1 Legal Versus Ethical Processing and the Social Impact of Privacy-Affecting Designs 5.2 Identifying and Minimizing Bias in Automated Decision-Making
6 Chapter 6: Minimizing Privacy Risk During Data Collection
6.1 Notice, Consent Management, and Choice Architecture 6.2 Managing Privacy Risk in Automatic Collection and Telemetry 6.3 Extracting Personal Data from Publicly Available Sources 6.4 Data Retention, Destruction, and Cryptographic Erasure 6.5 Data Minimization Techniques 6.6 Implementing Data Processing Segregation
7 Chapter 7: Minimizing Privacy Risk During Data Use: Aggregation, PETs, and Secondary Use
7.1 Minimizing Privacy Risk from Aggregation and Inference 7.2 De-identification, Pseudonymization, and Tokenization 7.3 Microdata Protection: k-Anonymity, l-Diversity, and t-Closeness 7.4 Advanced PETs: Differential Privacy, Secure Multi-Party Computation, and Homomorphic Encryption 7.5 Secondary Use, Function Creep, and Purpose Limitation Controls
8 Chapter 8: Minimizing Privacy Risk During Dissemination
8.1 Disclosure and Accessibility Controls: APIs, Egress, and Clean Rooms 8.2 Countering Distortion, Exposure, Breach of Confidentiality, Blackmail, and Appropriation 8.3 Defense in Depth: Identity, Access Management, and Authentication
9 Chapter 9: Intrusion, Decisional Interference, and Dark Patterns
9.1 Minimizing Intrusion and Decisional Interference: Behavioral Advertising, Profiling, Cyberbullying, and Social Engineering 9.2 Dark Patterns and Deceptive Design
10 Chapter 10: Privacy Risks Related to Software Security
10.1 Detecting and Fixing Software Privacy Vulnerabilities 10.2 Intrusion Detection and Prevention, and Privacy in Change Management 10.3 Service Providers, Third-Party SDKs, and Supply Chain Risk
11 Chapter 11: Tracking and Surveillance Technologies
11.1 Privacy Risks and Impacts of E-Commerce 11.2 Audio and Video Surveillance, Wearables, and IoT 11.3 Privacy Issues Around Biometrics: Facial, Voice, Fingerprint, and DNA Data 11.4 Privacy Risks and Impacts of Location Tracking 11.5 Internet Monitoring and Web Tracking: Cookies, Fingerprinting, and GPC
12 Chapter 12: Workplace Technologies: AI, Communications, and Monitoring
12.1 Identifying and Minimizing Privacy Risks in AI, Machine Learning, and Deep Learning 12.2 Privacy Risks of Communications Technologies: Video Calls, Messaging, Mobile Devices, Social Media, and Gaming 12.3 Workplace Monitoring and BYOD Privacy
13 Chapter 13: Monitoring and Managing Privacy Risk
13.1 Privacy Audits, IT Control Reviews, KRIs, and KPIs 13.2 Completing Privacy and Data Protection Impact Assessments
14 Chapter 14: Implementing Privacy by Design
14.1 Cavoukian's Seven Foundational Principles of Privacy by Design 14.2 Privacy Goals, Objectives, and Translating High-Level Specifications into Low-Level Specifications 14.3 Hoepman's Privacy Design Strategies: Data-Oriented Strategies 14.4 Hoepman's Privacy Design Strategies: Process-Oriented Strategies
15 Chapter 15: Evaluating Privacy Risks in User Experiences
15.1 Privacy in User Experience: UX Concepts and Usability Testing 15.2 Value Sensitive Design
16 Chapter 16: Privacy Engineering Objectives, Architecture, and the Development Life Cycle
16.1 NIST Privacy Engineering Objectives: Predictability, Manageability, and Disassociability 16.2 Enterprise Architecture, Data Flow Mapping, Lineage, and Cross-Border Transfers 16.3 Managing Privacy Risk in the Software Development Life Cycle 16.4 Applying LINDDUN Threat Modeling in Engineering
17 Chapter 17: Managing and Monitoring Privacy Controls
17.1 Data Inventories, Records of Processing Activities, and Asset Discovery 17.2 Privacy Code Review and Runtime Behavior Monitoring
6.3 Extracting Personal Data from Publicly Available Sources
Key Takeaways Explain This Topic AI
Publicly accessible personal data remains protected by privacy law, as twelve data protection authorities stressed in their August 2023 joint statement on data scraping.
The robots.txt protocol is an advisory crawler convention; it neither secures data nor provides a legal basis for collecting it.
An organization extracting public data should document a purpose and lawful basis, collect only allowlisted fields, filter sensitive and children's data, pseudonymize early, honor opt-outs, and keep raw scrapes briefly.
When individual notice is impossible, GDPR Article 14(5)(b) still requires measures such as a public notice explaining the collection and how to object.
Defenses against hostile scrapers include rate limiting, TLS fingerprinting (JA3/JA4), behavioral challenges, and limiting what profiles expose to logged-out visitors.
Last updated: October 2026
6.3 Extracting Personal Data from Publicly Available Sources
Quick Summary: "It's public" is not a legal basis. Personal data on websites, social networks, public registers, and code repositories remains protected by privacy law, and scraping it at scale creates new risks: profiles that never existed before, re-identification, and use in contexts people never expected. The BoK asks technologists to minimize risk when their own organization extracts public data, and this section also covers how to protect your users from other people's scrapers.
Organizations extract public data for many reasons: search indexing, price monitoring, threat intelligence, academic research, recruiting, and, increasingly, training AI models. Each use raises the same questions: is there a lawful basis, how much personal data is really needed, and what happens to the people whose data is collected?
Automated Web Scraping, Bot Traffic, and Privacy Boundaries
Automated web scraping involves programmatic extraction of content from web properties using headless browsers (e.g., Puppeteer, Playwright), HTTP client libraries, or specialized crawling clusters. While scraping is widely used for search indexing and price monitoring, it poses severe privacy risks when applied to user profiles, community forums, and public directories.
The Technical Role and Limitations of robots.txt
The Robots Exclusion Protocol ( robots.txt ) is a text file placed at the root of a domain that instructs automated bots which URL paths they should not crawl: User-agent: *
- Disallow: /user/profiles/
- Disallow: /api/internal/
- Crawl-delay: 10
From a technical and legal standpoint:
robots.txt is an advisory convention , not an access control mechanism. It provides zero technical enforcement: malicious or unauthorized crawlers can simply ignore the file.
Disallowing paths in robots.txt does not make data confidential, nor does allowing paths grant lawful permission or user consent under privacy laws.
Privacy Implications of Public Data Extraction
A common engineering misconception is that data published publicly on the internet is exempt from privacy regulations. Regulatory enforcement bodies (including European DPAs and the FTC) have repeatedly rejected this premise:
Lack of Lawful Basis: Scraping public social media profiles, facial photos, or forum contributions to build commercial identity databases or train generative AI models violates GDPR Article 6, as data subjects have neither given consent nor reasonably expected mass automated commercial exploitation.
Re-Identification and Mosaic Attacks: Automated scrapers can aggregate disparate public datasets (e.g., professional resumes, voting registration records, public forum posts, and code repositories). When linked via unique handles or cross-referenced demographic markers, this aggregated profile destroys the contextual privacy the user relied upon when posting in individual spaces.
Technical Perimeter Defenses Against Automated Scraping
Organizations implement multi-layered defenses at the web perimeter to protect user data from automated harvesting:
TLS Client Fingerprinting (JA3 and JA4): Standard web scraping libraries (e.g., Python Requests, Curl, Go HTTP) produce distinct TLS Client Hello packet configurations—including specific cipher suites, extensions, and elliptic curves—that differ markedly from genuine consumer browsers (Chrome, Safari, Firefox). Web Application Firewalls (WAFs) compute the JA3/JA4 cryptographic hash of incoming TLS handshakes, immediately flagging or blocking non-browser signatures.
HTTP/2 and Network Flow Analysis: Inspects HTTP/2 connection settings (such as SETTINGS_HEADER_TABLE_SIZE , WINDOW_UPDATE sequences, and pseudo-header ordering) to detect headless automation runtimes even when request user-agent headers are spoofed.
Algorithmic Rate Limiting: Deploying Token Bucket or Leaky Bucket algorithms at the API gateway or CDN layer to limit the number of requests per IP address, autonomous system number (ASN), or API token within rolling time windows.
Interactive Challenge Gates: Triggering cryptographic proof-of-work challenges or CAPTCHA verifications when behavioral heuristics (such as absence of mouse movement, instantaneous form completion, or erratic navigation intervals) suggest automated scraping agents.
Minimizing Risk When Your Organization Extracts Public Data
Data protection authorities have made their expectations clear. In August 2023, twelve members of the Global Privacy Assembly's enforcement working group issued a joint statement on data scraping , reminding social media companies and scrapers that publicly accessible personal data is still subject to privacy law, and followed up in October 2024 with a concluding statement on the safeguards platforms had adopted. In the EU, the EDPB's Opinion 28/2024 on AI models (December 2024) discusses web-scraped training data and lists measures that can tip a legitimate-interests assessment, such as excluding certain sources and honoring opt-outs. A privacy technologist designing a collection pipeline can apply these steps:
Step Technique
1. Define the purpose and lawful basis Write down why the data is needed; in the EU, document a legitimate-interests assessment and, for large-scale or novel scraping, a DPIA.
2. Choose sources deliberately Exclude sites whose terms or robots.txt prohibit crawling, sites aimed at children, and sources rich in sensitive data (health forums, dating sites) unless the purpose requires them and safeguards exist.
3. Collect only the needed fields Parse pages into an allowlisted schema (for example, product name and price) and discard names, photos, and contact details at the crawler instead of storing full pages.
4. Filter sensitive and special-category data Run classifiers to drop or mask health, sexual orientation, religious, political, biometric, and children's data before storage.
5. Pseudonymize or aggregate early Replace usernames with keyed hashes, or keep only counts and statistics when individual records are not needed.
6. Honor opt-outs Maintain a suppression list for people and sites that object, and respect machine-readable opt-out signals where they exist.
7. Avoid re-identification by combination Do not join scraped data with customer records or other datasets unless that combination is part of the documented purpose.
8. Set short retention and provenance Record where and when each record was collected, and delete raw scrapes once the derived dataset is built.
9. Provide transparency When individual notice is impossible, GDPR Article 14(5)(b) still requires appropriate measures such as a public notice describing the collection and how to object.
These steps apply the same minimization logic used everywhere else in the CIPT: reduce identifiability and volume at the earliest point, then govern what remains.