Study guide
12.1 Identifying and Minimizing Privacy Risks in AI, Machine Learning, and Deep Learning
Certified Information Privacy Technologist CIPT
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CIPT Exam Overview and Study Plan
1.1 CIPT Exam Facts, Blueprint, and Study Plan
2 Chapter 2: The Privacy Technologist's Legal and Procedural Role
2.1 Privacy Function Roles: DPO, Data Owner, Steward, Custodian, Legal, and Security 2.2 Translating Legal and Regulatory Requirements into Technical Controls 2.3 Notices, Policies, and Oversight of Privacy Operations, Audits, and Third Parties
3 Chapter 3: Technical Responsibilities: Risk Concepts and Incident Response
3.1 Risk Concepts: Threat, Vulnerability, Attack, Exploit, and Privacy Risk 3.2 Technical Support for Privacy Incident and Breach Response
4 Chapter 4: Privacy Risk Models, Frameworks, and Threat Models
4.1 Privacy Risk Models and Frameworks: Contextual Integrity, Calo, Solove, FIPPs, OECD, NIST, and FAIR 4.2 Privacy Threat Models: LINDDUN and MITRE PANOPTIC
5 Chapter 5: Data Ethics and Automated Decision-Making
5.1 Legal Versus Ethical Processing and the Social Impact of Privacy-Affecting Designs 5.2 Identifying and Minimizing Bias in Automated Decision-Making
6 Chapter 6: Minimizing Privacy Risk During Data Collection
6.1 Notice, Consent Management, and Choice Architecture 6.2 Managing Privacy Risk in Automatic Collection and Telemetry 6.3 Extracting Personal Data from Publicly Available Sources 6.4 Data Retention, Destruction, and Cryptographic Erasure 6.5 Data Minimization Techniques 6.6 Implementing Data Processing Segregation
7 Chapter 7: Minimizing Privacy Risk During Data Use: Aggregation, PETs, and Secondary Use
7.1 Minimizing Privacy Risk from Aggregation and Inference 7.2 De-identification, Pseudonymization, and Tokenization 7.3 Microdata Protection: k-Anonymity, l-Diversity, and t-Closeness 7.4 Advanced PETs: Differential Privacy, Secure Multi-Party Computation, and Homomorphic Encryption 7.5 Secondary Use, Function Creep, and Purpose Limitation Controls
8 Chapter 8: Minimizing Privacy Risk During Dissemination
8.1 Disclosure and Accessibility Controls: APIs, Egress, and Clean Rooms 8.2 Countering Distortion, Exposure, Breach of Confidentiality, Blackmail, and Appropriation 8.3 Defense in Depth: Identity, Access Management, and Authentication
9 Chapter 9: Intrusion, Decisional Interference, and Dark Patterns
9.1 Minimizing Intrusion and Decisional Interference: Behavioral Advertising, Profiling, Cyberbullying, and Social Engineering 9.2 Dark Patterns and Deceptive Design
10 Chapter 10: Privacy Risks Related to Software Security
10.1 Detecting and Fixing Software Privacy Vulnerabilities 10.2 Intrusion Detection and Prevention, and Privacy in Change Management 10.3 Service Providers, Third-Party SDKs, and Supply Chain Risk
11 Chapter 11: Tracking and Surveillance Technologies
11.1 Privacy Risks and Impacts of E-Commerce 11.2 Audio and Video Surveillance, Wearables, and IoT 11.3 Privacy Issues Around Biometrics: Facial, Voice, Fingerprint, and DNA Data 11.4 Privacy Risks and Impacts of Location Tracking 11.5 Internet Monitoring and Web Tracking: Cookies, Fingerprinting, and GPC
12 Chapter 12: Workplace Technologies: AI, Communications, and Monitoring
12.1 Identifying and Minimizing Privacy Risks in AI, Machine Learning, and Deep Learning 12.2 Privacy Risks of Communications Technologies: Video Calls, Messaging, Mobile Devices, Social Media, and Gaming 12.3 Workplace Monitoring and BYOD Privacy
13 Chapter 13: Monitoring and Managing Privacy Risk
13.1 Privacy Audits, IT Control Reviews, KRIs, and KPIs 13.2 Completing Privacy and Data Protection Impact Assessments
14 Chapter 14: Implementing Privacy by Design
14.1 Cavoukian's Seven Foundational Principles of Privacy by Design 14.2 Privacy Goals, Objectives, and Translating High-Level Specifications into Low-Level Specifications 14.3 Hoepman's Privacy Design Strategies: Data-Oriented Strategies 14.4 Hoepman's Privacy Design Strategies: Process-Oriented Strategies
15 Chapter 15: Evaluating Privacy Risks in User Experiences
15.1 Privacy in User Experience: UX Concepts and Usability Testing 15.2 Value Sensitive Design
16 Chapter 16: Privacy Engineering Objectives, Architecture, and the Development Life Cycle
16.1 NIST Privacy Engineering Objectives: Predictability, Manageability, and Disassociability 16.2 Enterprise Architecture, Data Flow Mapping, Lineage, and Cross-Border Transfers 16.3 Managing Privacy Risk in the Software Development Life Cycle 16.4 Applying LINDDUN Threat Modeling in Engineering
17 Chapter 17: Managing and Monitoring Privacy Controls
17.1 Data Inventories, Records of Processing Activities, and Asset Discovery 17.2 Privacy Code Review and Runtime Behavior Monitoring
12.1 Identifying and Minimizing Privacy Risks in AI, Machine Learning, and Deep Learning
Key Takeaways Explain This Topic AI
Models can memorize personal data: researchers extracted verbatim names, phone numbers, and addresses from GPT-2's training data in 2021, so a trained model is not automatically anonymous.
Membership inference determines whether a person's record was in the training data, and model inversion reconstructs sensitive attributes from a model's outputs.
The EDPB's Opinion 28/2024 says an AI model trained on personal data can be considered anonymous only if personal data cannot be extracted or inferred with reasonable means.
Generative AI deployments add prompt leakage, retrieval that ignores document permissions, over-privileged agents, and hallucinated statements about people; OWASP's 2025 LLM Top 10 lists sensitive information disclosure as LLM02.
Mitigations include training-data minimization and deduplication, differential privacy in training (DP-SGD), permission-aware retrieval, input and output filtering, no-training contract terms, red-teaming, and governance under NIST AI RMF, ISO/IEC 42001, and the EU AI Act.
Last updated: October 2026
12.1 Identifying and Minimizing Privacy Risks in AI, Machine Learning, and Deep Learning
Quick Summary: Artificial intelligence (AI) systems learn from data, and much of that data is personal. Privacy risk therefore appears in the training data (where it came from and whether it may be used), inside the model (what it memorized), at inference (what attackers can extract and what users type in), and in the outputs (what the model says about people). Section 5.2 covered bias in automated decisions; this section covers the privacy risks of AI as a workplace and product technology.
The BoK lists AI, machine learning (ML), and deep learning under workplace technologies, reflecting how quickly employees and products have adopted them. Deep learning models, including large language models (LLMs), are trained on enormous datasets and contain billions of parameters, which makes their behavior powerful and hard to audit.
Risks Across the AI Life Cycle
Loading diagram...
1. Training Data
Lawful basis and purpose limitation: customer data collected to provide a service is often reused to train models without a compatible purpose or consent (Section 7.5).
Web scraping: publicly available data remains personal data (Section 6.3). The EDPB's Opinion 28/2024 (December 2024) explains how legitimate interests may or may not support training on such data and lists mitigating measures.
Sensitive data: training sets often contain health, financial, and children's data that was never meant for model training.
2. Memorization and Extraction
Neural networks can memorize rare sequences seen during training. In 2021, Carlini and colleagues extracted hundreds of verbatim training examples from GPT-2, including names, phone numbers, email addresses, and physical addresses. Memorization increases with model size and with data that appears many times. A model that can reproduce personal data is itself a store of personal data.
3. Inference Attacks on Models
Membership inference: determining whether a specific person's record was in the training set, for example by observing that the model is unusually confident on that record (Shokri and colleagues, 2017). Membership alone can be sensitive if the dataset is, say, patients with a particular disease.
Model inversion: reconstructing sensitive features of training data from model outputs (Fredrikson and colleagues showed reconstruction of recognizable faces from a face-recognition model in 2015).
Attribute inference: using a model to predict attributes a person never disclosed.
Because of these attacks, the EDPB concluded in Opinion 28/2024 that an AI model trained on personal data can be treated as anonymous only if the likelihood of extracting personal data from it, or obtaining it through queries, is insignificant with reasonable means.
4. Generative AI in Daily Use
Prompt leakage: employees paste confidential or personal data into public chatbots. In 2023 Samsung restricted staff use of generative AI after engineers pasted internal source code into a public tool.
Retrieval-augmented generation (RAG): a chatbot that searches internal documents can reveal content the user is not permitted to see if retrieval ignores document permissions.
Agents with excessive access: AI agents that can read mailboxes or call APIs can exfiltrate data when manipulated by prompt injection hidden in a web page or email.
Vector databases: embeddings of documents and conversations are derived personal data that must be secured, inventoried, and deleted on request.
Hallucinations about people: models generate confident false statements about real individuals, raising accuracy and defamation concerns (Section 8.2).
The OWASP Top 10 for LLM Applications (2025) lists Prompt Injection as LLM01 and Sensitive Information Disclosure as LLM02 , alongside risks such as system prompt leakage, excessive agency, and vector and embedding weaknesses.
5. Rights Requests
Erasure and correction are hard once data is embedded in model weights. "Machine unlearning" research is immature, so practical options include removing the data from future training sets, filtering outputs, retraining on schedule, and avoiding training on data likely to be the subject of requests.
Controls That Minimize AI Privacy Risk
Stage Control
Data Document sources and lawful basis; exclude sensitive and children's data unless justified; scrub direct identifiers; deduplicate training data (reduces memorization)
Training Differentially private training ( DP-SGD ) bounds what any one record can influence; federated learning keeps raw data on devices
Testing Red-team for training-data extraction, membership inference, and prompt injection before release
Deployment Input DLP that blocks or redacts personal data in prompts; output filters for personal data; permission-aware retrieval that enforces the user's access rights; least-privilege tools for agents
Vendors Enterprise terms that prohibit training on customer prompts, define retention, and specify data location
Governance AI inventory, impact assessments (DPIA and, for certain deployers, the AI Act's fundamental rights impact assessment), model cards, and incident response for AI failures
Governance Frameworks
NIST AI Risk Management Framework (AI RMF 1.0, January 2023) with its functions Govern, Map, Measure, and Manage, and the Generative AI Profile (NIST AI 600-1, July 2024) , which lists data privacy among the risks unique to or worsened by generative AI.
ISO/IEC 42001:2023 , a certifiable AI management system standard.
EU AI Act , with transparency duties for chatbots and synthetic content from August 2, 2026 and high-risk obligations from December 2, 2027 for Annex III systems (Section 5.2).