Study guide
1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
Google Cloud Certified Professional Data Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Security, IAM, and Data Protection
1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
2 Chapter 2: Reliability and Compliance Architecture
2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack
3 Chapter 3: Cross-Cloud and Federated Governance
3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance
4 Chapter 4: Data Ingestion and Pipeline Planning
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion
5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub
5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow
6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows
7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics
7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management
8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion
8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage
9 Chapter 9: Storage Selection and Cloud Storage Architecture
9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools
10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL
10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery
11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore
11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections
12 Chapter 12: Enterprise Data Warehousing with BigQuery
12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions
13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery
13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling
14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML
14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles
15 Chapter 15: Operations, Orchestration, and Data Governance
15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services
1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
Key Takeaways Explain This Topic AI
Google Cloud Sensitive Data Protection (formerly Cloud DLP) provides automated discovery, inspection, and classification of sensitive data across BigQuery, Cloud Storage, and real-time streaming pipelines using over 150 built-in infoTypes.
Custom infoTypes extend standard pattern recognition through regular expressions (Regex), dictionary matchers from Cloud Storage wordlists, and proximity-based context rules that dynamically boost likelihood scores.
Cryptographic Deterministic Tokenization (Crypto-Deterministic using AES-SIV) preserves referential integrity and joinability across pseudonymized tables without exposing raw sensitive identifiers like Social Security Numbers.
Format-Preserving Encryption (FPE via AES-FFX) transforms sensitive plaintext strings into surrogate tokens that retain the exact character set and length of the original data, ensuring compatibility with strict legacy database schemas.
High-throughput streaming Dataflow pipelines integrating Sensitive Data Protection must micro-batch records into grouped API requests and call regional endpoints to prevent request serialization and quota exhaustion.
Last updated: September 2026
1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to discover and sanitize Personally Identifiable Information (PII) and Protected Health Information (PHI). You must master the configuration of built-in and custom infoTypes , contrast deterministic tokenization against format-preserving encryption (FPE) , calculate statistical privacy guarantees ( k-anonymity , l-diversity ), and architect high-throughput Dataflow pipelines that call the Sensitive Data Protection API without exhausting quotas or degrading stream throughput.
Enterprise data warehouses and data lakes routinely ingest diverse data streams containing credit card numbers, national identification codes, medical identifiers, and personal addresses. Left unprotected, these sensitive attributes expose organizations to catastrophic regulatory penalties under GDPR, HIPAA, CCPA, and PCI-DSS. Google Cloud Sensitive Data Protection (historically and commonly known on the exam as Cloud DLP ) serves as the core platform service for discovering, classifying, and transforming sensitive data at enterprise scale.
1. Sensitive Data Protection Architecture: Discovery, Inspection, and Classification
Sensitive Data Protection operates across three primary planes: automated discovery, on-demand inspection, and real-time de-identification transformations: +-----------------------------------------------------------------------------+
- | SENSITIVE DATA PROTECTION CAPABILITIES |
- +-----------------------------------------------------------------------------+
- | 1. Discovery Service | 2. Inspection Jobs | 3. De-identification |
- | - Continuous profiling | - Deep scans of GCS, | - Masking, redaction |
- | - Generates data risk | BigQuery, Datastore | - Tokenization (FPE) |
- | & sensitivity tags | - Automated sampling | - Bucketing |
- | - Dataplex integration | - Pub/Sub notifications | - Streaming Dataflow |
- +-----------------------------------------------------------------------------+
- Automated Discovery and Profiling
Integrated with Dataplex , the Sensitive Data Protection discovery service automatically and continuously profiles BigQuery tables across organizations, folders, and projects. It scans schemas and row samples to assign Sensitivity Levels ( High , Moderate , Low ) and Data Risk Scores ( High , Medium , Low ). These profiles generate metadata tags that feed into Dataplex catalog policies without requiring manual scan configurations.
Storage Inspection Jobs
Data engineers can configure batch inspection jobs to scan stationary data assets:
Cloud Storage: Inspects structured (CSV, JSON, Avro, Parquet), semi-structured, and unstructured data (text files, PDFs, scanned TIFF/JPEG images via Optical Character Recognition OCR).
BigQuery: Scans partitioned and clustered tables. Supports row-level sampling (e.g., scan only a random 10% sample or limit scan to the first 50,000 rows) to minimize processing costs on petabyte-scale datasets.
Cloud SQL & Datastore: Scans operational database tables and entity kinds.
2. infoTypes Taxonomy: Built-in, Custom Regex, and Dictionaries
An infoType is a defined detector representing a specific category of sensitive data (e.g., credit card number, personal email address, passport ID).
Built-in infoTypes
Google Cloud provides over 150 predefined built-in detectors spanning international regulatory frameworks:
CREDIT_CARD_NUMBER : Matches major financial cards using length checks and the Luhn checksum algorithm.
US_SOCIAL_SECURITY_NUMBER : Identifies 9-digit US tax IDs with valid issuing area and group checks.
EMAIL_ADDRESS : Validates RFC-compliant email structures.
GCP_CREDENTIALS : Scans for leaked Google Cloud service account keys, API keys, and OAuth secrets.
IP_ADDRESS : Detects IPv4 and IPv6 addresses.
MEDICAL_RECORD_NUMBER : Locates clinical patient record identifiers across various national formats.
Custom infoTypes
When built-in detectors do not cover proprietary business identifiers, data engineers configure custom infoTypes:
Regular Expression (Regex) Detectors: Matches strict alphanumeric patterns, such as internal employee badges or proprietary account codes (e.g., EMP-[0-9]{6}-[A-Z]{2} ).
Dictionary Detectors: Matches tokens against a predefined list of words. Dictionaries can be passed inline (up to tens of thousands of words) or sourced directly from a text file in Cloud Storage (supporting millions of phrases, such as internal product project names or specialized medication terms).
Surrogate Detectors: Used to identify tokens that have already undergone de-identification so they can be tracked, audited, or selectively re-identified.
Likelihood Scoring and Context Rules
Every finding returned by an inspection scan is evaluated with a Likelihood Score : VERY_UNLIKELY ⟶ UNLIKELY ⟶ POSSIBLE ⟶ LIKELY ⟶ VERY_LIKELY \text{VERY\_UNLIKELY} \longrightarrow \text{UNLIKELY} \longrightarrow \text{POSSIBLE} \longrightarrow \text{LIKELY} \longrightarrow \text{VERY\_LIKELY} VERY_UNLIKELY ⟶ UNLIKELY ⟶ POSSIBLE ⟶ LIKELY ⟶ VERY_LIKELY
Data engineers fine-tune detection accuracy using Context Rules (Hotwords):
If a 9-digit numeric string is encountered, its default likelihood might be POSSIBLE .
If the phrase "SSN" , "Social Security" , or "taxpayer ID" appears within a proximity window of 30 characters preceding the number, a context rule elevates the likelihood to VERY_LIKELY .
Conversely, if the token is preceded by "Order ID:" , negative context rules can downgrade the likelihood to VERY_UNLIKELY or exclude the finding entirely, eliminating false positives.
3. De-identification Techniques and Mathematical Guarantees
De-identification modifies sensitive data so that it can be safely used for analytics, machine learning, and testing without violating privacy regulations. The choice of technique depends on whether the transformation must be reversible, whether referential integrity must be maintained across tables, and whether downstream database schemas are rigid. Raw PII Record: { SSN: "123-45-6789", Age: 42, Zip: "94103", Card: "4111-2222-3333-4444" }
- │
- ┌──────────────────────────────┼──────────────────────────────┐
- ▼ ▼ ▼
- [ Character Masking ] [ Deterministic Token ] [ Bucketing / Coarsening ]
- Card: "############4444" SSN: "TOK_9xL2aBq1v" Age: "40-49"
- (Irreversible / Display) (Reversible with KMS key, Zip: "94100"
- preserves SQL JOINs) (k-Anonymity defense)
- 1. Masking and Redaction
Character Masking: Replaces characters with a fixed symbol (e.g., replacing all but the last four digits of a credit card number with # : ############4444 ).
Redaction: Completely removes the matching text string or blackens the bounding box region in an image file (e.g., redacting handwritten SSNs from scanned medical intake PDFs).
2. Cryptographic Hashing with Salt
Uses a secure hash function (e.g., SHA-256) combined with a cryptographic salt.
One-Way Transformation: Completely irreversible. Protects against rainbow table dictionary attacks.
Limitation: Cannot be reversed even by authorized administrators; does not preserve formatting.
3. Cryptographic Deterministic Tokenization (Crypto-Deterministic)
The Golden Rule for Data Warehousing: Uses a symmetric key with AES-SIV (Synthetic Initialization Vector) to encrypt the plaintext.
Referential Integrity: Identical plaintexts always produce the exact same ciphertext token across different tables, databases, and batch runs (e.g., SSN 123-45-6789 always encrypts to TOK_8f92jklmno ).
SQL Analytical Value: Data analysts can perform SQL JOIN , GROUP BY , and COUNT(DISTINCT) operations across de-identified tables without ever exposing the real underlying identity. -- Analytical JOIN executed across tokenized tables without exposing raw SSN
- SELECT a.tokenized_ssn, b.credit_rating, a.total_balance
- FROM `analytics_dw.customer_accounts` a
- JOIN `analytics_dw.credit_scores` b
- ON a.tokenized_ssn = b.tokenized_ssn
- WHERE a.total_balance > 10000;
- 4. Format-Preserving Encryption (FPE)
The Legacy Integration Standard: Uses the AES-FFX mode of operation to encrypt data while strictly preserving both the character length and the character set of the input string.
Example: A 16-digit numeric credit card string 4111222233334444 is encrypted into another 16-digit numeric string 8942103948572019 that passes downstream schema validation and length constraints.
Use Case: Critical when downstream systems rely on legacy database schemas (e.g., VARCHAR(16) or strict numeric type checks) that would fail if replaced by long hex strings or surrogate tags.
Reversibility: FPE is completely reversible by authorized users who hold the wrapping key in Cloud KMS.
5. Statistical Anonymity Models (k-Anonymity and l-Diversity)
Even if direct identifiers (names, SSNs) are removed, attackers can frequently re-identify individuals by cross-referencing "quasi-identifiers" (e.g., ZIP Code, Gender, Birth Date) against public voter registries.
k-Anonymity: A dataset possesses k k k -anonymity if each distinct combination of quasi-identifiers occurs at least k k k times across the dataset. In a k = 5 k=5 k = 5 dataset, any individual's quasi-identifiers are indistinguishable from at least 4 other individuals. Achieved via bucketing (e.g., converting Age 43 to range [40-49] ) and generalization (e.g., truncating ZIP 94103 to 941** ).
l-Diversity: Extends k k k -anonymity by ensuring that within every group of k k k indistinguishable records, sensitive attributes (e.g., medical diagnoses) have at least l l l distinct, well-represented values. This prevents homogeneity attacks (e.g., if all 5 people in an equivalence class have Cancer, k k k -anonymity fails to protect privacy).
- Technique Reversible? Preserves Joins? Preserves Format/Length? Primary Exam Scenario
- Masking No No Optional UI presentation; hiding customer credit cards on receipts
- Crypto-Hash No Yes (with same salt) No Non-reversible record matching; one-way integrity checks
Crypto-Deterministic Yes (with KMS key) Yes No (Generates surrogate) Joining pseudonymized tables in BigQuery across datasets
- Format-Preserving (FPE) Yes (with KMS key) Yes Yes Passing data into legacy databases with rigid schema limits
- Bucketing / Generalization No No No Satisfying k k k -anonymity and l l l -diversity compliance
- 4. Inspection and De-identification Templates
In enterprise governance, separating security policy definition from pipeline implementation is mandatory:
InspectTemplate: Encapsulates the infoTypes to scan, minimum likelihood thresholds, maximum findings limits, and custom exclusion rules.
DeidentifyTemplate: Encapsulates the exact transformation logic (e.g., apply FPE with KMS key X X X to CREDIT_CARD_NUMBER , mask EMAIL_ADDRESS with * ).
Security teams define these templates centrally at the Organization or Folder level. Data engineers developing Dataflow pipelines or BigQuery scheduled jobs simply reference the centralized template Resource ID (e.g., organizations/123/deidentifyTemplates/global-pii-mask ). If regulatory standards mandate updated masking rules, the security team updates the centralized template, and all downstream data pipelines instantly adopt the new rules without code redeployment.
5. High-Throughput Pipeline Architecture: Dataflow Integration
Integrating the Sensitive Data Protection API into streaming Apache Beam / Cloud Dataflow pipelines presents significant performance and quota challenges. Calling the DLP API synchronously for every individual event is an architectural anti-pattern that will fail in production. +--------------------------------------------------------------------------------+
- | PRODUCTION DATAFLOW DLP STREAMING PIPELINE |
- +--------------------------------------------------------------------------------+
- | |
- | [ Pub/Sub Stream ] |
- | │ |
- | ▼ |
- | [ Windowing / GroupIntoBatches ] <-- Batches 100-500 events or 500 KB |
- | │ |
- | ▼ |
- | [ Call Regional DLP API ] <-- dlp.us-central1.googleapis.com |
- | │ Using centralized DeidentifyTemplate |
- | ├───> Success ──────────────> [ BigQuery Analytics Tables ] |
- | │ |
- | └───> Transient Error / ─────> [ Exponential Backoff Retry ] |
- | Permanent Failure ────> [ Dead-Letter Pub/Sub / GCS Queue ] |
- +--------------------------------------------------------------------------------+
- The Three Production Rules for Dataflow DLP Pipelines
Batching via GroupIntoBatches :
The Sensitive Data Protection API has strict per-project rate limits on requests per minute, but each request can inspect or de-identify up to 500 KB or tens of thousands of characters.
Always insert a Beam transform ( GroupIntoBatches or utilize the official Beam DlpDeidentifyText PTransform) to aggregate incoming streaming messages into micro-batches (e.g., 100 to 500 records per batch) before invoking the API.
Regional Endpoints:
By default, API calls may route to global endpoints. In high-throughput architectures, configure the pipeline to call the regional endpoint matching the Dataflow workers (e.g., dlp.us-central1.googleapis.com ). This reduces round-trip network latency and enforces data residency.
Dead-Letter Error Handling:
Corrupted messages or payloads exceeding size limits must not crash the streaming pipeline. Implement a dead-letter queue (DLQ) pattern: unparseable or failed transformations are caught, enriched with error metadata, and routed to a secondary Pub/Sub dead-letter topic or Cloud Storage bucket for manual remediation.
6. Realistic Exam Scenarios & Architecture Pitfalls
Scenario / Problem Common Architecture Anti-Pattern Correct Google Cloud Architecture
Preserving Table Joins on PII ; Two separate analytics datasets contain customer transactions and credit scores. Both tables contain SSNs. The data engineering team must de-identify the SSNs but maintain the ability to join records on the SSN field. Applying character masking ( XXX-XX-1234 ) or random UUID substitution. Apply Cryptographic Deterministic Tokenization (Crypto-Deterministic) using an AES-SIV key stored in Cloud KMS. The same SSN will consistently yield the same surrogate token across both tables, preserving SQL JOIN integrity.
Streaming Dataflow Pipeline Throttling ; A streaming Dataflow pipeline de-identifying clickstream data crashes due to 429 RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Requests' . Requesting a 1,000x quota increase or scaling up the number of Dataflow worker VMs. Implement micro-batching in the pipeline using GroupIntoBatches to group hundreds of individual records into a single multi-record DLP API request, staying well within per-minute API request limits.
Legacy Database Ingestion Failure ; A legacy relational database must ingest de-identified credit card numbers. The database schema strictly defines the column as NUMERIC(16) . Standard cryptographic tokens fail because they contain alphanumeric characters. Converting the column to text in the source or stripping non-numeric characters from random hash strings. Utilize Format-Preserving Encryption (FPE) via the AES-FFX mode. FPE outputs an encrypted surrogate that is strictly 16 numeric digits, fully satisfying the downstream legacy schema constraints.
Re-identifying Patient Data in Emergencies ; A clinical research hospital stores de-identified patient records in BigQuery. In clinical emergencies, authorized physicians must be able to re-identify the original patient record. Using one-way SHA-256 cryptographic hashing or character redaction. Use Reversible Surrogate Tokenization (Crypto-Deterministic or FPE) wrapped by a Cloud KMS key. Restrict access to the KMS unwrapping key and re-identification template to authorized physicians via Cloud IAM.