Study guide

15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage

Google Cloud Certified Professional Data Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Security, IAM, and Data Protection

1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking

2 Chapter 2: Reliability and Compliance Architecture

2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack

3 Chapter 3: Cross-Cloud and Federated Governance

3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance

4 Chapter 4: Data Ingestion and Pipeline Planning

4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion

5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub

5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics

6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow

6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows

7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics

7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management

8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion

8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage

9 Chapter 9: Storage Selection and Cloud Storage Architecture

9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools

10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL

10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery

11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore

11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections

12 Chapter 12: Enterprise Data Warehousing with BigQuery

12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions

13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery

13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling

14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML

14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles

15 Chapter 15: Operations, Orchestration, and Data Governance

15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services

15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage

Key Takeaways Explain This Topic AI

Dataplex establishes a unified logical data mesh across distributed Cloud Storage buckets and BigQuery datasets without physically relocating data or duplicating underlying assets.

Data is organized hierarchically into Lakes (representing business domains), Zones (Raw zones for landing unstructured/semi-structured data with permissive schema, and Curated zones for cleansed, partitioned Parquet or BigQuery tables with strict schema enforcement), and Assets (physical GCS buckets or BQ datasets).

Dataplex Automated Discovery continuously crawls object storage to infer schemas, detect Hive-partitioning schemes, track schema evolution, and automatically register tables in BigQuery Metastore and Dataproc Metastore.

Automated Data Quality (AutoDQ) evaluates declarative, YAML-defined quality rules (null checks, range validation, regex matching, freshness) serverlessly, publishing metrics to Cloud Monitoring and alerting via Cloud Logging.

Dataplex Data Lineage automatically captures end-to-end transformation provenance across BigQuery, Dataflow, Cloud Composer, and Dataproc, tracking table-level and column-level mutations to support root-cause analysis and compliance auditing.

Last updated: September 2026

15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to design and implement enterprise data governance, metadata discovery, and quality architectures using Dataplex . Key topics include: structuring logical data hierarchies using Lakes , Zones (Raw vs. Curated) , and Assets ; configuring Automated Discovery for schema inference and Hive partition harvesting; defining declarative Data Quality (AutoDQ) and Data Profiling tasks; implementing centralized IAM security and policy propagation ; and tracking end-to-end data provenance using Dataplex Data Lineage across BigQuery, Dataflow, Cloud Composer, and Dataproc.

As enterprise data estates expand across hundreds of Cloud Storage buckets, heterogeneous BigQuery datasets, and hybrid lakehouse topologies, centralized data management models break down. Monolithic data teams become operational bottlenecks, data consumers struggle to discover authoritative assets, and fragmented security policies expose organizations to compliance breaches under regulations like GDPR and CCPA. Google Cloud Dataplex solves these challenges by providing an intelligent, unified data fabric that automates metadata discovery, enforces distributed governance policies, standardizes data quality, and tracks end-to-end lineage across multi-project environments without moving or duplicating underlying data.

1. Dataplex Architecture: The Logical Governance Mesh

Traditional data governance required centralized platforms where data had to be physically extracted and loaded into a single repository to apply unified controls. Dataplex separates the logical governance plane from the physical storage tier . Data remains in its native, cost-effective storage locations—whether Cloud Storage buckets or BigQuery datasets—while Dataplex provides centralized discovery, security administration, and lifecycle orchestration. +───────────────────────────────────────────────────────────────────────────────────────────+

  • | DATAPLEX LOGICAL GOVERNANCE MESH |
  • +───────────────────────────────────────────────────────────────────────────────────────────+
  • | |
  • | DATAPLEX LAKE: "enterprise_customer_analytics" (Domain Container) |
  • | +─────────────────────────────────────────────────────────────────────────────────────+ |
  • | | | |
  • | | RAW DATA ZONE (Landing Area / Permissive Schema) | |
  • | | +──────────────────────────────────+ +─────────────────────────────────────────+ | |
  • | | | ASSET: Cloud Storage Bucket | | ASSET: BigQuery Landing Dataset | |
  • | | | gs://raw-clickstream-telemetry/ | | project-ingest:landing_events | |
  • | | | (JSON, CSV, Avro landing files) | | (Unpartitioned staging tables) | |
  • | | +──────────────────────────────────+ +─────────────────────────────────────────+ | |
  • | | │ │ | |
  • | | ▼ ▼ | |
  • | | CURATED DATA ZONE (Certified Analytics Marts / Strict Schema Enforcement) | |
  • | | +──────────────────────────────────+ +─────────────────────────────────────────+ | |
  • | | | ASSET: Cloud Storage Lakehouse | | ASSET: BigQuery Production Marts | |
  • | | | gs://curated-lakehouse-parquet/ | | project-dw:customer_marts | |
  • | | | (Optimized Parquet / Iceberg) | | (Partitioned & Clustered tables) | |
  • | | +──────────────────────────────────+ +─────────────────────────────────────────+ | |
  • | | | |
  • | +─────────────────────────────────────────────────────────────────────────────────────+ |
  • +───────────────────────────────────────────────────────────────────────────────────────────+
  • The Dataplex Logical Hierarchy

Dataplex organizes data using a four-tier hierarchical abstraction:

Lake: The top-level administrative container that maps directly to a business domain or department (e.g., marketing_lake , supply_chain_lake , risk_and_compliance_lake ). IAM roles, data governance parameters, and audit policies assigned at the Lake level cascade down to all enclosed zones and assets.

Zone: A logical grouping of data assets within a lake categorized by business readiness, regulatory status, or data maturity. Dataplex provides two specialized zone types:

Raw Zone: Houses freshly ingested data in its native formats (e.g., JSON, CSV, IoT logs). Used for landing zones, sandbox exploration, and unstructured files. Schema enforcement is relaxed/permissive to avoid blocking ingestion.

Curated Zone: Houses cleansed, validated, structured data intended for enterprise BI, analytical reporting, and machine learning models. Requires standardized formats (e.g., Parquet, ORC, BigQuery native tables). Schema enforcement is strict; unexpected schema mutations or data type alterations can trigger ingestion alerts or quarantine actions.

Asset: Maps directly to a physical storage resource in Google Cloud. An asset points to either a Cloud Storage bucket or a BigQuery dataset . Attaching an asset to a zone does not migrate or re-encode data; Dataplex merely attaches its metadata harvesting and security harness to the existing resource.

Entity: The logical representation of tabular data discovered by Dataplex within an asset (e.g., a specific table schema, partition structure, or file group) registered automatically in the Dataplex Catalog, BigQuery Metastore, and Dataproc Metastore.

2. Raw Zones vs. Curated Zones: Operational Tradeoffs

Designing a robust enterprise data mesh requires establishing clear boundaries between exploratory data landing areas and certified analytical products.

Architectural Dimension Raw Data Zone Curated Data Zone

Primary Objective Fast, high-throughput landing; raw data preservation; exploratory discovery Certified enterprise analytics, BI dashboards, regulatory reporting, ML features

Supported File Formats Any text or binary format (JSON, CSV, Avro, XML, uncompressed logs) Standardized, high-performance columnar formats (Parquet, ORC, BigQuery tables)

Schema Enforcement Permissive: Automatically accommodates schema evolution and unexpected column additions without failing ingestion. Strict: Schema changes must be explicitly approved; invalid schemas or unexpected types are quarantined or rejected.

Data Cleansing Status Unvalidated, duplicate-tolerant, raw upstream payloads Deduplicated, cleansed, typed, standardized, and referentially validated

Storage Resources Dedicated Cloud Storage ingestion buckets, BigQuery raw staging datasets Optimized Cloud Storage lakehouse buckets, partitioned/clustered BigQuery datasets

Target Audience Data engineers, data scientists, specialized ingestion microservices Business intelligence analysts, financial controllers, downstream operational applications

Exam Trap: Placing uncurated JSON files or unpartitioned raw text files into a Curated Zone violates Dataplex architectural standards. Curated zones enforce structured format validation; storing raw, unindexed files in a curated zone causes metadata harvesting errors and pollutes certified analytical repositories.

3. Automated Discovery, Schema Inference, and Partition Harvesting

One of the most powerful capabilities of Dataplex is its serverless Automated Discovery engine, which eliminates the need to write custom crawler scripts or cataloging pipelines. +───────────────────────────────────────────────────────────────────────────────────────────+

  • | DATAPLEX AUTOMATED METADATA DISCOVERY PIPELINE |
  • +───────────────────────────────────────────────────────────────────────────────────────────+
  • | |
  • | CLOUD STORAGE ASSET: gs://lakehouse-sales/ |
  • | ├── year=2026/month=09/day=15/orders_001.parquet |
  • | └── year=2026/month=09/day=15/orders_002.parquet |
  • | │ |
  • | ▼ |
  • | DATAPLEX DISCOVERY CRAWLER (Scheduled / Event-Driven) |
  • | - Inspects file magic bytes and columnar headers (Parquet/ORC/Avro) |
  • | - Extracts Hive-style directory partitioning: [year: INT, month: INT, day: INT] |
  • | - Infers unified schema across millions of distributed object files |
  • | │ |
  • | ▼ |
  • | METASTORE SYNCHRONIZATION & REGISTRATION |
  • | +──────────────────────────────────+ +─────────────────────────────────────────────+ |
  • | | BigQuery External Tables | | Dataproc Metastore (Hive) | |
  • | | - Table: `lakehouse_sales.orders`| | - Hive Table: `lakehouse_sales.orders` | |
  • | | - Automatically queryable via SQL| | - Instant Spark / Presto availability | |
  • | +──────────────────────────────────+ +─────────────────────────────────────────────+ |
  • +───────────────────────────────────────────────────────────────────────────────────────────+
  • How Discovery Operates

Crawler Scanning: The crawler scans attached Cloud Storage assets at scheduled intervals (or triggered upon pipeline events). It samples files, inspects binary headers, and identifies file formats without reading full file payloads.

Partition Inference: Dataplex natively detects Hive-style directory partitioning schemes (e.g., gs://bucket/table/year=2026/month=09/data.parquet ). It maps these folder hierarchies into typed partition keys ( year: INTEGER , month: INTEGER ).

Metastore Registration: Once entities and partitions are identified, Dataplex automatically registers or updates external table definitions in BigQuery and Dataproc Metastore (DPMS) . Analysts can immediately run standard SQL queries against the newly discovered files in BigQuery without executing manual CREATE EXTERNAL TABLE statements.

Schema Evolution Management: When new columns appear in incoming Parquet files, the discovery crawler detects the changes. In a Raw Zone, Dataplex automatically updates the table schema definition. In a Curated Zone, unexpected column changes can trigger an alert in Cloud Logging for administrative review.

4. Automated Data Quality (AutoDQ) and Data Profiling

Enterprise decision-making collapses when analytics rely on corrupt or drifting data. Dataplex provides built-in, serverless Data Profiling and Automated Data Quality (AutoDQ) tasks that run directly within the Google Cloud ecosystem without requiring external computing clusters.

Data Profiling

Dataplex Data Profiling automatically scans specified BigQuery tables to compute statistical summaries across columns:

Metric Generation: Calculates row counts, null percentages, distinct value ratios, min/max values, string length distributions, and numeric percentiles.

Data Anomaly Detection: Identifies distribution drift (e.g., a column suddenly shifting from 0.1% nulls to 40% nulls after an upstream schema migration).

Zero Compute Management: Profiling queries execute using BigQuery slots serverlessly; no VM clusters, Cloud Functions, or Dataproc clusters are required.

Automated Data Quality (AutoDQ) Engine

AutoDQ allows data engineers to define declarative, YAML-based data quality rules that execute on scheduled intervals or as gates within orchestration pipelines. # dataplex-auto-dq-sales.yaml

metadata:

name: sales_quality_gate

spec:

rules:

  • column: order_id
  • dimension: COMPLETENESS
  • nonNullExpectation: {}
  • column: order_id
  • dimension: UNIQUENESS
  • uniquenessExpectation: {}
  • column: order_amount
  • dimension: VALIDITY

rangeExpectation:

  • minValue: 0.01
  • maxValue: 1000000.00
  • column: customer_email
  • dimension: CONFORMANCE

regexExpectation:

  • regex: '^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$'
  • dimension: INTEGRITY

sqlAssertionExpectation:

sqlExpression: "SELECT COUNT(*) FROM `${project}.${dataset}.orders` o LEFT JOIN `${project}.${dataset}.customers` c ON o.customer_id = c.customer_id WHERE c.customer_id IS NULL"

Core Rule Dimensions

Completeness: Ensures required fields do not contain null values ( nonNullExpectation ).

Uniqueness: Validates primary key uniqueness without duplicates ( uniquenessExpectation ).

Validity / Range: Enforces business logic boundaries on numeric or date values ( rangeExpectation ).

Conformance / Format: Validates strings against standard regex patterns (e.g., email addresses, phone numbers, postal codes).

Freshness: Asserts that the latest timestamp in the table is within an acceptable threshold (e.g., MAX(ingestion_time) >= CURRENT_TIMESTAMP() - INTERVAL 2 HOUR ).

SQL Assertions: Custom SQL queries for referential integrity across multiple tables (e.g., verifying that all foreign keys match existing primary keys).

Operational Integration

AutoDQ publishes rule execution results directly to Cloud Monitoring as custom metrics. If violation counts exceed configured thresholds, Dataplex triggers alerts via Cloud Monitoring notification channels (Slack, PagerDuty, email) and exports detailed violation audit records to BigQuery for triage.

5. End-to-End Data Lineage Tracking

Regulatory mandates (such as BCBS 239, HIPAA, and GDPR) require organizations to demonstrate complete provenance of financial and sensitive metrics. Dataplex integrates natively with the Google Cloud Data Lineage API to automatically capture and visualize transformation graphs. +───────────────────────────────────────────────────────────────────────────────────────────+

  • | DATAPLEX AUTOMATED DATA LINEAGE |
  • +───────────────────────────────────────────────────────────────────────────────────────────+
  • | |
  • | [ Ingestion: Datastream / Cloud Storage ] |
  • | │ |
  • | ▼ |
  • | (Table: raw_crm.customers_landing) |
  • | │ |
  • | ▼ [ BigQuery Query / Composer Task: clean_customers ] |
  • | (Table: staging_clean.dim_customers) |
  • | │ |
  • | ├─────────────────────────────────┐ |
  • | ▼ ▼ |
  • | [ Dataflow Pipeline: enrich_orders ] [ BigQuery SQL: aggregate_revenue ] |
  • | │ │ |
  • | ▼ ▼ |
  • | (Table: marts.customer_revenue) (Table: marts.executive_kpis) |
  • | │ │ |
  • | ▼ ▼ |
  • | [ Looker Business Dashboard ] [ Vertex AI Model Training Feature Store ] |
  • +───────────────────────────────────────────────────────────────────────────────────────────+
  • Automatic Lineage Capture Mechanisms

Data Lineage requires zero manual instrumentation in application code for standard Google Cloud processing engines:

BigQuery: Automatically captures lineage for SQL queries, DDL statements ( CREATE TABLE ... AS SELECT ), DML statements ( MERGE , INSERT INTO ), and table copy jobs.

Cloud Dataflow: Apache Beam jobs compiled with Google-provided templates or Dataflow SDK automatically report read/write PCollection steps to the Lineage API.

Cloud Composer / Airflow: Cloud Composer automatically exports DAG run dependencies and task instance transformations directly to Dataplex.

Dataproc: Open-source Spark jobs leverage the Spark Lineage plugin to emit transformation events directly into Dataplex.

Operational Use Cases for Lineage

Root-Cause Analysis: When an executive Looker dashboard displays an unexpected revenue dip, engineers inspect the lineage graph backward from the BI table to pinpoint which upstream Dataflow pipeline or raw ingestion job corrupted the data.

Impact Analysis: Before deprecating a column or modifying a table schema in staging_clean , engineers inspect downstream lineage dependencies to identify which dashboards, reports, or Vertex AI models will be broken by the migration.

Compliance and Auditing: Demonstrates to regulatory auditors exactly how sensitive data fields (e.g., personally identifiable information) flow from landing storage buckets into downstream training datasets or external partner feeds.

6. Security Governance and Policy Propagation

Dataplex unifies security management across distributed storage by propagating centralized policies down to underlying physical assets:

Hierarchical IAM: Defining roles/dataplex.dataReader or custom security policies at the Lake level automatically grants appropriate permissions across all underlying BigQuery datasets and Cloud Storage buckets within that lake, eliminating the need to maintain hundreds of individual bucket ACLs.

Attribute-Based Access Control (ABAC): Dataplex integrates with BigQuery Policy Tags to enforce column-level security and dynamic data masking (e.g., hashing credit card numbers or masking email addresses) across both native BigQuery tables and BigLake external tables.

Integration with Sensitive Data Protection (Cloud DLP): Dataplex automated profiling tasks can trigger Cloud DLP inspection templates to automatically identify sensitive data columns (SSNs, medical codes) and tag them with appropriate governance classifications without manual human intervention.

7. Architectural Anti-Patterns and Exam Traps

Operational Scenario Architectural Anti-Pattern Correct Google Cloud Architecture

Physical Data Duplication for Governance ; A company copies 500 TB of Cloud Storage data across 20 projects into a single centralized bucket in an attempt to enforce unified security policies. Moving physical data to achieve governance centralization, resulting in duplicated storage costs, sync latency, and broken pipelines. Use Dataplex Lakes and Zones . Attach existing distributed buckets and datasets as logical Assets without moving physical bytes. Dataplex enforces centralized IAM and discovery virtually.

Relaxed Schema in Curated Analytics Zones ; A data team directs raw JSON event streams with unvalidated, evolving structures directly into a Curated Zone, causing downstream Looker reports to crash on unexpected types. Using Curated Zones as initial landing areas with permissive schema policies. Direct raw, unvalidated ingestion into a Raw Zone with permissive schema inference. Transform, clean, and validate data before writing standardized Parquet or BigQuery tables into the Curated Zone .

Custom GCE Daemons for Data Quality ; An engineer deploys a fleet of Compute Engine VMs running Python cron scripts that execute SELECT COUNT(*) queries to check for null values across 200 BigQuery tables. Building and maintaining custom VM infrastructure for routine data quality validation. Implement Dataplex Automated Data Quality (AutoDQ) using declarative YAML specifications. AutoDQ runs serverlessly inside BigQuery and exports metrics directly to Cloud Monitoring.

Manual Documentation of Pipeline Provenance ; A compliance team attempts to satisfy GDPR auditing requirements by having engineers manually document table transformations and dependencies in Google Sheets. Relying on static, error-prone manual documentation for pipeline dependencies. Enable Dataplex Data Lineage . Google Cloud automatically records and renders end-to-end transformation graphs across BigQuery, Dataflow, Cloud Composer, and Dataproc.

Loading diagram... Dataplex Mesh Governance, Discovery, Quality Gates, and End-to-End Lineage Flow