ExamFinch.com My Study

Study guide

11.3 Data Governance, Cataloging, and Discovery with Dataplex

Practice this exam

Google Cloud Professional Data Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 About the Google Cloud Professional Data Engineer Exam 0.2 Study Strategy & Cloud Data Engineering Methodology

2 Chapter 1: Designing for Security and Compliance

1.1 IAM Roles, Service Accounts, and Least Privilege for Data 1.2 Data Encryption, Key Management, and Privacy with Cloud DLP 1.3 Compliance, Data Sovereignty, and Geographic Placement 1.4 Multi-Environment Architecture: Development, Staging, and Production

3 Chapter 2: Designing for Reliability and Resilience

2.1 High Availability, Disaster Recovery, and Failover Patterns 2.2 Fault Tolerance and ACID Guarantees in Distributed Systems 2.3 Data Validation, Cleansing, and Quality Assurance with Dataform 2.4 AI-Assisted Data Preparation and LLM Query Generation

4 Chapter 3: Architecture Patterns and Flexibility

3.1 Modern Data Architectures: Data Lake, Warehouse, Lakehouse, and Mesh 3.2 Data Migration Strategies and Hybrid/Multi-Cloud Integration 3.3 Database Migration Service and Heterogeneous Database Migrations

5 Chapter 4: Ingestion Architectures: Pub/Sub and Streaming Sources

4.1 Cloud Pub/Sub Core Architecture and Ingestion 4.2 Pub/Sub Advanced Patterns: Dead-Lettering, Ordering, and Schemas 4.3 Change Data Capture (CDC) with Datastream and Batch Ingestion

6 Chapter 5: Stream and Batch Processing with Apache Beam and Cloud Dataflow

5.1 Apache Beam Core Concepts and Pipeline Design 5.2 Advanced Windowing, Watermarks, and Triggers 5.3 Dataflow Pipeline Optimization, Scaling, and Templates

7 Chapter 6: Data Transformation with Dataproc, Data Fusion, and Workflow Orchestration

6.1 Apache Spark and Hadoop with Cloud Dataproc 6.2 Visual Data Pipelines with Cloud Data Fusion 6.3 Workflow Orchestration with Cloud Composer 6.4 AI Data Enrichment inside Data Pipelines

8 Chapter 7: Storage Service Selection and Cloud Storage Architecture

7.1 Google Cloud Storage Service Selection Decision Matrix 7.2 Cloud Storage Architecture, Classes, and Lifecycle Management

9 Chapter 8: Deep Dive into Google BigQuery Storage

8.1 BigQuery Storage Architecture and Data Organization 8.2 Partitioning and Clustering for Performance and Cost 8.3 BigLake and Hybrid Querying across Clouds 8.4 Data Warehouse Modeling: Star Schemas, Denormalization, and Slowly Changing Dimensions

10 Chapter 9: Operational and NoSQL Databases: Bigtable and Cloud Spanner

9.1 Cloud Bigtable Architecture and Row Key Design 9.2 Cloud Bigtable Schema Management and Performance Tuning 9.3 Cloud Spanner: Globally Scalable Relational Database

11 Chapter 10: BigQuery Query Performance and Optimization

10.1 BigQuery SQL Tuning and Performance Best Practices 10.2 BigQuery BI Engine and Materialized Views 10.3 Data Sharing and Analytics Hub 10.4 Preparing Data for Visualization and BI Tool Connectivity

12 Chapter 11: Machine Learning Readiness and Data Governance

11.1 Machine Learning with BigQuery ML 11.2 Feature Engineering and Data Preparation for AI/ML 11.3 Data Governance, Cataloging, and Discovery with Dataplex 11.4 Unstructured Data, Embeddings, and Retrieval-Augmented Generation

13 Chapter 12: Site Reliability Engineering and Monitoring for Data Systems

12.1 SRE Principles for Data: SLIs, SLOs, and SLAs 12.2 Observability: Cloud Monitoring, Cloud Logging, and Alerting 12.3 Troubleshooting Data Pipelines and Workload Failures

14 Chapter 13: Workload Automation, CI/CD, and Cost Optimization

13.1 CI/CD and Version Control for Data Pipelines 13.2 BigQuery Cost Management and Capacity Planning 13.3 Holistic Cost Optimization across GCP Data Services 13.4 Query Workload Management: Interactive vs. Batch Jobs, Queues, and Quotas

11.3 Data Governance, Cataloging, and Discovery with Dataplex

Key Takeaways Explain This Topic AI

Google Cloud Dataplex provides an intelligent data fabric that unifies distributed data across Cloud Storage buckets and BigQuery datasets into logical Lakes, Zones (Raw vs Curated), and Assets without moving or copying physical data.

Dataplex automates enterprise metadata discovery by continuously crawling registered assets, inferring structured schemas, detecting Hive-style directory partitioning, and publishing external tables directly to BigQuery and the catalog.

Data Catalog was deprecated on February 3, 2025 and shut down on June 1, 2026; its replacement is Dataplex Universal Catalog, renamed Knowledge Catalog on April 10, 2026, where tag templates map to global aspect types and tags map to optional aspects, while policy tags and policy tag taxonomies are explicitly not deprecated.

Column-level security is enforced in BigQuery through Dataplex Policy Tag taxonomies: users with roles/datacatalog.categoryFineGrainedReader access raw plaintext, while roles/datacatalog.maskedUser enforces dynamic runtime data masking (Default, Nullify, Hash SHA-256, Regex) without breaking analytical SQL queries.

Dataplex Automated Data Quality (Auto DQ) executes serverless, declarative quality rules (null checks, uniqueness, range boundaries, regex validation) against BigQuery and Cloud Storage, while automated data lineage visualizes complete end-to-end data provenance across BigQuery, Dataflow, and Dataproc.

Last updated: September 2026

11.3 Data Governance, Cataloging, and Discovery with Dataplex

Important

Data governance, fine-grained access control, and metadata management are core pillars of the Google Cloud Professional Data Engineer exam. Expect in-depth questions on how to architect decentralized data lakes using Dataplex , how to implement column-level security and dynamic data masking using Policy Tags in BigQuery, how to configure Row-Level Security policies, and how to automate data quality validation and lineage tracking without operating custom compute infrastructure.

As enterprise data architectures expand, organizations transition from monolithic data warehouses toward distributed data mesh topologies. Modern enterprises manage petabytes of data distributed across thousands of Cloud Storage buckets (containing raw Parquet, ORC, JSON, and Avro files) and hundreds of BigQuery datasets across multiple projects and regions.

This distribution introduces severe governance vulnerabilities: data silos proliferate, schema drift goes undetected, sensitive Personally Identifiable Information (PII) is copied without audit trails, and data quality degrades. Google Cloud Dataplex and its Knowledge Catalog (the catalog the exam guide calls Dataplex Catalog) provide the centralized intelligent data fabric required to govern, secure, and discover distributed data assets at scale.

Dataplex Hierarchical Architecture: Lakes, Zones, and Assets

Dataplex organizes distributed physical storage resources into a structured, domain-oriented logical hierarchy without requiring physical data movement, consolidation, or replication. +-------------------------------------------------------------------------+

  • | Dataplex Lake (Domain) |
  • | e.g., 'sales-domain-lake' or 'customer-360' |
  • +-------------------------------------------------------------------------+
  • |
  • +---------------------------+---------------------------+
  • | |
  • v v
  • +---------------------------------+ +---------------------------------+
  • | Raw Data Zone | | Curated Data Zone |
  • | * Ingestion landing area | | * Cleaned, analytical data |
  • | * Accepts varied raw formats | | * Enforces schema standards |
  • | * Relaxed validation rules | | * Optimized Parquet / BQ tables|
  • +---------------------------------+ +---------------------------------+
  • | |
  • v v
  • +---------------------------------+ +---------------------------------+
  • | Assets Mapped | | Assets Mapped |
  • | • Cloud Storage Bucket | | • BigQuery Analytical Dataset |
  • | (gs://raw-telemetry-landing) | | (project:curated_sales_mart) |
  • | • Cloud Storage Bucket | | • Cloud Storage Curated Parquet|
  • | (gs://partner-csv-drop) | | (gs://curated-lake-parquet) |
  • +---------------------------------+ +---------------------------------+
  • 1. Lakes (Business Domains)

A Lake represents the top-level logical container in Dataplex, typically aligned with a business domain, functional unit, or regulatory jurisdiction (e.g., retail-operations-lake , financial-risk-lake , healthcare-analytics-lake ). Lakes define the primary administrative boundary for resource policy, billing assignment, and domain-wide security defaults.

2. Zones (Storage Categories and Standards)

Inside a Lake, data assets are organized into Zones . Zones categorize data according to its processing maturity, data readiness, and access requirements. Dataplex provides two foundational zone types:

Raw Zones: Designed for landing zones and raw ingestion pipelines. Raw zones accept unstructured, semi-structured, and structured data in arbitrary formats (raw JSON, CSV, logs). Validation rules can be relaxed, allowing upstream systems to land files without immediate schema failure.

Curated Zones: Designed for processed, cleaned, and standardized analytical data. Curated zones enforce specific storage formats (such as Parquet, ORC, or native BigQuery managed tables). Data in curated zones must adhere to strict schema definitions and partitioning standards, making it immediately queryable by business analysts and BI dashboards.

3. Assets (Physical Storage Mapping)

An Asset maps an underlying physical Google Cloud storage resource directly into a Dataplex Zone. Assets can point to:

A Cloud Storage bucket (or bucket prefix).

A BigQuery dataset .

Zero Data Movement: Attaching a Cloud Storage bucket or BigQuery dataset as an asset does not move, copy, or transform the underlying bytes. The data remains in its native storage tier (Colossus or GCS), retaining its existing performance characteristics, while Dataplex overlays unified governance, automated discovery, metadata management, and centralized security access policies.

Automated Metadata Discovery and Cataloging

Manually documenting schemas, tables, and partitions across thousands of Cloud Storage buckets is impossible at enterprise scale. Dataplex incorporates a fully managed, serverless metadata discovery engine.

When a Cloud Storage bucket is registered as an asset, the Dataplex discovery service automatically:

Scans Storage Files: Inspects structured and semi-structured files (Parquet, ORC, Avro, CSV, JSON) stored within the bucket.

Infers Schemas: Analyzes file headers and records to infer column names, nested record definitions, and data types (such as INT64 , TIMESTAMP , STRING ).

Extracts Hive-Style Partitions: Parses directory path patterns matching Hive partitioning conventions (e.g., gs://bucket/sales/year=2026/month=09/day=14/data.parquet ) and identifies the partition keys and data types.

Publishes External Tables to BigQuery & the Catalog: Automatically creates and updates external table definitions in the BigQuery metastore and entries in the Dataplex Catalog. Analysts can query sales immediately using standard SQL without manually executing CREATE EXTERNAL TABLE statements.

Detects Schema Evolution and Drift: Tracks when new columns are added or data types change across files, alerting administrators when schema drift or file corruption occurs.

Cataloging: From Data Catalog to Knowledge Catalog

The exam guide lists "Dataplex, Dataplex Catalog" among the tools for building a data platform, and this is an area where the product naming moved faster than most study material. Get the current state straight before anything else:

Data Catalog — the standalone catalog product — was deprecated on February 3, 2025 and shut down on June 1, 2026. Its business glossary was deprecated on May 19, 2025 with the same June 1, 2026 shutdown.

Its replacement is Dataplex Universal Catalog , which as of April 10, 2026 is called Knowledge Catalog . The API, client library, CLI, and IAM names are unchanged, which is why the exam guide's "Dataplex Catalog" wording, the dataplex.googleapis.com API, and the roles/dataplex.* IAM roles all still line up.

Policy tags and policy tag taxonomies are explicitly not deprecated. They remain the mechanism for BigQuery column-level access control, covered in the next part of this section. Do not let the catalog transition confuse you into thinking column-level security changed — it did not.

The Concept Mapping You Must Know

Older material describes Data Catalog's tag model. Google documents a direct mapping into Knowledge Catalog's aspect model:

Data Catalog Concept Knowledge Catalog Equivalent Note

Tag template (regional) Aspect type (global) Aspect type definitions are replicated to all regions, preserving cross-regional references

  • Tag Optional aspect Attached instance of structured metadata on an entry
  • Standard concepts such as Schema System aspect types Provided by Knowledge Catalog rather than user-defined

Custom entries and entry groups Entries and entry groups Imported as read-only during the preparatory phase, then moved to read-write

If an exam question uses "tag template," read it as "aspect type." If it uses "policy tag," it means exactly what it has always meant.

Search Across the Catalog

The catalog indexes BigQuery tables and views, Dataplex assets, Pub/Sub topics, Spanner databases, and Cloud Storage data, and supports faceted search across physical resource names, column names, project IDs, and attached metadata:

  • Search by attached metadata: tag:DataClassification=RESTRICTED
  • Search by column: column:credit_card_number
  • Search by resource type and parent: type=TABLE parent:project.sales_dataset
  • Structured Metadata Instead of Free Text

Rather than relying on unstructured descriptions, governance is enforced with a formal schema — a tag template in the legacy vocabulary, an aspect type in the current one: Aspect type: ComplianceGovernance

  • DataClassification (ENUM: PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED)
  • ContainsPII (BOOLEAN: TRUE, FALSE)
  • DataOwner (STRING: security-team@example.com)
  • RetentionPeriodDays(INT64: 2555)
  • LastAuditedDate (TIMESTAMP: 2026-09-14 00:00:00 UTC)

Structured metadata is what makes catalog search a governance control rather than a documentation exercise: an auditor can enumerate every asset where ContainsPII = TRUE in one query, which is impossible when the same fact lives in a free-text description.

Separately, the catalog distinguishes metadata that everyone with read access can see from metadata gated behind its own fine-grained IAM permissions . That separation is what lets security, audit, and compliance teams attach sensitive annotations — vulnerability assessments, legal-hold markers, detailed PII findings — to an asset without exposing those annotations to every analyst who can query the underlying table.

Column-Level Security and Dynamic Data Masking

Enterprise data warehouses frequently store sensitive columns (such as Social Security Numbers, credit card numbers, salaries, or email addresses) alongside non-sensitive operational attributes in the same table. Granting table-level access exposes sensitive PII, while duplicating tables into "sanitized" versions creates maintenance overhead and data sprawl.

BigQuery addresses this using Policy Tags managed through Dataplex Catalog taxonomy hierarchies (policy tags and policy tag taxonomies are explicitly not deprecated).

  • Step-by-Step Architecture for Policy Tag Enforcement 1. Define Taxonomy & Policy Tags (Dataplex Catalog)
  • Taxonomy: PII_Taxonomy
  • ├── High_Sensitivity (Taxonomy Node)
  • │ └── SSN_Policy_Tag
  • └── Low_Sensitivity (Taxonomy Node)
  • └── Email_Policy_Tag
  • 2. Associate Policy Tag with BigQuery Column
  • CREATE TABLE customer_records (
  • customer_id STRING,
  • signup_date DATE,
  • email_address STRING OPTIONS(policy_tags=['projects/.../policyTags/Email_Policy_Tag']),
  • ssn STRING OPTIONS(policy_tags=['projects/.../policyTags/SSN_Policy_Tag'])
  • );
  • 3. Assign IAM Permissions & Masking Rules
  • Compliance Officer -> roles/datacatalog.categoryFineGrainedReader (Unmasked Plaintext)
  • Data Analyst -> roles/datacatalog.maskedUser + SHA-256 Masking Rule (Hashed Output)
  • Unprivileged User -> No Policy Tag Role (Query Fails if column referenced)
  • Dynamic Data Masking Rules

When users without plaintext access query a table, standard column-level security blocks the entire query with an authorization failure. Dynamic Data Masking allows non-privileged users to query the table successfully while masking the sensitive values dynamically at query execution time.

To enable dynamic data masking, administrators grant the user or group roles/datacatalog.maskedUser on the policy tag and configure a Data Masking Rule on the tag:

Default Masking Value: Returns the standard system default value based on the column's data type: "" (empty string) for STRING , 0 for integer types, 0.0 for floats, 0001-01-01 for DATE , and false for BOOL .

Nullify: Returns a SQL NULL for the masked column across all rows.

Hash (SHA-256): Computes a deterministic SHA-256 cryptographic hash of the input value.

Why this is critical for analytics: Because SHA-256 hashing is deterministic ( H ( x ) = H ( x ) H(x) = H(x) H ( x ) = H ( x ) ), analysts can perform COUNT(DISTINCT ssn) , GROUP BY ssn , or JOIN operations across tables on the hashed identifier without ever decrypting or viewing the real Social Security Number.

Regex Masking (Custom Pattern Masking): Applies regular expression replacement rules (e.g., masking credit card digits so only the last four are visible: XXXX-XXXX-XXXX-1234 , or masking email domains: XXX@company.com ).

Masking Option Transformation Description Supported Data Types Sample Output (Input: '123-45-6789') Enterprise Analytical Use Case

Default Masking Replaces value with data type default STRING , NUMERIC , INT64 , DATE , TIMESTAMP "" (Empty String) Preventing sensitive field display on operational dashboards

Nullify Replaces value with SQL NULL All BigQuery primitive data types NULL Zero-information exposure; calculations safely ignore null values

Hash (SHA-256) Computes deterministic SHA-256 hash string STRING , BYTES a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e Unique entity counting, joins, and aggregations without PII visibility

Regex Masking Replaces regex matches with static mask STRING ***-**-6789 Partial exposure for customer support lookup verification

Row-Level Security (RLS) in BigQuery

While column-level security protects vertical slices of a table, multi-tenant architectures and data mesh domains require restricting horizontal slices (rows) based on user identity or organizational roles. BigQuery provides Row-Level Security through row access policies.

  • Defining Row Access Policies
  • Row access policies are created using standard DDL: CREATE OR REPLACE ROW ACCESS POLICY regional_sales_filter
  • ON `enterprise_sales.orders`
  • GRANT TO ('group:emea-analysts@company.com')
  • FILTER USING (region = 'EMEA');
  • CREATE OR REPLACE ROW ACCESS POLICY us_sales_filter
  • ON `enterprise_sales.orders`
  • GRANT TO ('group:us-analysts@company.com')
  • FILTER USING (region = 'US');
  • Key Row-Level Security Characteristics for the Exam

Transparent Filtering: When a member of emea-analysts@company.com executes SELECT * FROM enterprise_sales.orders , BigQuery automatically and transparently rewrites the query execution plan to inject WHERE region = 'EMEA' . The user cannot bypass this predicate.

Full-Table Fallback Role: Users without an applicable row access policy receive zero rows by default, unless they possess roles/bigquery.filteredDataViewer or admin permissions that grant access to unrestricted rows.

Coexistence with Column-Level Security: Row access policies and policy tags operate simultaneously. A query plan filters unauthorized rows first via Row-Level Security, then masks or redacts unauthorized columns via Dynamic Data Masking.

Sensitive Data Discovery and Protection (Cloud DLP / SDP)

Before enforcing policy tags, data engineers must locate where sensitive PII resides across terabytes of untagged legacy data. Sensitive Data Protection (formerly Cloud DLP) provides automated inspection and de-identification capabilities:

Automated Profiling: Dataplex integrates directly with Sensitive Data Protection to profile BigQuery tables and Cloud Storage buckets automatically, detecting over 150 built-in infoTypes (such as US_SOCIAL_SECURITY_NUMBER , CREDIT_CARD_NUMBER , PASSPORT , EMAIL_ADDRESS ).

De-Identification Techniques:

Masking & Redaction: Obscures characters partially or completely.

Bucketing / Generalization: Converts continuous values into coarse categories (e.g., ages to 10-year ranges) to prevent re-identification.

Cryptographic Hashing with Salt: Deterministic tokenization enabling entity correlation without storing raw identifiers.

Format-Preserving Encryption (FPE): Encrypts data while preserving original length and character set (e.g., a 16-digit credit card number encrypts into a ciphertext 16-digit number), allowing legacy schemas to process tokens without schema breakage.

Dataplex Hierarchy and Security Governance Reference

Hierarchy Level Architectural Scope Primary Purpose Underlying Mapped Resources Governance & Security Capabilities

Lake Domain / Business Unit High-level organizational boundary (e.g., Sales, HR) Contains multiple Raw and Curated Zones Domain-wide IAM policies, default CMEK encryption, central billing perimeters

Raw Zone Storage maturity: Ingestion Landing area for untransformed, heterogeneous data Cloud Storage buckets with JSON, CSV, logs Accommodates schema flexibility, captures data before validation standards

Curated Zone Storage maturity: Analytical Governed area for cleaned, structured, queryable data BigQuery datasets, Cloud Storage Parquet/ORC Strict schema validation, optimized columnar formats, automated query indexing

Asset Storage entity mapping Maps physical cloud storage into Dataplex hierarchy 1 Cloud Storage bucket OR 1 BigQuery dataset Zero-copy mapping, automated metadata discovery, partition detection

Policy Tag Column security Fine-grained column-level access control & dynamic masking BigQuery table schema columns categoryFineGrainedReader (plaintext), maskedUser (dynamic masking rules)

Row Access Policy Row security Horizontal data isolation based on identity BigQuery table rows FILTER USING (predicate) , transparent query plan filtering

Automated Data Quality (Auto DQ) and Lineage Tracking

Enterprise governance requires proving that data is reliable, timely, and untampered with.

Dataplex Automated Data Quality (Auto DQ)

Dataplex Auto DQ provides a fully managed, serverless framework for defining and evaluating data quality rules directly against BigQuery tables and Cloud Storage assets without deploying Dataproc or Dataflow pipelines.

Declarative Quality Rules: Data engineers define data quality specifications in declarative YAML or SQL syntax, including:

completeness_check : Validates that null percentages do not exceed a strict threshold.

uniqueness_check : Verifies primary key uniqueness.

range_check : Ensures values fall between acceptable minimum and maximum bounds (e.g., 0 ≤ transaction_amount ≤ 100 , 000 0 \le \text{transaction\_amount} \le 100,000 0 ≤ transaction_amount ≤ 100 , 000 ).

regex_match : Validates formatting standards (such as phone numbers, zip codes, or ISO timestamps).

custom_sql_expression : Executes arbitrary SQL assertions for complex business logic.

Automated Scheduling & Alerts: Rules execute on automated schedules (hourly, daily, or post-ingestion). Results publish aggregated Data Quality Scores directly to Dataplex Catalog, triggering Cloud Monitoring alerts or Pub/Sub notifications if quality thresholds fail.

Automated End-to-End Data Lineage Tracking

Understanding how data flows through enterprise transformations is critical for root-cause error analysis, impact analysis prior to schema migrations, and regulatory compliance (such as BCBS 239 or GDPR).

Dataplex automatically tracks and visualizes Data Lineage without requiring manual instrumenting code:

Native Integration: Tracks transformations executed by BigQuery SQL queries , Cloud Dataflow pipelines (Apache Beam), and Cloud Dataproc Spark jobs .

Visual DAG Graph: Renders an interactive directed acyclic graph in the Google Cloud console showing the source tables/buckets, intermediate transformation processes, destination tables, and downstream Looker dashboards.

Impact Analysis: When a table schema is modified or a pipeline fails, data engineers inspect downstream lineage nodes instantly to identify every report, model, or dashboard impacted by the change.

Loading diagram... Dataplex Architecture and Security Governance: Raw vs Curated Zones and Policy Tag Dynamic Data Masking