Study guide

3.3 Analytics Hub and Federated Computational Governance

Google Cloud Certified Professional Data Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Security, IAM, and Data Protection

1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking

2 Chapter 2: Reliability and Compliance Architecture

2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack

3 Chapter 3: Cross-Cloud and Federated Governance

3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance

4 Chapter 4: Data Ingestion and Pipeline Planning

4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion

5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub

5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics

6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow

6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows

7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics

7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management

8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion

8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage

9 Chapter 9: Storage Selection and Cloud Storage Architecture

9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools

10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL

10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery

11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore

11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections

12 Chapter 12: Enterprise Data Warehousing with BigQuery

12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions

13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery

13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling

14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML

14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles

15 Chapter 15: Operations, Orchestration, and Data Governance

15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services

3.3 Analytics Hub and Federated Computational Governance

Key Takeaways Explain This Topic AI

Dataplex implements federated computational governance by establishing a logical data mesh hierarchy of Lakes, Zones (Raw and Curated), and Assets over distributed Cloud Storage and BigQuery resources.

Raw zones accept unstructured, semi-structured, or multi-format data with schema evolution, while Curated zones enforce structured formats (Parquet, ORC, BigQuery native) and strict quality standards.

Dataplex automates data discovery, metadata harvesting, and schema registration into Dataplex Universal Catalog and BigQuery, enabling automated data profiling and declarative data quality SLAs.

Analytics Hub enables decentralized, zero-copy data sharing through Data Exchanges and Listings, allowing subscribers to query live publisher data in place via linked datasets without data duplication.

In Analytics Hub linked datasets, subscribers pay for their own query execution compute (slots) while publishers pay only for underlying storage, eliminating compute noisy-neighbor problems and cost leakage.

Last updated: September 2026

3.3 Analytics Hub and Federated Computational Governance

Exam Focus: The Professional Data Engineer exam rigorously tests your understanding of Data Mesh principles , Dataplex logical architectures (Lakes, Zones, and Assets), declarative data quality SLA rules , and Analytics Hub zero-copy data sharing . You must be able to design multi-tenant governance models that isolate compute costs, enforce centralized compliance, and empower decentralized domain teams.

As organizations grow in scale and complexity, centralized enterprise data architectures encounter severe organizational bottlenecks. In a traditional centralized data lake or data warehouse model, a single centralized data engineering team is responsible for ingesting, transforming, modeling, and securing data for every business unit (e.g., Marketing, Supply Chain, Finance, Risk). This centralized model breaks down because:

Domain Disconnect: Centralized data engineers lack deep contextual knowledge of domain-specific business rules, leading to misunderstandings, delayed schema adjustments, and brittle transformation pipelines.

Monolithic Bottlenecks: Every new data request, schema migration, or access control change must wait in an enterprise-wide queue.

Governance Drift: Shadow data systems emerge across disparate cloud projects, undermining compliance, auditability, and regulatory posture.

To overcome these hurdles, modern data organizations adopt the Data Mesh architectural paradigm. Data Mesh is built upon four core pillars:

Domain-Driven Ownership: Business units that understand the data own and manage their analytical data as a product.

Data as a Product: Data products must be discoverable, addressable, trustworthy, secure, and interoperable across domain boundaries.

Self-Serve Data Infrastructure: A central platform team provides automated, cloud-native tools that allow domain teams to ingest, curate, and share data without managing underlying infrastructure.

Federated Computational Governance: Centralized automated policies, compliance guardrails, and security standards are programmatically enforced across decentralized domains.

In Google Cloud, Dataplex and Analytics Hub provide the foundational enterprise infrastructure for implementing federated computational governance and self-serve data sharing.

Dataplex Architecture: Lakes, Zones, and Assets

Dataplex is an intelligent data management platform that provides a unified logical abstraction over physically distributed data residing in Google Cloud Storage buckets and BigQuery datasets across multiple projects. It allows organizations to organize, monitor, and govern data centrally without physically moving or copying files. +-------------------------------------------------------------------------+

  • | ENTERPRISE ORGANIZATION |
  • +-------------------------------------------------------------------------+
  • |
  • v
  • +-------------------------------------------------------------------------+
  • | DATAPLEX LAKE |
  • | Logical Domain Boundary (e.g., Supply Chain Domain / Project: sc-data)|
  • +-------------------------------------------------------------------------+
  • | |
  • v v
  • +-----------------------------------+ +---------------------------------+
  • | RAW ZONE | | CURATED ZONE |
  • | - Tolerates schema drift | | - Strictly structured formats |
  • | - Multi-format ingestion | | (Parquet, ORC, BQ tables) |
  • | - Landing data & staging logs | | - Cleaned, validated, modeled |
  • +-----------------------------------+ +---------------------------------+
  • | |
  • v v
  • +-----------------------------------+ +---------------------------------+
  • | DATAPLEX ASSETS | | DATAPLEX ASSETS |
  • | - GCS Bucket: gs://sc-raw-drop/ | | - GCS Bucket: gs://sc-curated/ |
  • | - BQ Dataset: sc_staging_landing | | - BQ Dataset: sc_clean_orders |
  • +-----------------------------------+ +---------------------------------+
  • Dataplex Organizational Hierarchy

Dataplex Lake: Represents a high-level logical container that maps to a specific business domain, organizational unit, or regulatory perimeter (e.g., marketing-analytics-lake , clinical-trials-lake ). A lake can span multiple Google Cloud projects.

Data Zones: Subdivisions within a lake that categorize data based on readiness, structure, and lifecycle maturity. Zones establish common security, discovery, and governance boundaries.

Raw Data Zone: Designed for data ingestion and landing. Raw zones tolerate semi-structured and unstructured data, accommodate schema drift, and support varied formats such as JSON, CSV, text, and binary files. Raw zones are typically used for transient staging or landing pipelines.

Curated Data Zone: Reserved for cleansed, validated, and modeled data ready for downstream analytical consumption. Curated zones enforce strict format requirements (primarily columnar Parquet, ORC, or native BigQuery tables). Data in curated zones must adhere to predefined schemas and pass automated data quality validations.

Data Assets: The actual underlying storage resources attached to a zone. An asset references either a Google Cloud Storage bucket or a BigQuery dataset. Data remains physically in its home project and bucket/dataset; Dataplex merely attaches to it logically.

Core Architecture Constraint: A physical Cloud Storage bucket or BigQuery dataset can be attached as an asset to only one Dataplex data zone at any given time. It cannot be duplicated across multiple zones.

Automated Discovery, Profiling, and Data Quality SLAs

Dataplex continuously inspects, catalogs, and audits registered assets through three automated background capabilities:

1. Automated Metadata Discovery

When a Cloud Storage bucket is attached to a Dataplex zone, the Dataplex metadata discovery service automatically crawls the bucket at scheduled intervals. It:

Ingests directory structures and infers hive-style partition layouts (e.g., year=2026/month=09/day=15/ ).

Parses file headers and schemas for Parquet, ORC, Avro, and delimited text files.

Automatically registers external tables in the BigQuery metastore and entries in Dataplex Universal Catalog.

Detects and tracks schema evolution over time, logging newly introduced or deprecated columns.

2. Automated Data Profiling

Data engineers must understand the statistical shape and health of incoming data before feeding production pipelines. Dataplex Data Profiling runs serverless analytical scans over BigQuery tables and Cloud Storage assets to calculate:

Column-level statistical distributions (minimum, maximum, average, median, quartiles).

Null counts, null percentages, and distinct value cardinality.

Uniqueness ratios and frequency distributions for categorical fields.

Data profiling jobs execute using Google-managed serverless compute, publishing profile metrics to the Dataplex console and Cloud Logging without consuming user-managed Dataproc clusters.

3. Declarative Data Quality and SLA Enforcement

Data reliability requires continuous validation against explicit Service Level Agreements (SLAs). Dataplex Data Quality allows data owners to author declarative data quality rule sets using YAML syntax or the Google Cloud Console. These rules validate data integrity and publish pass/fail scorecards. # Dataplex Data Quality Specification Example: Orders Validation

metadata:

version: 1

ruleBindings:

  • entityURI: "//bigquery.googleapis.com/projects/enterprise-sales/datasets/curated_orders/tables/orders"
  • column: order_id

rules:

  • non_null_check
  • unique_check
  • entityURI: "//bigquery.googleapis.com/projects/enterprise-sales/datasets/curated_orders/tables/orders"
  • column: total_amount

rules:

- range_check:

  • min_value: 0.01
  • max_value: 1000000.00
  • entityURI: "//bigquery.googleapis.com/projects/enterprise-sales/datasets/curated_orders/tables/orders"
  • column: order_status

rules:

- set_expectation:

  • values: ["PENDING", "PROCESSING", "SHIPPED", "DELIVERED", "CANCELLED"]
  • entityURI: "//bigquery.googleapis.com/projects/enterprise-sales/datasets/curated_orders/tables/orders"

rules:

- sql_assertion:

  • sql_statement: "SELECT COUNT(*) FROM `${target_dataset}.${target_table}` WHERE tax_amount > total_amount"
  • threshold: 0

Dataplex compiles these YAML rules into serverless execution jobs (using Cloud Dataflow or Dataproc Serverless). When rules fail, Dataplex updates the table's quality scorecard, writes audit events to Cloud Logging, and triggers Cloud Monitoring alerts to page on-call data engineers before poisoned data enters downstream ML or financial reporting models.

Analytics Hub: Decentralized, Zero-Copy Data Sharing

While Dataplex governs data within and across enterprise boundaries, Analytics Hub provides the self-serve marketplace and consumption engine for sharing data products across projects, organizational units, and external commercial partners. +-------------------------------------------------------------------------+

  • | DATA PUBLISHER (Finance Domain) |
  • | - Source BigQuery Dataset: finance_prod.quarterly_revenue |
  • | - Authorized Views / Dynamic Masking applied |
  • +-------------------------------------------------------------------------+
  • |
  • v
  • +-------------------------------------------------------------------------+
  • | ANALYTICS HUB DATA EXCHANGE |
  • | Private Enterprise Exchange / Commercial Listing |
  • | - Listing: "Corporate Quarterly Revenue by Segment" |
  • +-------------------------------------------------------------------------+
  • |
  • Subscribe (Zero Data Duplication)
  • |
  • v
  • +-------------------------------------------------------------------------+
  • | DATA SUBSCRIBER (Marketing Domain) |
  • | - Linked Dataset: marketing_dw.linked_finance_revenue |
  • | - Live query reads publisher's storage in place |
  • | - Subscriber pays for BigQuery query compute slots |
  • | - Publisher pays only for underlying table storage |
  • +-------------------------------------------------------------------------+
  • Core Architecture Components

Data Exchange: A secure, centralized registry where data publishers create listings and consumers browse available datasets. Exchanges can be Private (scoped to an organization's internal IAM perimeter) or Public (accessible across all Google Cloud customers, including commercial data marketplaces).

Listings: Metadata descriptions of shared data products published to an exchange. A listing points to either a native BigQuery dataset, a BigLake table, or an Authorized View (which can encapsulate row-level filtering and column redaction).

Linked Datasets: When a consumer subscribes to an Analytics Hub listing, Analytics Hub provisions a read-only Linked Dataset in the subscriber's chosen Google Cloud project.

Zero-Copy Query Pushdown and Workload Isolation

The architectural brilliance of Analytics Hub lies in its zero-copy data sharing model:

No Physical Replication: Subscribing to a listing does not copy or replicate any underlying data blocks across projects. The linked dataset is a symbolic, read-only reference pointing to the publisher's live BigQuery tables.

Zero ETL Lag: Any update, insertion, or deletion committed to the publisher's dataset is instantly visible to subscribers in real time without batch ETL synchronization.

Complete Compute Cost and Workload Isolation: When the subscriber executes a query against the linked dataset, the query compute execution runs entirely on the subscriber's compute slots (or on-demand query billing). The publisher's production slot capacity and database performance are completely isolated and immune to noisy-neighbor contention.

Storage Cost Separation: The publisher pays for the base storage of the single underlying dataset. The subscriber incurs zero additional storage costs.

Comparative Evaluation: Enterprise Data Sharing Patterns

Sharing Mechanism Data Duplication Compute Isolation Setup & Management Overhead Cross-Organization Support Security & Governance

Analytics Hub Linked Datasets Zero (queries read in place) Complete (subscriber pays query compute) Low (managed self-serve portal) Full (native cross-org OIDC/IAM) Centralized (publisher policies & tags persist)

Direct Cross-Project Dataset IAM Zero Partial (requires project-level dataset grants) High (complex IAM maintenance across projects) Limited (requires individual IAM bindings) Fragile (brittle IAM drift across teams)

Authorized Views / Datasets Zero Partial (views must run in publisher's project or linked) Moderate (requires SQL view maintenance) Supported High (restricts underlying columns and rows)

ETL Pipeline Replication (Dataflow/DTS) High (100% duplicate data) Complete (separate environments) Very High (monitoring, failures, lag, code) Supported (via bucket/API transfers) Poor (duplicate copies increase exfiltration surface)

  • Concrete Exam Scenario
  • Scenario: Decentralized Healthcare Data Mesh Platform

A national healthcare network operates under a Data Mesh model across four distinct business domains:

Clinical Operations: Generates raw electronic health record (EHR) logs and HL7 messages.

Pharmacy Management: Curates medication dispensing records and supply chain inventory.

Genomics Research: Analyzes DNA sequencing BAM/VCF files and clinical trial metrics.

Executive Finance & Billing: Models reimbursement claims, hospital overhead, and insurance billing.

The Chief Information Security Officer (CISO) and Enterprise Data Architect establish four non-negotiable requirements:

Raw, unstructured clinical logs must be quarantined and cannot be queried by business users until validated.

All curated tables must pass automated data quality validations (null checks on Patient ID, range checks on dosage amounts) with automated alerting.

The Genomics team must be able to query Pharmacy's curated medication tables in real time without copying data, and Genomics queries must never impact Pharmacy's production database compute capacity.

Patient PII must be masked automatically for unauthorized researchers.

The Certified Data Engineer Architecture

Deploy Dataplex Lakes by Domain: Create four Dataplex Lakes corresponding to each domain: lake-clinical , lake-pharmacy , lake-genomics , and lake-finance .

Establish Raw and Curated Zones:

In lake-clinical , establish a Raw Data Zone attaching the landing GCS buckets. Only clinical ingestion pipelines have access to this zone. Automated discovery scans incoming files.

In lake-pharmacy , establish a Curated Data Zone attaching the BigQuery datasets containing validated dispensing records. Disallow non-columnar or unstructured assets.

Automate Data Quality SLAs in Dataplex: In lake-pharmacy , configure Dataplex Data Quality tasks on the dispensing table. Enforce rules checking that patient_id IS NOT NULL , dispensed_quantity > 0 , and drug_ndc_code matches the national drug code regex. Configure alerts to notify DataOps if quality scores drop below 99.5%.

Publish Curated Products via Analytics Hub: The Pharmacy team creates an internal private Data Exchange and publishes a Listing for curated_medication_dispensing . PII columns are tagged with Dataplex Policy Tags enforcing dynamic masking.

Zero-Copy Subscription by Genomics: The Genomics team subscribes to the listing, creating a Linked Dataset in their genomics-analytics-prod project. Genomics researchers query the data using their own project's BigQuery slot reservations. The data remains physically in Pharmacy's storage, updates reflect instantaneously, and zero patient data is duplicated across projects.

Common Exam Pitfalls and Gotchas

Pitfall 1: Attaching a Bucket to Multiple Dataplex Zones: A Cloud Storage bucket or BigQuery dataset cannot belong to multiple Dataplex zones simultaneously. If an exam question describes dividing subfolders of a single bucket between a raw zone and a curated zone, that design is invalid; separate buckets (or separate datasets) are required.

Pitfall 2: Confusing Raw and Curated Zone Capabilities: Placing unstructured files (e.g., raw JPEG medical scans or unstructured text files) into a Curated Data Zone will cause Dataplex validation errors. Curated zones strictly require structured, performance-optimized formats.

Pitfall 3: Assuming Analytics Hub Incurs Storage Costs for Subscribers: Subscribers to an Analytics Hub listing do not pay for storage; they only pay for the query compute (slots or bytes scanned) consumed when executing queries against the linked dataset.

Pitfall 4: Granting Direct Project IAM Instead of Using Analytics Hub: When an exam question requires sharing datasets with dozens of external partner companies or autonomous internal domains with complete query compute isolation, granting direct BigQuery dataset IAM permissions is an administrative anti-pattern. Analytics Hub linked datasets provide automated lifecycle management, access auditing, and zero-copy workload isolation.

Loading diagram... Federated Data Mesh Governance with Dataplex and Analytics Hub