Study guide

3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control

Google Cloud Certified Professional Data Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Security, IAM, and Data Protection

1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking

2 Chapter 2: Reliability and Compliance Architecture

2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack

3 Chapter 3: Cross-Cloud and Federated Governance

3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance

4 Chapter 4: Data Ingestion and Pipeline Planning

4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion

5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub

5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics

6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow

6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows

7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics

7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management

8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion

8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage

9 Chapter 9: Storage Selection and Cloud Storage Architecture

9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools

10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL

10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery

11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore

11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections

12 Chapter 12: Enterprise Data Warehousing with BigQuery

12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions

13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery

13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling

14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML

14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles

15 Chapter 15: Operations, Orchestration, and Data Governance

15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services

3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control

Key Takeaways Explain This Topic AI

BigLake decouples analytical compute from underlying object storage across Google Cloud Storage, AWS S3, and Azure Data Lake Storage Gen2, providing a unified storage abstraction with centralized policy enforcement.

BigLake enables fine-grained access control—specifically row-level security (RLS), column-level security (CLS), and dynamic data masking—over open file formats without granting end users direct storage bucket permissions.

Open table and file format support encompasses Apache Iceberg, Delta Lake, Apache Hudi, Apache Parquet, ORC, and Avro, enabling zero-copy analytics across diverse analytical engines like BigQuery, Cloud Dataproc (Spark), and Trino.

The BigLake Metastore offers a fully managed, serverless Apache Hive Metastore (HMS)-compatible catalog that maintains synchronized schema definitions and partition statistics across multi-engine lakehouses.

Delegated authorization operates via BigQuery CLOUD_RESOURCE connections, where Google-managed service identities assume storage access roles, eliminating storage-level credential sprawl and preventing direct object exfiltration bypasses.

Last updated: September 2026

3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to architect governed, multi-tenant lakehouses using BigLake . You must master delegated authorization via CLOUD_RESOURCE connections, understand how to enforce row-level security (RLS) and column-level security (CLS) with dynamic data masking without granting users storage permissions, optimize query latency over object storage using metadata caching , and integrate open table formats like Apache Iceberg with BigLake Metastore .

Traditional data lake architectures suffer from an inherent architectural tension between accessibility and governance. When analytical engines like BigQuery, Apache Spark, Trino, or Presto query raw data stored in cloud object stores—such as Google Cloud Storage (GCS), Amazon Simple Storage Service (AWS S3), or Azure Data Lake Storage Gen2 (ADLS)—organizations have historically been forced to grant end users or analytical execution roles broad read permissions ( storage.objects.get in GCP, or s3:GetObject in AWS). This coarse-grained authorization model creates critical enterprise vulnerabilities:

Direct Access Bypass: Users with bucket-level read access can bypass analytical query controls, downloading raw underlying files (such as Parquet, CSV, or ORC) directly to their local environments, which completely circumvents analytical masking rules and row filters.

Duplicative Access Control Silos: Securing data across multiple analytical engines requires configuring duplicate access control lists, view definitions, and masking scripts in every distinct compute tool.

Compute-Storage Lock-In: Migrating or querying across clouds typically mandates expensive, time-consuming extraction, transformation, and loading (ETL) pipelines that duplicate data and bloat cloud storage costs.

BigLake resolves this architectural impasse by serving as a unified storage engine that decouples compute engines from physical storage while enforcing centralized, fine-grained access controls. BigLake presents a consistent storage abstraction over Google Cloud Storage, AWS S3, and Azure ADLS Gen2. Rather than granting end users direct identity and access management (IAM) permissions on the underlying object storage buckets, all data access is mediated through the BigLake API and BigQuery Connection Service using delegated service identities. +-------------------------------------------------------------------------+

  • | ANALYTICAL COMPUTE ENGINES |
  • | BigQuery SQL | Cloud Dataproc (Spark) | Open-Source Engines |
  • +-------------------------------------------------------------------------+
  • |
  • v
  • +-------------------------------------------------------------------------+
  • | BIGLAKE STORAGE ENGINE |
  • | - Row-Level Security (RLS) Filters - Column Policy Tags (Masking) |
  • | - Intelligent Metadata Caching - Delegated Service Identity |
  • +-------------------------------------------------------------------------+
  • |
  • v
  • +-------------------------------------------------------------------------+
  • | HETEROGENEOUS OBJECT STORAGE |
  • | Google Cloud Storage | Amazon S3 | Azure ADLS Gen2 |
  • | (Parquet, Iceberg, ORC, Avro, Delta Lake, Apache Hudi) |
  • +-------------------------------------------------------------------------+
  • BigLake Fine-Grained Access Control (FGAC) and Delegated Authorization

In standard federated queries—such as basic BigQuery external tables configured directly over Cloud Storage—BigQuery executes queries using the identity of the user running the query. As a result, the end user must possess direct storage.objectViewer or storage.objects.get permissions on the target GCS bucket. Under this legacy federation model, fine-grained access controls like column masking, column-level security (CLS), and row-level security (RLS) cannot be reliably enforced because the user can trivially circumvent BigQuery by reading the raw files with gsutil , gcloud storage , or Cloud Storage client libraries.

BigLake fundamentally changes this trust boundary through delegated authorization using a CLOUD_RESOURCE connection. When a BigLake table is defined, it is linked to a Google Cloud connection resource that encapsulates a dedicated, Google-managed service account.

Access Delegation Workflow

Underlying Bucket Permissions: The underlying storage bucket grants read access exclusively to the BigQuery Connection's service account (e.g., service-123456789@gcp-sa-bigquery-connection.iam.gserviceaccount.com ). End users and analysts are explicitly granted zero read permissions on the storage bucket.

Table and Catalog Permissions: The end user is granted bigquery.tables.getData on the BigLake table and the appropriate access roles on the BigQuery dataset.

Policy Evaluation: When the user issues a SQL query against the BigLake table, BigQuery intercepts the query, evaluates the user's identity against Dataplex Universal Catalog policy tags and row-level security predicates, and pushes the authorized projection and predicate filters down to the BigLake storage engine.

Storage Retrieval: The BigLake engine uses the connection's delegated service account to retrieve only the authorized byte ranges and partitions from the physical storage layer. [ End User / Analyst ]

│

│ 1. Submits Query: SELECT * FROM biglake_table WHERE ...

  • │ (User has bigquery.tables.getData, ZERO GCS Bucket IAM)
  • ▼
  • [ BigQuery & BigLake Engine ]
  • │
  • │ 2. Evaluates Dataplex Policy Tags (Column Masking)
  • │ 3. Applies Row-Level Access Policy (Row Filtering)
  • │ 4. Uses Connection SA (roles/storage.objectViewer)
  • ▼
  • [ Cloud Storage Bucket / S3 / ADLS ]
  • │ 5. Returns ONLY authorized byte ranges
  • Row-Level Security (RLS) on Object Storage

BigLake enables organizations to apply row-level security policies directly to data residing in GCS, S3, or ADLS without altering the underlying files. For instance, a single set of Parquet files in Cloud Storage can be filtered dynamically so that regional sales analysts only see records matching their authorized geography: -- Create a row-level access policy on a BigLake table over Cloud Storage

  • CREATE ROW ACCESS POLICY emea_analyst_filter
  • ON `enterprise_lakehouse.curated_sales_biglake`
  • GRANT TO ('group:emea-sales-analytics@example.com')
  • FILTER USING (region = 'EMEA');
  • Column-Level Security (CLS) and Dynamic Data Masking

By leveraging Dataplex taxonomy policy tags, data governors can attach classification tags (such as High_PII or Financial_Class ) to specific columns within the BigLake table schema. Depending on the caller's IAM roles:

Users with the Fine-Grained Reader role see cleartext values.

Users without the role are blocked from reading the column entirely, or see masked values if a Data Masking Rule (e.g., SHA-256 hash, default value, or partial redact) is assigned to the policy tag.

Because users have no direct storage bucket permissions, they cannot bypass this masking by downloading the Parquet files.

Open Table Formats and Storage Engine Pruning

Modern enterprise data architectures have shifted from unmanaged collections of flat files to open table formats that provide database-like ACID transactions, schema evolution, time-travel, and partition compaction directly on top of cost-effective cloud object storage. BigLake provides native support for both standard file formats and advanced open table formats:

Format Table Format Type Metadata Mechanism BigLake Optimization Support

Parquet Columnar File Format File footer metadata Column projection, dictionary filtering, byte-range vectorization

  • ORC Columnar File Format Stripes and file footers Predicate pushdown, stripe-level skipping
  • Avro Row-based File Format Embedded JSON schema Schema evolution, streaming ingestion landing

Apache Iceberg Open Table Format Hierarchical metadata snapshots ( .json , manifest-list , manifest.avro ) Native partition pruning, min/max statistics skipping, time-travel queries

Delta Lake Open Table Format JSON-based transaction log ( _delta_log/ ) File skipping via checkpoint Parquet statistics

  • Apache Hudi Open Table Format Timeline metadata, commit logs Read-optimized queries on copy-on-write tables
  • Apache Iceberg Integration

Apache Iceberg has emerged as the premier open table format for enterprise lakehouses. In an Iceberg table, metadata is organized in a tree hierarchy: table metadata files point to manifest lists, which point to individual manifest files tracking actual data files alongside column-level metrics (lower and upper bounds, null counts).

BigLake reads Iceberg metadata natively. When a query with a WHERE transaction_date BETWEEN '2026-01-01' AND '2026-01-31' predicate is executed against an Iceberg BigLake table, BigLake inspects the Iceberg manifest files, prunes non-qualifying data files at the metadata layer, and only issues I/O requests for the exact Parquet files containing matching records. This reduces physical byte scanning by orders of magnitude compared to traditional directory-listing approaches.

BigLake Metastore: Serverless Hive Metastore Replacement

In a multi-engine lakehouse where BigQuery, Cloud Dataproc (running Spark, Trino, or Presto), and external containerized query engines interact with the same data lake, maintaining a synchronized catalog is critical. Historically, organizations maintained self-hosted Apache Hive Metastore (HMS) instances on Compute Engine or Cloud SQL, which required manual patching, database scaling, high-availability maintenance, and connection proxy tuning.

The BigLake Metastore is a fully managed, serverless, HMS-compatible metadata service designed specifically for open table formats (especially Apache Iceberg). Key architectural characteristics include:

HMS API Compatibility: BigLake Metastore exposes endpoints compatible with the Apache Hive Metastore client library, allowing Dataproc Spark jobs and Trino clusters to use BigLake Metastore as their catalog without code modifications.

Unified Catalog Abstraction: Tables registered in BigLake Metastore are immediately discoverable and queryable in BigQuery, and changes committed by Spark jobs (such as partition additions or schema evolution) become visible to BigQuery without manual catalog synchronization.

Fine-Grained Catalog Access: Permissions on schemas, tables, and partitions inside the metastore are managed using Google Cloud IAM policies rather than proprietary database grants.

BigLake Table Creation Syntax and Configuration

Creating a BigLake table requires linking an external table definition to a Cloud Resource Connection. Below is the end-to-end procedural workflow for deploying a governed BigLake table over Cloud Storage Parquet files.

  • Step 1: Create the Cloud Resource Connection # Create a regional connection in the target Google Cloud region
  • bq mk --connection \
  • -location=us-central1 \
  • -project_id=enterprise-analytics-prod \
  • -connection_type=CLOUD_RESOURCE \
  • biglake-gcs-connection
  • # Describe the connection to obtain the Google-managed service account identity
  • bq show --location=us-central1 --connection enterprise-analytics-prod.us-central1.biglake-gcs-connection

Step 2: Grant Storage Permissions to the Connection Service Account # Assign Storage Object Viewer to the connection's dedicated service account

  • gcloud storage buckets add-iam-policy-binding gs://analytics-curated-lakehouse \
  • -member="serviceAccount:service-9876543210@gcp-sa-bigquery-connection.iam.gserviceaccount.com" \
  • -role="roles/storage.objectViewer"

Step 3: Create the BigLake Table via DDL -- Create a partitioned BigLake external table with metadata caching enabled

  • CREATE EXTERNAL TABLE `enterprise-analytics-prod.curated_customer.transactions_biglake`
  • WITH CONNECTION `enterprise-analytics-prod.us-central1.biglake-gcs-connection`
  • OPTIONS (
  • format = 'PARQUET',
  • uris = ['gs://analytics-curated-lakehouse/transactions/*.parquet'],
  • metadata_cache_mode = 'AUTOMATIC',
  • max_staleness = INTERVAL '30' MINUTE
  • );
  • Metadata Caching and Performance Optimization

External object stores introduce latency because BigQuery must list files in the bucket or read manifest files before constructing the query plan. For datasets containing hundreds of thousands of files, file listing alone can add 15 to 60 seconds of query planning overhead.

BigLake solves this with Metadata Caching :

metadata_cache_mode = 'AUTOMATIC' : BigQuery periodically refreshes cached file metadata (file lists, schema, and column statistics) automatically within the specified max_staleness interval. If queries run frequently, the system ensures the cache remains primed.

metadata_cache_mode = 'MANUAL' : The data engineering team explicitly triggers a cache refresh using the BQ.REFRESH_EXTERNAL_METADATA_CACHE system procedure following batch ETL ingestion jobs: CALL BQ.REFRESH_EXTERNAL_METADATA_CACHE('enterprise-analytics-prod.curated_customer.transactions_biglake');

Architecture and Trade-Off Comparison

Understanding when to select BigLake versus native BigQuery managed tables or legacy external tables is a core competency tested on the Professional Data Engineer exam.

Architecture Attribute BigQuery Native Managed Tables BigQuery External Tables (Legacy) BigLake External Tables BigLake Apache Iceberg Tables

Storage Location BigQuery internal Capacitor format GCS, AWS S3, Azure ADLS GCS, AWS S3, Azure ADLS GCS (Open Table Format)

Storage Pricing BigQuery active/long-term storage Cloud Storage / Object storage rates Cloud Storage / Object storage rates Cloud Storage rates

End-User Storage IAM Not required (table IAM only) Mandatory ( storage.objectViewer ) Not required (Connection SA only) Not required (Connection SA only)

  • Row-Level Security Full native support Unsupported Supported Supported
  • Column Policy Tags Full native support Unsupported Supported Supported
  • Dynamic Data Masking Full native support Unsupported Supported Supported

Multi-Engine Access Via BigQuery Storage Read API Direct file access (no governance) Supported via BigLake API / Connectors Open access via Iceberg + BigLake Metastore

ACID & Time-Travel Native (7-day fail-safe + time-travel) None Dependent on underlying format Full Iceberg ACID snapshot isolation

Query Latency Lowest (sub-second to seconds) High (file listing overhead) Low (with metadata caching) Very Low (file pruning via manifests)

  • Concrete Exam Scenario
  • Scenario: Regulated Multi-Tenant FinTech Data Lakehouse

A multinational fintech firm stores over 800 TB of historical loan disbursement and repayment data in Google Cloud Storage as Parquet files. The data engineering team must support three distinct analytical consumer groups:

Underwriting Analysts: Require access to applicant demographic data, loan amounts, and credit scores, but must never see unmasked Social Security Numbers (SSN) or bank account routing numbers.

Regional Compliance Officers: Must only see loan records originating within their assigned jurisdiction (e.g., California vs. New York).

Data Science Team (Dataproc Spark): Trains predictive default models using Apache Spark clusters and requires direct columnar access without intermediate SQL extraction bottlenecks.

The Anti-Pattern

The engineering team initially creates standard BigQuery external tables pointing to gs://fintech-loan-data/disbursements/*.parquet and grants all analysts roles/storage.objectViewer on the bucket so BigQuery can read the data. They attempt to configure authorized views to restrict rows and columns.

Why this fails: Because the analysts have storage.objectViewer on the GCS bucket, an analyst can simply download the Parquet files using gcloud storage cp or an unmanaged Python notebook, completely bypassing the authorized views and viewing unmasked SSNs and out-of-region loans in plain text.

The Recommended BigLake Architecture

Establish a Cloud Resource Connection: Deploy a BigQuery connection in the dataset region and capture the generated service account.

Enforce Least Privilege on Storage: Revoke all direct user access ( roles/storage.objectViewer ) from gs://fintech-loan-data . Grant roles/storage.objectViewer exclusively to the BigQuery Connection's service account.

Deploy BigLake Table with Schema Definitions: Define a BigLake table via CREATE EXTERNAL TABLE ... WITH CONNECTION . In the schema definition, tag the ssn and account_number columns with Dataplex Policy Tags configured for SHA-256 data masking.

Apply Row-Level Security: Define a CREATE ROW ACCESS POLICY filtering records by jurisdiction = SESSION_USER() or authorized group membership.

Configure Spark Access via BigLake Connector: Configure the Data Science team's Cloud Dataproc clusters with the BigLake Spark Connector . When Spark queries the table, BigLake enforces the same column masking and row-level policies directly on Spark RDDs/DataFrames without exposing the raw underlying bucket.

Common Exam Pitfalls and Gotchas

Pitfall 1: Granting Direct Bucket Access to End Users: If end users retain storage.objects.get permissions on the Cloud Storage bucket, the entire security perimeter of BigLake is compromised. Users can directly download the files and inspect masked columns or restricted rows. On the exam, the correct architecture always revokes user bucket permissions and relies solely on the connection service account.

Pitfall 2: Confusing Standard External Tables with BigLake Tables: Standard external tables omit the WITH CONNECTION clause. If a question asks why row-level security or policy tags cannot be attached to an external table, the root cause is that the table was created as a standard external table rather than a BigLake table.

Pitfall 3: Stale Metadata in Batch Ingestion Pipelines: When new Parquet files are appended to a GCS path behind a BigLake table with metadata_cache_mode = 'AUTOMATIC' and max_staleness = INTERVAL '4' HOUR , queries run immediately after ingestion will not see the new records until the cache expires or CALL BQ.REFRESH_EXTERNAL_METADATA_CACHE is explicitly invoked.

Pitfall 4: Cross-Region Connection Mismatches: A BigLake connection created in us-central1 cannot access a storage bucket located in europe-west3 or create a table in a dataset located in us-east4 . The BigQuery dataset, the Cloud Resource Connection, and the target Cloud Storage bucket must be co-located in matching regions or multi-regions.

Loading diagram... BigLake Delegated Security and Query Execution Engine