Study guide
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect)
Google Cloud Certified Professional Data Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Security, IAM, and Data Protection
1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
2 Chapter 2: Reliability and Compliance Architecture
2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack
3 Chapter 3: Cross-Cloud and Federated Governance
3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance
4 Chapter 4: Data Ingestion and Pipeline Planning
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion
5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub
5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow
6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows
7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics
7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management
8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion
8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage
9 Chapter 9: Storage Selection and Cloud Storage Architecture
9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools
10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL
10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery
11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore
11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections
12 Chapter 12: Enterprise Data Warehousing with BigQuery
12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions
13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery
13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling
14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML
14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles
15 Chapter 15: Operations, Orchestration, and Data Governance
15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect)
Key Takeaways Explain This Topic AI
Shared VPC cleanly separates enterprise network administration (centralized in a Host Project) from data processing workloads (isolated in Service Projects), allowing Dataflow workers and Dataproc clusters to consume vetted subnets without granting data engineers host network administrative rights.
Private Google Access (PGA) allows compute resources with only internal RFC 1918 private IP addresses to reach Google APIs (Cloud Storage, BigQuery, Pub/Sub) via internal routing to 'private.googleapis.com' (199.36.153.4/30) or 'restricted.googleapis.com' (199.36.153.8/30).
Private Service Connect (PSC) provides private, one-way IP endpoint connectivity to Google APIs, third-party SaaS platforms, and internal producer services using local consumer IP forwarding rules, eliminating VPC peering routing limits and overlapping RFC 1918 CIDR constraints.
Cloud Interconnect delivers enterprise hybrid data ingest via Dedicated (10/100 Gbps circuits) or Partner Interconnect with 99.99% multi-city SLA topologies, whereas Cloud HA VPN provides IPsec tunnels over public transit capped at 3 Gbps per tunnel.
Dataflow worker clusters deployed with '--no-use-public-ips' require either Private Google Access to communicate with GCP storage backends or Cloud NAT for egress to external third-party endpoints, alongside internal firewall rules opening TCP ports 12345-12346 for worker-to-worker shuffle communication.
Last updated: September 2026
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect)
Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to design secure, high-throughput network topologies for data ingestion. You must master the architectural differences between Shared VPC and VPC Peering , configure Private Google Access (PGA) and DNS steering for VPC Service Controls , implement Private Service Connect (PSC) to overcome RFC 1918 overlapping subnets, evaluate hybrid ingestion bandwidth across Dedicated/Partner Cloud Interconnect and Cloud HA VPN , and properly size worker egress and firewall rules for distributed compute engines like Cloud Dataflow and Cloud Dataproc .
Data ingestion is the critical circulatory system of any modern analytics platform. Before a single transformation runs in Apache Beam or a single query executes in BigQuery, data must traverse network perimeters separating operational databases, SaaS applications, multi-cloud object stores, and edge IoT devices from Google Cloud storage repositories. A poorly designed network topology introduces latency bottlenecks, escalates cross-region egress expenses, creates compliance vulnerabilities, and causes silent pipeline failures. Designing robust data ingestion pipelines demands rigorous planning across data sources, destination sinks, and underlying Google Cloud networking primitives.
1. Heterogeneous Data Sources and Analytical Sinks
Enterprise data architectures must ingest data from diverse origins, each imposing distinct protocol, security, and latency requirements: +-----------------------------------------------------------------------------------------+
- | INGESTION SOURCE TAXONOMY |
- +-----------------------------+-----------------------------+-----------------------------+
- | On-Premises & Colocation | SaaS & Webhooks | Multi-Cloud & Edge IoT |
- | - Oracle, SQL Server, DB2 | - Salesforce, Workday, SAP | - AWS S3, Azure ADLS Gen2 |
- | - Mainframe flat files | - REST APIs, Webhooks | - Telemetry, MQTT sensors |
- | - Latency-sensitive OLTP | - Rate-limited, pagination | - Terabyte-scale egress |
- +-----------------------------+-----------------------------+-----------------------------+
- │
- ▼
- +-----------------------------------------------------------------------------------------+
- | ENTERPRISE NETWORK TRANSIT FABRIC |
- | Shared VPC │ Private Google Access │ Private Service Connect │ Cloud Interconnect |
- +-----------------------------------------------------------------------------------------+
- │
- ▼
- +-----------------------------------------------------------------------------------------+
- | TARGET ANALYTICAL SINKS |
- | Cloud Storage (Raw Lake) │ BigQuery (Warehouse) │ Cloud Bigtable (Time-Series) |
- | Cloud Spanner (Global OLTP)│ Memorystore (Cache) │ Vertex AI Feature Store (ML) |
- +-----------------------------------------------------------------------------------------+
- Ingestion Source Categories
On-Premises Relational Databases & Mainframes: Enterprise transaction backbones (Oracle Exadata, Microsoft SQL Server, IBM DB2) running in corporate datacenters. Challenges include strict firewall perimeters, non-routable private networks, change log retention constraints, and operational read lock contention.
SaaS Platforms and Cloud Business Applications: Modern enterprise applications (Salesforce, ServiceNow, Marketo, Workday, Stripe) emitting data via REST APIs, GraphQL endpoints, and real-time event webhooks. Ingestion requires secure webhook ingress endpoints, authentication token refresh lifecycles, and resilient rate-limit backoff logic.
Multi-Cloud Repositories: High-volume object stores in competing clouds (Amazon S3, Azure Data Lake Storage Gen2) hosting curated data assets. Ingestion patterns utilize Google Cloud Storage Transfer Service, Workload Identity Federation, and dedicated cross-cloud interconnects to bypass public transit.
Edge Telemetry, Connected Vehicles, and IoT: Fleets of distributed physical hardware (industrial SCADA sensors, connected vehicles, smart meters) generating millions of lightweight events per second over MQTT, HTTP/2, and WebSockets funneling into Google Cloud Pub/Sub.
Analytical Sinks
Cloud Storage (Raw & Curated Lakehouse): The cost-effective landing zone for immutable raw files (Avro, Parquet, JSON, CSV). Supports multi-tiered lifecycle management (Standard -> Nearline -> Coldline -> Archive).
Google BigQuery: Serverless, highly scalable enterprise data warehouse. Ingestion targets include native managed tables (via batch load jobs or Storage Write API) and external BigLake tables.
Cloud Bigtable: Ultra-low-latency NoSQL wide-column database engineered for massive ingestion throughput (>100,000 QPS) and sub-10ms point reads. Ideal for real-time fraud scoring, time-series telemetry, and IoT stream ingestion.
Cloud Spanner: Horizontally scalable, globally distributed relational database guaranteeing external consistency (ACID) via TrueTime. Target for consolidated multi-region operational ledgers.
2. Google Cloud VPC Network Topology: Shared VPC vs. VPC Peering
Google Cloud Virtual Private Cloud (VPC) networks are global software-defined networks spanning all Google Cloud regions worldwide. Subnets are regional resources carved out of RFC 1918 private address spaces ( 10.0.0.0/8 , 172.16.0.0/12 , 192.168.0.0/16 ).
Important
The Default VPC automatically provisions an auto-mode network with default subnets in every Google Cloud region, pre-populating permissive firewall rules (such as default-allow-internal and default-allow-ssh from 0.0.0.0/0 ). Production data engineering environments must strictly avoid Default VPCs. Always build Custom Mode VPCs with explicit CIDR allocations, deterministic subnet boundaries, and zero-trust firewall configurations.
When organizing enterprise data pipelines across multiple Google Cloud projects, data architects must choose between Shared VPC and VPC Network Peering : +-----------------------------------------------------------------------------------------+
- | SHARED VPC ARCHITECTURE |
- +-----------------------------------------------------------------------------------------+
- | HOST PROJECT (Managed by Central Network & Security Operations) |
- | - Central Shared VPC Network ('vpc-enterprise-prod') |
- | - Subnet 1: 'dataflow-workers-uscentral1' (10.10.10.0/24) |
- | - Subnet 2: 'dataproc-clusters-uscentral1' (10.10.20.0/24) |
- | - Cloud Interconnect Attachments, Cloud Routers, Cloud NAT Gateways |
- | - Centralized Network Security & Firewall Rules |
- +-----------------------------------------------------------------------------------------+
- │ │
- │ IAM: roles/compute.networkUser │ IAM: roles/compute.networkUser
- ▼ ▼
- +-----------------------------------+ +-----------------------------------------------+
- | SERVICE PROJECT A | | SERVICE PROJECT B |
- | ('pipeline-ingest-prod') | | ('analytics-warehouse-prod') |
- | - Cloud Dataflow Workers | | - Cloud Dataproc Spark Clusters |
- | - Cloud Composer DAGs | | - BigQuery Datasets & BigLake Tables |
- | * Workers attach vNICs directly | | * Workers attach vNICs directly |
- | to Host Project Subnet 1 | | to Host Project Subnet 2 |
- +-----------------------------------+ +-----------------------------------------------+
- Shared VPC (The Enterprise Standard)
Shared VPC enables an enterprise to connect resources from multiple projects to a common Virtual Private Cloud network, allowing them to communicate with each other securely and efficiently using internal IP addresses.
Host Project: An administratively isolated project containing one or more Shared VPC networks, subnets, route tables, Cloud NAT gateways, Interconnect attachments, and firewall rules. Managed strictly by the central network engineering team.
Service Projects: Workload projects attached to the host project by a Shared VPC Administrator. Data engineers launch compute resources (Dataflow worker pools, Dataproc clusters, Cloud Composer GKE nodes) inside service projects, but their virtual network interfaces (vNICs) are instantiated directly inside the host project's subnets.
IAM Delegation and Least Privilege: Network administrators grant the roles/compute.networkUser role to the service project's service accounts (e.g., service-[SERVICE_PROJ_NUM]@dataflow-service-producer-prod.iam.gserviceaccount.com and the custom Dataflow worker service account) on specific host subnets. Data engineers can consume network capacity without possessing permissions to modify firewall rules, altering routing tables, or peering new networks.
VPC Network Peering
VPC Network Peering connects two independent VPC networks so that resources in each network can communicate with each other via internal IP addresses without public internet transit or intermediate NAT gateways.
Decentralized and Autonomous: Each project team maintains full administrative control over their respective VPC, routing tables, and firewall configurations.
Non-Transitive Routing: VPC Peering is strictly non-transitive . If VPC A is peered with VPC B, and VPC B is peered with VPC C, VPC A cannot communicate with VPC C through VPC B. To enable communication, an explicit peering between A and C must be established.
Prohibition of Overlapping CIDRs: VPC Peering cannot be established if any subnet CIDR ranges overlap between the two VPC networks.
- Architecture Attribute Shared VPC VPC Network Peering
- Administrative Model Centralized (Host Project governed by NetOps) Decentralized (Each project owns its VPC)
- Routing Model Single unified routing table per VPC Separate routing tables; routes exchanged via peering
- Transitivity Fully transitive across all subnets in the VPC Strictly Non-Transitive (A <-> B <-> C != A <-> C)
- CIDR Overlap Impossible (Single address space allocated) Hard failure if subnets overlap
Cross-Organization Restricted to same Cloud Identity Organization Supported across distinct Cloud Identity Orgs
Data Platform Fit Default recommendation for internal data pipelines Inter-organizational and third-party SaaS integration
3. Private Google Access (PGA) and DNS Steering
In hardened enterprise environments, compute instances operating within data pipelines (Dataflow workers, Dataproc nodes, Compute Engine ingestion proxies) are explicitly configured without external public IP addresses ( --no-use-public-ips ). This eliminates public internet attack vectors. However, distributed pipelines must constantly communicate with Google APIs—reading data from Cloud Storage ( storage.googleapis.com ), loading tables into BigQuery ( bigquery.googleapis.com ), and pulling streaming messages from Pub/Sub ( pubsub.googleapis.com ).
How Private Google Access Works
Private Google Access (PGA) enables virtual machine instances and pipeline workers that possess only internal RFC 1918 private IP addresses to communicate privately with Google APIs and services.
Subnet Configuration: PGA is toggled at the individual subnet level: gcloud compute networks subnets update dataflow-worker-subnet \
- -region=us-central1 \
- -enable-private-ip-google-access
Internal Routing: When a worker VM initiates an HTTPS request to storage.googleapis.com , the packet is routed through the default internet gateway route ( 0.0.0.0/0 ). However, Google's software-defined network (Andromeda) intercepts packets destined for Google public VIPs and keeps the traffic entirely within Google's private network fabric, never routing packets across the public internet.
Private Google Access Domains: Default vs. Private vs. Restricted
Depending on compliance and security requirements, organizations route Google API traffic through different Virtual IP (VIP) ranges:
- API Endpoint VIP Range Supported Services VPC Service Controls Support
- Default ( *.googleapis.com ) Public Google Anycast IPs All Google Cloud APIs No (Exfiltration risk present)
- private.googleapis.com 199.36.153.4/30 Most Google Cloud APIs Partial (Does not block non-supported APIs)
restricted.googleapis.com 199.36.153.8/30 Only VPC-SC supported services Mandatory for VPC Service Controls +-----------------------------------------------------------------------------------------+
- | RESTRICTED GOOGLE APIS DNS STEERING |
- +-----------------------------------------------------------------------------------------+
- | Dataflow Worker VM (Private IP: 10.10.10.15, No Public IP) |
- | │ |
- | │ 1. DNS Query: Resolve 'bigquery.googleapis.com' |
- | ▼ |
- | Cloud DNS Private Response Policy / Managed Zone ('googleapis.com') |
- | │ |
- | │ 2. CNAME: *.googleapis.com -> restricted.googleapis.com |
- | │ 3. A Records: restricted.googleapis.com -> 199.36.153.8, .9, .10, .11 |
- | ▼ |
- | Resolves to 199.36.153.8 |
- | │ |
- | │ 4. HTTPS API Request routed to 199.36.153.8 |
- | ▼ |
- | Custom Route: 199.36.153.8/30 -> Next Hop: Default Internet Gateway |
- | │ |
- | ▼ |
- | Google Andromeda Fabric (VPC Service Controls Perimeter Verified) |
- | └── Successfully accesses BigQuery Dataset inside Enterprise Perimeter |
- +-----------------------------------------------------------------------------------------+
- Enforcing Restricted VIPs for Data Exfiltration Prevention
When enforcing VPC Service Controls (VPC-SC) perimeters around BigQuery and Cloud Storage, routing requests to default public IP addresses is blocked. To enforce the restricted VIP:
Cloud DNS Configuration: Create a private Cloud DNS zone for googleapis.com with a CNAME record mapping *.googleapis.com to restricted.googleapis.com. , and create four A records mapping restricted.googleapis.com. to 199.36.153.8 , 199.36.153.9 , 199.36.153.10 , and 199.36.153.11 .
Custom Routing: In the VPC network, configure a custom route for destination 199.36.153.8/30 with the next hop set to the default-internet-gateway .
Result: All API requests originating from private VMs are steered deterministically to the restricted VIP, ensuring strict VPC Service Controls perimeter compliance.
4. Private Service Connect (PSC)
While VPC Peering and Private Google Access address many connectivity requirements, they present notable operational limitations: VPC Peering cannot handle overlapping RFC 1918 subnets, introduces routing quota ceilings, and exposes entire network topologies. Private Service Connect (PSC) is Google Cloud's modern, software-defined private connectivity framework.
How Private Service Connect Operates
PSC allows consumers to connect privately to services in another VPC (whether operated by Google, a third-party SaaS vendor like Snowflake or MongoDB Atlas, or an internal enterprise producer) using private endpoints (forwarding rules) deployed directly within the consumer's own subnet. +-----------------------------------------------------------------------------------------+
- | PRIVATE SERVICE CONNECT (PSC) ARCHITECTURE |
- +-----------------------------------------------------------------------------------------+
- | CONSUMER VPC ('vpc-analytics-prod') | PRODUCER VPC ('vpc-saas-partner') |
- | | |
- | Subnet: 'analytics-subnet' (10.100.0.0/24) | Subnet: 'producer-subnet' (10.100.0.0/24)|
- | | [Identical RFC 1918 CIDR Overlap!] |
- | [ Dataflow Worker VM ] | |
- | │ | |
- | ▼ (Internal Request) | |
- | [ PSC Forwarding Rule / Endpoint ] | |
- | Internal IP: 10.100.0.50 | |
- | │ | |
- | └────────────── Google Transit Fabric ──────────────────┐ |
- | (Software-Defined Geneve Tunnel) ▼ |
- | | [ Service Attachment ] |
- | | NAT Subnet: 172.16.0.0/24 (SNAT) |
- | | │ |
- | | ▼ |
- | | [ Internal Load Balancer (ILB) ] |
- | | │ |
- | | ▼ |
- | | [ Producer Database / Analytics Engine ] |
- +-----------------------------------------------------------------------------------------+
- Key Architectural Superpowers of PSC for Data Pipelines
Zero CIDR Conflict (Handles Overlapping Subnets): The consumer VPC and the producer VPC can both use the identical RFC 1918 CIDR block (e.g., 10.100.0.0/24 ). On the producer side, the Service Attachment uses a dedicated NAT Subnet to perform Source Network Address Translation (SNAT), completely decoupling the consumer's IP space from the producer's IP space.
Strictly Unidirectional Access: Traffic can only flow from the consumer endpoint to the producer's published service attachment. The producer has zero visibility into, and cannot initiate connections back into, the consumer VPC.
Granular Service-Level Segmentation: Instead of peering two entire networks and managing complex firewall rules, PSC exposes only a single, specific internal load-balanced service.
PSC for Google APIs: Data engineers can deploy a PSC endpoint with a dedicated internal IP address (e.g., 10.10.10.50 ) to access all Google APIs. By mapping *.p.googleapis.com to this local IP, all pipeline API traffic travels through a local consumer endpoint without needing default internet gateway routes.
5. Hybrid and Cross-Cloud Connectivity: Cloud Interconnect vs. Cloud HA VPN
Ingesting petabytes of historical data or streaming continuous transactional deltas from on-premises corporate datacenters into Google Cloud requires reliable, high-bandwidth hybrid connectivity. The primary choices are Cloud Interconnect and Cloud HA VPN : +-----------------------------------------------------------------------------------------+
- | HYBRID CONNECTIVITY: INTERCONNECT VS. HA VPN |
- +-----------------------------------------------------------------------------------------+
- | DEDICATED INTERCONNECT (Physical Cross-Connect) |
- | On-Premises Router ──(10G / 100G Direct Fiber)──> Google Edge Colocation ──> Google VPC |
- | * Features: Dedicated physical circuits, lowest deterministic latency, 99.99% SLA |
- +-----------------------------------------------------------------------------------------+
- | PARTNER INTERCONNECT (Service Provider Transit) |
- | On-Premises Router ──(VLAN Attachment)──> Equinix / Megaport ──> Google VPC |
- | * Features: Sub-10G to 50G, for facilities without physical Google Colocation presence |
- +-----------------------------------------------------------------------------------------+
- | CLOUD HA VPN (IPsec over Public Internet) |
- | On-Premises VPN ──(IPsec Tunnels via BGP, max 3 Gbps/tunnel)──> Cloud HA VPN Gateway |
- | * Features: Fast setup, encrypted over public internet, throughput capped |
- +-----------------------------------------------------------------------------------------+
- Dedicated Interconnect
Physical Direct Link: Enterprise provisions physical fiber connections (10 Gbps or 100 Gbps circuits) directly into a Google Cloud Colocation Facility (Point of Presence / PoP).
Highest Throughput and Deterministic Latency: Bypasses the public internet entirely. Ideal for continuous replication of high-volume OLTP databases and high-frequency analytical feeds.
Availability Topologies:
99.9% Availability (Non-Critical): Requires two single connections to two distinct edge routers in a single colocation facility in a single metropolitan area.
99.99% Availability (Production Mission-Critical Standard): Requires at least four connections across two independent colocation facilities (two distinct metros) connected to two distinct Cloud Routers in paired Google Cloud regions.
Partner Interconnect
Provider-Mediated Transit: Connects on-premises datacenters to Google Cloud through a supported third-party telecommunications partner (e.g., Equinix Fabric, Megaport, AT&T NetBond).
Bandwidth Granularity: Capacities range from 50 Mbps up to 50 Gbps per VLAN attachment.
Selection Criteria: The definitive choice when the enterprise data center is not physically co-located in a Google colocation facility or when bandwidth requirements do not justify a dedicated 10 Gbps/100 Gbps physical circuit.
Cloud HA VPN
Encrypted Internet Transit: Establishes IPsec VPN tunnels between an on-premises VPN gateway and a Google Cloud HA VPN gateway over public internet infrastructure.
High Availability Topology: Cloud HA VPN provides an SLA of 99.99% availability by requiring two active interfaces (Interface 0 and Interface 1) configured with dynamic routing via Border Gateway Protocol (BGP).
Bandwidth Limits: Each HA VPN tunnel is hard-capped at 3 Gbps (approximately 250,000 packets per second). Achieving higher throughput requires deploying multiple parallel gateways and ECMP (Equal-Cost Multi-Path) routing, but aggregate throughput rarely scales efficiently beyond 10–15 Gbps.
- Evaluation Metric Dedicated Interconnect Partner Interconnect Cloud HA VPN
- Physical Medium Direct physical fiber Service provider VLAN IPsec over public internet
- Bandwidth Sizing 10 Gbps or 100 Gbps per link 50 Mbps to 50 Gbps 3 Gbps per tunnel
- Availability SLA 99.9% (2 links) / 99.99% (4 links) 99.9% / 99.99% 99.99% (dual active tunnels)
- Traffic Encryption Unencrypted by default (Add MACsec) Unencrypted by default Fully encrypted (IPsec)
- Provisioning Time Weeks to months (cross-connects) Days to weeks Minutes to hours
Data Ingestion Fit Daily migrations > 10 TB; high OLTP CDC Sustained 1 Gbps – 20 Gbps feeds Moderate batch feeds (< 3 Gbps)
Bandwidth Ingestion Sizing Equation
When planning network infrastructure for hybrid data ingestion, calculate required bandwidth based on data volume, ingestion window, and transport overhead: Required Bandwidth (Gbps) = Data Volume (Bits) × ( 1 + Overhead ) Ingestion Window (Seconds) \text{Required Bandwidth (Gbps)} = \frac{\text{Data Volume (Bits)} \times (1 + \text{Overhead})}{\text{Ingestion Window (Seconds)}} Required Bandwidth (Gbps) = Ingestion Window (Seconds) Data Volume (Bits) × ( 1 + Overhead ) Example: 50 TB daily migration in a 6-hour overnight window with 20% protocol overhead: \text{Example: } 50\text{ TB daily migration in a 6-hour overnight window with 20\% protocol overhead:} Example: 50 TB daily migration in a 6-hour overnight window with 20% protocol overhead: Volume = 50 × 10 12 × 8 bits = 400 × 10 12 bits = 400 , 000 Gb \text{Volume} = 50 \times 10^{12} \times 8\text{ bits} = 400 \times 10^{12}\text{ bits} = 400,000\text{ Gb} Volume = 50 × 1 0 12 × 8 bits = 400 × 1 0 12 bits = 400 , 000 Gb Window = 6 hours × 3 , 600 seconds = 21 , 600 seconds \text{Window} = 6\text{ hours} \times 3,600\text{ seconds} = 21,600\text{ seconds} Window = 6 hours × 3 , 600 seconds = 21 , 600 seconds Required Bandwidth = 400 , 000 21 , 600 × 1.20 ≈ 22.2 Gbps \text{Required Bandwidth} = \frac{400,000}{21,600} \times 1.20 \approx 22.2\text{ Gbps} Required Bandwidth = 21 , 600 400 , 000 × 1.20 ≈ 22.2 Gbps
Architectural Conclusion: A 3 Gbps Cloud HA VPN cannot meet this SLA. The architecture mandates either a 50 Gbps Partner Interconnect or Dedicated Interconnect.
6. Worker Egress Caps, Cloud NAT, and Firewall Rules
Distributed data processing engines (Dataflow, Dataproc) deploy fleets of virtual machine worker instances that perform heavy network I/O during pipeline execution.
Compute Engine & Dataflow Egress Bandwidth Caps
Virtual machine network throughput is bounded by the instance's vCPU allocation and networking configuration. Standard VMs receive 2 Gbps per vCPU up to a maximum of 10–16 Gbps.
For bandwidth-intensive ingestion pipelines shuffling hundreds of gigabytes across workers, selecting instances with higher vCPU counts or configuring Tier 1 High Bandwidth Networking (unlocking 50 to 100 Gbps per VM) is critical to prevent network throttling.
Cloud NAT for Outbound Internet Egress
When Dataflow workers run with --no-use-public-ips , Private Google Access allows them to reach Google APIs privately. However, if the pipeline must pull data from an external third-party SaaS REST API or an external public web service, Private Google Access cannot route that traffic.
Solution: Deploy Cloud NAT (Network Address Translation) on the Cloud Router serving the worker subnet. Cloud NAT provides managed, highly available outbound internet connectivity for instances lacking external IPs without allowing unsolicited inbound connections from the internet. # Deploy Cloud NAT for private Dataflow worker internet egress
- gcloud compute routers create nat-router \
- -network=vpc-data-prod \
- -region=us-central1
- gcloud compute routers nats create worker-nat-gateway \
- -router=nat-router \
- -region=us-central1 \
- -auto-allocate-nat-external-ips \
- -nat-all-subnet-ip-ranges
- Essential Firewall Rules for Dataflow and Dataproc Clusters
Distributed pipelines communicate internally between worker nodes during shuffle and coordination phases. Misconfigured firewall rules are the leading cause of pipeline execution timeouts:
Dataflow Internal Shuffle Communication: When Dataflow executes without Streaming Engine (or during heavy batch operations), workers communicate directly across TCP ports 12345 and 12346 . The VPC must have an ingress firewall rule allowing internal TCP traffic on ports 12345-12346 between worker instances tagged with dataflow .
Dataproc Internal Communication: Apache Spark and Hadoop require open ingress across all internal cluster nodes for HDFS NameNode/DataNode communication (ports 8020, 9870, 9866) and YARN resource management (ports 8088, 8030-8032). # Firewall rule allowing internal shuffle communication for Dataflow workers
- gcloud compute firewall-rules create allow-dataflow-internal-shuffle \
- -network=vpc-data-prod \
- -action=ALLOW \
- -direction=INGRESS \
- -target-tags=dataflow \
- -source-tags=dataflow \
- -rules=tcp:12345-12346
- 7. Concrete Exam Scenarios & Architecture Pitfalls
Scenario / Architecture Challenge Common Anti-Pattern Correct Google Cloud Architecture
Secure Multi-Project Ingestion ; An enterprise data engineering team needs to launch Dataflow workers that consume vetted corporate subnets while preventing data engineers from creating or modifying network firewall rules. Granting data engineers roles/compute.networkAdmin in a single monolithic project. Implement Shared VPC . Host project holds subnets and firewall rules governed by NetOps. Data engineers launch workers in a Service Project , with the Dataflow service account granted roles/compute.networkUser strictly on the worker subnet.
Overlapping Subnets with External Partner ; An ingestion pipeline needs to pull streaming logs from a partner's MongoDB database. Both the partner VPC and enterprise VPC use CIDR 10.50.0.0/16 . Attempting to deploy VPC Network Peering and requesting the partner re-address their entire production network. Deploy Private Service Connect (PSC) . The partner publishes an internal load balancer via a PSC Service Attachment with a dedicated NAT subnet. The enterprise consumes it via an internal PSC Endpoint IP, completely bypassing CIDR overlap.
Dataflow Workers Failing in Private VPC ; Dataflow pipeline launched with --no-use-public-ips hangs indefinitely during startup and reports timeout communicating with Cloud Storage. Assigning public IP addresses to all Dataflow worker instances to resolve connectivity. Enable Private Google Access on the worker subnet. Configure Cloud DNS to resolve *.googleapis.com to restricted.googleapis.com (199.36.153.8/30) with custom route to default internet gateway.
Petabyte Database Ingest Bandwidth Sizing ; An enterprise needs to ingest 40 TB of database change logs every 8 hours from an on-premises datacenter with guaranteed 99.99% availability. Deploying two Cloud HA VPN tunnels over the public internet. Deploy Dedicated Interconnect with at least four 10 Gbps links across two distinct colocation facilities (dual-metro) to satisfy the 14+ Gbps sustained bandwidth requirement and achieve the 99.99% SLA.