Study guide
4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection
Google Cloud Certified Professional Data Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Security, IAM, and Data Protection
1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
2 Chapter 2: Reliability and Compliance Architecture
2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack
3 Chapter 3: Cross-Cloud and Federated Governance
3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance
4 Chapter 4: Data Ingestion and Pipeline Planning
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion
5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub
5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow
6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows
7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics
7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management
8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion
8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage
9 Chapter 9: Storage Selection and Cloud Storage Architecture
9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools
10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL
10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery
11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore
11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections
12 Chapter 12: Enterprise Data Warehousing with BigQuery
12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions
13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery
13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling
14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML
14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles
15 Chapter 15: Operations, Orchestration, and Data Governance
15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services
4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection
Key Takeaways Explain This Topic AI
Batch ingestion maximizes resource efficiency by utilizing ephemeral compute with 100% duty cycles and spot/preemptible instances, whereas streaming pipelines require persistent provisioned baseline capacity and stateful window buffering to maintain sub-second latency SLAs.
BigQuery batch load jobs are completely free of compute charges when loading from Cloud Storage using Google's shared multi-tenant slot pool, subject to a quota of 1,500 load jobs per table per day.
The BigQuery Storage Write API provides high-throughput, lower-cost ($0.025/GB after 2 TB/month free) ingestion supporting committed streams (instant, exactly-once), default streams (low-latency at-least-once), and pending streams (atomic multi-stream two-phase commits).
Cloud Pub/Sub serves as an enterprise shock absorber for bursty and spiky data streams, retaining unbounded message streams up to 7 days, and offers direct Pub/Sub BigQuery Subscriptions to ingest JSON or Avro payloads without executing custom Dataflow pipelines.
Architecture selection balances operational complexity and cost: Cloud Data Fusion provides visual ETL at high baseline cluster cost, Dataflow Batch with Flexible Resource Scheduling (FlexRS) slashes non-urgent batch processing costs up to 40%, and Dataflow Streaming with Streaming Engine minimizes worker memory bottlenecks by externalizing pipeline state.
Last updated: September 2026
4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection
Exam Focus: The Professional Data Engineer exam rigorously tests your ability to evaluate the latency vs. cost curve , select between free BigQuery batch load jobs and the BigQuery Storage Write API , architect resilient ingestion buffers using Cloud Pub/Sub , and select the optimal execution framework among Dataflow Batch (with FlexRS) , Dataflow Streaming (with Streaming Engine) , and Cloud Data Fusion .
A foundational responsibility of the data engineer is matching analytical processing paradigms to business requirements. Choosing between batch, micro-batch, and continuous streaming is never merely a technical preference; it is an economic and operational commitment. Processing data in real time introduces non-linear cost escalations, complex state management, and stringent failure recovery requirements. A data architect must systematically justify why a workload requires continuous streaming rather than cost-effective scheduled batch execution.
1. Batch vs. Micro-Batch vs. Continuous Streaming: Evaluation Framework
Data pipelines process information across a spectrum of latency SLAs, processing semantics, and resource consumption profiles: +-----------------------------------------------------------------------------------------+
- | THE LATENCY VS. COST SPECTRUM |
- +-----------------------------------------------------------------------------------------+
- | BATCH PROCESSING MICRO-BATCH PROCESSING CONTINUOUS STREAMING |
- | (Daily / Hourly Cadence) (Seconds to Minutes Cadence) (Sub-Second / Real-Time) |
- | - Latency: Hours to Days - Latency: 10s to 5 mins - Latency: Milliseconds |
- | - Ephemeral Compute Clusters - Short-lived Tasks/Cloud Run - Always-on Compute Engine |
- | - 100% Resource Duty Cycle - Periodic Buffer Ingestion - Stateful Windows & Watermarks
- | - Free BigQuery Load Jobs - Storage Write API (Buffered) - Storage Write API (Committed)
- | - Lowest Cost Profile ($) - Moderate Cost Profile ($$) - Highest Cost Profile ($$$)|
- +-----------------------------------------------------------------------------------------+
- The Three Processing Paradigms
Batch Processing (Bounded Datasets):
Data Semantics: Operates on complete, bounded datasets that are finalized and static at the time of execution (e.g., historical sales logs for the previous business day).
Execution Model: High-throughput, distributed table scans and shuffles. Compute resources are provisioned on demand, run to completion at maximum CPU utilization, and terminate immediately.
Failure Model: Deterministic replay. If a worker fails, the batch framework simply re-executes the failed task partition.
Micro-Batch Processing (Time-Sliced Windows):
Data Semantics: Accumulates unbounded continuous data into discrete time-bounded chunks (e.g., buffering incoming events for 60 seconds or accumulating 50 MB files before processing).
Execution Model: Short, recurring jobs triggered by Cloud Scheduler, Cloud Functions, or micro-batch engines (like Spark Streaming).
Trade-Off: Provides near-real-time freshness without the operational complexity and cost of maintaining continuous, always-on streaming state.
Continuous Streaming Processing (Unbounded Event Streams):
Data Semantics: Treats data as an infinite, continuous stream of events arriving at arbitrary times. Events may arrive out of order, delayed by network partitions, or duplicated.
Execution Model: Long-running, persistent compute workers (e.g., Apache Beam pipelines on Cloud Dataflow). Requires managing complex streaming constructs: event time vs. processing time, tumbling/sliding/session windows, triggers, and watermarks.
Failure Model: Stateful checkpointing. Engines continuously commit state snapshots to persistent storage to guarantee exactly-once processing upon worker failure.
The Non-Linear Latency vs. Cost Escalation Curve
As business requirements push latency targets down from hours to sub-second real time, infrastructure costs escalate exponentially:
Compute Efficiency: Batch compute achieves near 100% duty cycle efficiency during execution. Streaming compute must maintain idle headroom capacity (typically running at 30–50% average CPU utilization) to absorb sudden event spikes without lagging.
Pricing Structures: Batch loading into BigQuery from Cloud Storage is completely free of compute charges. In contrast, streaming data into BigQuery incurs dedicated ingestion fees per gigabyte processed.
State Storage Overhead: Stateful windowing (such as computing a 30-day running average per customer in real time) requires storing gigabytes or terabytes of volatile pipeline state in memory or external persistent caches.
- Evaluation Dimension Batch Processing Micro-Batch Processing Continuous Streaming
- Data Bound Bounded (Fixed size) Semi-bounded (Time chunks) Unbounded (Infinite stream)
- Latency SLA 1 hour – 24+ hours 30 seconds – 15 minutes Sub-second – 5 seconds
- State Management Stateless between runs Minimal checkpointing Complex (Watermarks, windows)
- Compute Sizing Ephemeral (Scale to zero) Serverless / Scheduled bursts Persistent (Always-on baseline)
- Cost Optimization Spot/Preemptible, FlexRS Cloud Run, scheduled Composer Streaming Engine, right-sizing
Typical Services BigQuery Loads, Dataflow Batch Cloud Functions, Dataproc Serverless Dataflow Streaming, Pub/Sub
2. Compute Resource Sizing and Cost Optimization Models
Optimizing cloud data engineering spend requires matching compute topologies to the elasticity profile of the workload.
Batch Compute Optimization: Ephemeral Clusters, Spot VMs, and FlexRS
Ephemeral Dataproc Clusters: Never maintain a permanent, running Apache Spark or Hadoop cluster for batch processing. Deploy ephemeral Dataproc clusters via Cloud Composer or workflow templates that spin up, execute a specific Spark job, and immediately delete the cluster upon completion. Cluster creation overhead is typically under 90 seconds.
Spot VMs / Preemptible VMs: Ephemeral batch nodes can leverage Google Cloud Spot VMs, which offer a 60% to 91% discount compared to standard compute pricing. In Cloud Dataproc, secondary worker nodes can be configured as 100% Spot instances. If Google reclaims a secondary worker VM, the Spark master redistributes the lost task partitions to remaining nodes without job failure.
Dataflow Flexible Resource Scheduling (FlexRS): For non-time-sensitive batch jobs that can complete within a flexible 6-hour execution window (such as overnight billing calculations), Dataflow FlexRS reduces compute costs by up to 40% . FlexRS orchestrates a cost-optimized combination of preemptible and standard VMs, delaying job startup until excess Google compute capacity becomes available: # Launch a Dataflow batch pipeline using FlexRS for cost reduction
- gcloud dataflow flex-template run overnight-transaction-aggregator \
- -template-file-gcs-location=gs://templates/aggregator.json \
- -region=us-central1 \
- -parameters flexrsGoal=COST_OPTIMIZED,workerMachineType=n2-standard-4
- Streaming Compute Optimization: Autoscaling Headroom and Streaming Engine
Baseline Provisioning vs. Autoscaling Lag: In streaming pipelines, CPU-based autoscaling cannot react instantaneously to sudden 10x traffic surges; spinning up new worker VMs takes 2 to 4 minutes, during which backlogs accumulate and watermarks stall. Architects must configure baseline worker allocations ( --numWorkers ) sized to comfortably handle ordinary peak hours, allowing autoscaling ( --maxNumWorkers ) to handle unexpected surges.
Dataflow Streaming Engine: In legacy Dataflow worker architectures, pipeline execution state, window aggregation buffers, and shuffle operations were executed entirely on local persistent disks attached to the worker VMs. This introduced severe CPU and disk I/O bottlenecks. Streaming Engine decouples state storage and shuffle from worker VMs, offloading state execution to a dedicated, highly optimized Google Cloud backend service. This drastically reduces worker VM vCPU and memory sizing requirements, leading to smoother autoscaling and lower overall pipeline costs.
3. BigQuery Ingestion Architecture: Free Batch Loads vs. Storage Write API
BigQuery decouples compute (query slots) from storage (Capacitor columnar storage). Understanding how data enters BigQuery is one of the most frequently tested topics on the exam. +-----------------------------------------------------------------------------------------+
- | BIGQUERY INGESTION ARCHITECTURES |
- +-----------------------------------------------------------------------------------------+
- | PATH A: FREE BATCH LOAD JOBS (From Cloud Storage) |
- | Raw Files (Avro/Parquet) ──> Cloud Storage ──(bq load / Load Job)──> BigQuery Storage |
- | * Compute Cost: 100% FREE (Uses Google Shared Slot Pool) |
- | * Quota: 1,500 loads/table/day; Latency: Minutes |
- +-----------------------------------------------------------------------------------------+
- | PATH B: STORAGE WRITE API (Unified High-Performance Streaming) |
- | Streaming Producers ──(gRPC Streams via Protobuf)──> BigQuery Capacitor Storage |
- | * Compute Cost: $0.025 per GB (2 TB Free/month) |
- | * Features: Exactly-once, Committed/Pending/Default Streams, Sub-second latency |
- +-----------------------------------------------------------------------------------------+
- | PATH C: LEGACY STREAMING API (tabledata.insertAll - DEPRECATED) |
- | Legacy Client ──(HTTP JSON Post)──> Streaming Buffer ──> Capacitor Storage |
- | * Compute Cost: $0.05 per GB (Double the cost of Storage Write API!) |
- | * Limitations: At-least-once, best-effort deduplication, no multi-stream transactions |
- +-----------------------------------------------------------------------------------------+
- BigQuery Free Batch Load Jobs
Shared Multi-Tenant Compute: Loading data into BigQuery tables from Cloud Storage (or local files) using load jobs ( bq load CLI, BigQuery Load Job API, or Cloud Console) is completely free of BigQuery compute charges . Google absorbs the compute cost using its shared multi-tenant slot pool.
File Format Performance:
Avro (Optimal): Apache Avro is the fastest and most efficient format for loading data into BigQuery. Because Avro is a row-oriented binary format with an embedded JSON schema, BigQuery reads blocks in parallel across hundreds of workers simultaneously without schema inference overhead.
Parquet / ORC (Good): Columnar formats load efficiently but require parsing column footers across files, making them slightly slower to load than Avro (though optimal for querying directly in GCS via BigLake).
CSV / JSON (Slowest): Plaintext formats require expensive character parsing, delimiter escaping, and type coercion. In uncompressed CSV/JSON, BigQuery can parallelize reads; however, gzipped CSV/JSON cannot be parallelized (a single thread must read the entire compressed file sequentially), resulting in severely degraded load performance.
Quotas and Constraints: Standard tables allow up to 1,500 load jobs per table per day (averaging approximately one load per minute) and 100,000 load jobs per project per day. Attempting to execute batch loads every 5 seconds will rapidly exhaust table quotas.
Legacy Streaming API ( tabledata.insertAll )
Cost: Costs $0.010 per 200 MB ($0.05 per gigabyte).
Limitations: Uses HTTP POST requests containing JSON strings. Guarantees only at-least-once delivery. Deduplication relies on the client passing an insertId , which BigQuery tracks in memory on a best-effort basis for only 1 minute. Architectural Guidance: Deprecated for new designs. Migrate all workloads to the Storage Write API.
Modern Standard: BigQuery Storage Write API
The Storage Write API is a unified, high-performance streaming ingestion API built on gRPC and Protocol Buffers (Protobuf) that streams data directly into BigQuery storage nodes.
Cost: $0.025 per gigabyte (50% cheaper than legacy streaming), with the first 2 TB per month free .
Architectural Stream Modes:
Committed Stream: Provides exactly-once processing semantics. Records written to the stream are committed immediately and become instantly queryable. Ideal for mission-critical transactional event feeds where duplicate records cannot be tolerated.
Default Stream: Provides at-least-once delivery with the lowest latency and highest throughput. Does not support manual commit operations; acts as a direct replacement for legacy streaming inserts at half the price.
Pending Stream: Implements two-phase commit (2PC) semantics for batch and micro-batch pipelines. Workers write millions of records into multiple parallel pending streams in an uncommitted state. Once all workers finish, a single CommitWriteStream API call atomically commits all streams simultaneously. If any worker fails, the uncommitted streams are discarded, preventing partial data writes.
Buffered Stream: Flushes records in micro-batches; provides commit offsets to guarantee exactly-once processing across pipeline restarts.
- Feature BigQuery Batch Load Jobs Legacy Streaming API Storage Write API (Committed/Pending)
- Pricing Free (Google shared slots) $0.050 / GB ($0.01 / 200 MB) $0.025 / GB (First 2 TB/mo free)
- Latency to Query Minutes (Batch schedule) Sub-second Sub-second (Committed / Default)
- Delivery Semantics Exactly-once (Atomic job) At-least-once (Best-effort dedup) Strictly Exactly-Once
- Transactions Table-level atomic commit None (Row-level appends) Atomic Multi-Stream Commits
- Quotas 1,500 loads/table/day High throughput limits 100,000+ writes/sec scalable
- Data Format Avro, Parquet, CSV, JSON HTTP REST (JSON strings) Binary Protocol Buffers (gRPC)
- 4. Ingestion Shock Absorbers: Absorbing Spiky and Bursty Workloads
Directly exposing ingestion compute workers or database endpoints to incoming client traffic during flash sales, breaking news events, or IoT telemetry bursts is a common architectural anti-pattern that leads to database connection exhaustion and dropped records.
Cloud Pub/Sub as an Elastic Ingestion Buffer
Google Cloud Pub/Sub serves as a fully managed, globally distributed asynchronous messaging backbone that decouples message producers from downstream processing pipelines:
Infinite Horizontal Elasticity: Automatically scales to absorb hundreds of thousands of events per second without pre-provisioning or partition rebalancing.
Durable Message Storage: Retains unacknowledged messages for up to 7 days by default (configurable up to 31 days). If a downstream Dataflow pipeline crashes or BigQuery encounters quota throttling, messages safely queue in Pub/Sub with zero data loss.
Seek and Replayability: If a production pipeline deployment introduces a transformation bug that writes corrupted records to storage, engineers can roll back the pipeline code and use Pub/Sub's Seek feature to rewind the subscription's acknowledgement state to a specific timestamp, re-ingesting the historical message stream cleanly.
Serverless Pub/Sub to BigQuery Direct Subscriptions
Historically, landing streaming events from Pub/Sub into BigQuery required deploying and maintaining an Apache Beam streaming pipeline on Cloud Dataflow simply to read Pub/Sub messages and call BigQuery APIs. This introduced substantial VM compute costs, worker management overhead, and monitoring complexity.
Pub/Sub BigQuery Subscriptions: Google Cloud now allows Pub/Sub topics to stream messages directly into BigQuery tables without deploying Dataflow or writing custom consumer code.
Under the Hood: Google Cloud manages serverless workers that read Pub/Sub messages and write directly to BigQuery using the Storage Write API .
Dead-Letter Topics: If a message payload fails schema validation or contains malformed JSON, it is automatically routed to a designated dead-letter topic for debugging, preventing pipeline stalls.
When to Use: Use Pub/Sub BigQuery Subscriptions whenever streaming events require direct landing into BigQuery without complex in-flight enrichment, windowed aggregations, or cross-stream joins.
5. Technology Selection Matrix: Dataflow vs. Cloud Data Fusion vs. Dataproc
Choosing the appropriate compute engine for data ingestion and transformation requires balancing developer skillsets, pipeline complexity, and operational total cost of ownership (TCO): +-----------------------------------------------------------------------------------------+
- | PIPELINE ENGINE SELECTION LOGIC |
- +-----------------------------------------------------------------------------------------+
- | |
- | Do you have existing Apache Spark / Hadoop / Flink codebases? |
- | ├── YES ──> CLOUD DATAPROC (Ephemeral Clusters / Dataproc Serverless) |
- | │ |
- | └── NO ──> Does your team require a Visual Drag-and-Drop GUI (No-Code/Low-Code)? |
- | ├── YES ──> CLOUD DATA FUSION (CDAP-based visual ETL; enterprise analysts) |
- | │ |
- | └── NO ──> Do you require unified Batch/Streaming with sub-second latency, |
- | complex windowing, or auto-scaling Apache Beam pipelines? |
- | ├── YES ──> CLOUD DATAFLOW (Dataflow Batch with FlexRS or |
- | │ Dataflow Streaming with Streaming Engine) |
- | │ |
- | └── NO ──> Direct SQL transforms on raw data? |
- | └──> BIGQUERY ELT (dbt, Scheduled Queries, Procedures)
- +-----------------------------------------------------------------------------------------+
- 1. Cloud Dataflow
Underlying Engine: Fully managed execution runner for Apache Beam (supporting Java, Python, and Go).
Core Strengths: Unified programming model for both batch and streaming. Serverless operations with automated dynamic work rebalancing (eliminating stragglers). Seamless integration with Streaming Engine and FlexRS.
Best For: Complex streaming event processing, tumbling/sliding/session windows, out-of-order event handling with watermarks, and high-scale transformations.
2. Cloud Data Fusion
Underlying Engine: Fully managed, visual data integration service built on the open-source CDAP (Cask Data Application Platform) ecosystem.
Core Strengths: Graphical drag-and-drop web UI (Pipeline Studio), pre-built library of over 150+ connectors to transactional databases, SaaS platforms, and mainframes. Built-in data profiling and wrangling.
Cost Profile: High fixed baseline cost. The Cloud Data Fusion instance runs 24/7 (Basic or Enterprise tier), and each pipeline execution spins up an underlying Cloud Dataproc cluster to execute the CDAP pipeline.
Best For: Enterprise ETL developers, business analysts, and compliance teams migrating legacy ETL workflows (Informatica, Talend) who mandate visual pipeline development without writing custom code.
3. Cloud Dataproc
Underlying Engine: Managed Apache Spark, Apache Hadoop, Presto, and Flink clusters.
Core Strengths: Fast cluster provisioning (<90 seconds), seamless integration with Google Cloud Storage via the Cloud Storage Connector ( gs:// replacing hdfs:// ).
Best For: Migrating existing, mature on-premises Spark and Hadoop codebases to the cloud with zero code rewrites.
Feature Cloud Dataflow Cloud Data Fusion Cloud Dataproc
Primary Paradigm Apache Beam (Unified Batch & Stream) Visual ETL / ELT (CDAP Framework) Apache Spark, Hadoop, Hive, Flink
Development Style Code-first (Java, Python, Go) Drag-and-Drop Visual Studio Code-first (PySpark, Scala, SQL)
Infrastructure Management 100% Serverless (Zero cluster config) Managed instance + Dataproc clusters Managed VMs or Dataproc Serverless
Streaming Latency Sub-second (Streaming Engine) Batch-oriented (Micro-batch possible) Seconds (Spark Structured Streaming)
Cost Model Pay per vCPU/GB-hr consumed High base instance fee + VM costs Compute Engine VM rates + Dataproc fee
- Autoscaling Dynamic Work Rebalancing (Instant) Dataproc worker scaling Dynamic Allocation / Serverless
- 6. Concrete Exam Scenarios & Architecture Pitfalls
Scenario / Architecture Challenge Common Anti-Pattern Correct Google Cloud Architecture
Cost-Effective Hourly Reporting ; A data team ingests 5 GB of web logs per hour into BigQuery for internal analytics queried twice a day. Using the legacy BigQuery streaming insert API continuously around the clock. Land log files in Cloud Storage and execute free BigQuery batch load jobs ( bq load ) via Cloud Composer or Cloud Functions. Zero compute cost for BigQuery loading.
Flash Sale Ingestion Surge ; An e-commerce mobile application experiences 20x checkout transaction spikes during seasonal promotions. Connecting mobile clients directly to an autoscaling Compute Engine ingestion service writing to BigQuery. Publish checkout events to Cloud Pub/Sub to act as a resilient shock absorber. Use a direct Pub/Sub BigQuery Subscription utilizing the Storage Write API to ingest records serverlessly.
Cost-Sensitive Overnight Batch ETL ; A financial institution processes a 10 TB overnight batch transformation job in Apache Beam that must finish within 8 hours. Running standard Dataflow Batch workers with default on-demand n1-standard-4 instances. Launch the Dataflow batch pipeline with Flexible Resource Scheduling (FlexRS) ( flexrsGoal=COST_OPTIMIZED ). FlexRS combines Spot VMs with standard VMs, reducing compute costs by up to 40%.
Zero-Code Enterprise Database Ingestion ; A team of data analysts with zero Java/Python coding skills needs to ingest data from 40 on-premises SQL Server databases into BigQuery. Forcing the analysts to write and maintain complex custom Apache Beam pipelines on Dataflow. Deploy Cloud Data Fusion . Analysts leverage the visual Pipeline Studio and pre-built JDBC connectors to design, orchestrate, and deploy ETL pipelines without writing code.