Study guide
11.2 Stream and Batch Processing: Dataflow & Dataproc
Google Professional Cloud Architect Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview, Strategy & Case Study Methodology
1.1 Exam Overview, Structure & Scoring Logistics 1.2 The Professional Cloud Architect Role & Exam Mindset 1.3 Deconstructing GCP Case Studies & Scenario Questions
2 Chapter 2: Designing for Business Requirements
2.1 Mapping Business Objectives, ROI & Total Cost of Ownership 2.2 Cost Optimization Strategies, Pricing Models & Trade-Offs 2.3 Business Continuity, Governance & Compliance Planning
3 Chapter 3: Designing for Technical Requirements & Availability
3.1 Scalability, Elasticity & Capacity Planning 3.2 High Availability, Disaster Recovery Patterns & RTO/RPO 3.3 Performance Optimization & Global Latency Reduction
4 Chapter 4: Compute Selection & Workload Architecture
4.1 Compute Engine Architecture, Sizing & Managed Instance Groups 4.2 Google Kubernetes Engine (GKE) Standard vs Autopilot & Microservices 4.3 Serverless Architectures: Cloud Run, Cloud Functions & Eventarc
5 Chapter 5: Storage, Database & Data Warehouse Architecture
5.1 Cloud Storage Architecture, Storage Classes & Lifecycle Management 5.2 Relational Databases: Cloud SQL, AlloyDB & Cloud Spanner 5.3 NoSQL & Big Data Storage: Firestore, Bigtable & BigQuery
6 Chapter 6: Networking, Hybrid Connectivity & Traffic Routing
6.1 VPC Design, Subnets, Shared VPC & VPC Peering 6.2 Hybrid & Multicloud Connectivity: Cloud VPN & Cloud Interconnect 6.3 Cloud Load Balancing, Cloud CDN & Cloud DNS
7 Chapter 7: Identity, Access & Resource Governance
7.1 Cloud IAM Hierarchy, Custom Roles & Service Accounts 7.2 Workload Identity Federation, Service Account Impersonation & IAP 7.3 Organization Policies, Tags & Resource Governance
8 Chapter 8: Security Architecture, Data Protection & Compliance
8.1 Data Encryption at Rest/Transit, Cloud KMS, CMEK & Cloud HSM 8.2 Network Security: Cloud Armor, VPC Service Controls & Firewalls 8.3 Compliance Frameworks, Sensitive Data Protection & Security Command Center
9 Chapter 9: DevOps, Automation & Infrastructure as Code
9.1 Infrastructure as Code (IaC) with Terraform on Google Cloud 9.2 CI/CD Pipelines: Cloud Build, Artifact Registry & Cloud Deploy 9.3 Deployment Strategies (Blue/Green, Canary) & GitOps with Config Sync 9.4 Interacting with Google Cloud Programmatically: Cloud SDK, Cloud Shell & Emulators
10 Chapter 10: Site Reliability Engineering & Operations Excellence
10.1 SRE Fundamentals: SLIs, SLOs, SLAs & Error Budgets 10.2 Observability: Cloud Monitoring, Logging & Alert Policies 10.3 Diagnosing Performance: Cloud Trace, Profiler & Error Reporting
11 Chapter 11: Data Pipelines, Integration & Modernization
11.1 Event-Driven Architectures with Cloud Pub/Sub & Eventarc 11.2 Stream and Batch Processing: Dataflow & Dataproc 11.3 API Management & Service Integration: Apigee & Cloud API Gateway 11.4 Migration Planning & Execution: Migration Center, DMS, Datastream & Transfer Options
12 Chapter 12: Well-Architected Framework & Official Case Study Solutions
12.1 Google Cloud Well-Architected Framework & FinOps Governance 12.2 Comprehensive Case Studies: EHR Healthcare & Cymbal Retail 12.3 Comprehensive Case Studies: Altostrat Media & KnightMotives Automotive
13 Chapter 13: AI, Machine Learning & Generative AI Architecture
13.1 AI/ML Platform Architecture: Vertex AI, Gemini Enterprise Agent Platform & AI Hypercomputer 13.2 Prebuilt AI APIs, Model Garden, Gemini Cloud Assist & Securing AI
11.2 Stream and Batch Processing: Dataflow & Dataproc
Key Takeaways Explain This Topic AI
Cloud Dataflow provides a fully managed, serverless execution engine for the Apache Beam SDK, unifying batch (bounded) and stream (unbounded) processing pipelines with end-to-end exactly-once semantics.
Dataflow eliminates stragglers and runtime bottlenecks via Dynamic Work Rebalancing and Streaming Engine, which offloads pipeline state and shuffle operations from worker VMs.
Stream processing in Dataflow manages time dimensions using Fixed, Sliding, and Session windows, Event-Time Watermarks, and allowed lateness with accumulating or discarding panes.
Dataflow Prime introduces resource-based horizontal and vertical autoscaling, right-fitting worker machine resources dynamically to individual pipeline execution stages.
Cloud Dataproc provides fast (<90s) managed clusters for Apache Spark and Hadoop, optimizing costs via ephemeral job-scoped clusters with Spot/Preemptible secondary workers, Dataproc Serverless, and Dataproc on GKE.
Last updated: August 2026
Stream and Batch Processing: Dataflow & Dataproc
Architectural Selection Principle: Selecting between Cloud Dataflow and Cloud Dataproc depends primarily on the software ecosystem, operational model, and pipeline semantics. Choose Cloud Dataflow for new, cloud-native pipelines requiring serverless operation, unified batch/stream processing via Apache Beam, advanced windowing, and zero infrastructure management. Choose Cloud Dataproc when migrating existing open-source big data ecosystems (Apache Spark, Hadoop, Hive, Presto, Flink), executing ephemeral job-scoped Spark workloads, or utilizing customized JVM configurations on cost-effective Spot VMs.
Cloud Dataflow: Architecture & Apache Beam Mechanics
Cloud Dataflow is a fully managed, serverless distributed execution framework for executing Apache Beam pipelines (written in Java, Python, Go, or SQL). Dataflow abstracts away all underlying compute infrastructure, handling provisioning, horizontal scaling, graph optimization, and fault tolerance automatically. +-----------------------------------------------------------------------------------------+
- | CLOUD DATAFLOW EXECUTION MODEL |
- +-----------------------------------------------------------------------------------------+
- | PIPELINE DEFINITION | Apache Beam SDK (Java / Python / Go / Beam SQL) |
- | | - Unified abstractions: PCollection (Data) & PTransform (Logic)|
- +-----------------------+-----------------------------------------------------------------+
- | GRAPH OPTIMIZATION | Execution Graph Optimization & Pipeline Fusion |
- | | - Fuses adjacent DoFns into single stages to eliminate I/O |
- +-----------------------+-----------------------------------------------------------------+
- | DISTRIBUTED RUNTIME | Dynamic Work Rebalancing + Streaming Engine |
- | | - State & Shuffle offloaded to managed Google backend |
- +-----------------------+-----------------------------------------------------------------+
- | AUTOSCALING | Horizontal Worker Scaling + Dataflow Prime Vertical Scaling |
- +-----------------------------------------------------------------------------------------+
- Core Apache Beam Abstractions
Pipeline: Encapsulates the entire data processing workflow from data ingestion to transformation and output sinking.
PCollection (Parallel Collection): Represents a distributed, multi-element dataset. A PCollection can be bounded (finite, fixed-size dataset for batch processing) or unbounded (continuous, infinite data stream for real-time processing).
PTransform (Parallel Transform): Represents an operation that takes one or more PCollections as input, performs transformation logic (e.g., ParDo , GroupByKey , Combine , Filter ), and outputs one or more PCollections .
Pipeline Fusion: Before executing a pipeline, Dataflow optimizes the execution graph by fusing adjacent ParDo operations into single execution steps. This eliminates unnecessary intermediate data serialization, disk writes, and network roundtrips between worker VMs.
Dynamic Work Rebalancing: In batch pipelines, uneven data distribution can cause certain worker nodes to become "stragglers," holding up pipeline completion. Dataflow continuously monitors worker progress in real time and dynamically splits remaining unprocessed work from lagging workers, redistributing it across idle workers to minimize overall execution time. DYNAMIC WORK REBALANCING (STRAGGLER MITIGATION)
- Worker 1 (Fast): [ Finished Chunk A ] ---> [ IDLE ] <==================+
- Worker 2 (Lagging): [ Processing Chunk B .............................. ] |
- |
- Dataflow Runtime Splits Work Automatically |
- v |
- Worker 2 (Relieved):[ Processing Chunk B1 ] |
- Worker 1 (Assisting): [ Takes Over Chunk B2 ] <-------+
- Dataflow Streaming Engine & Exactly-Once Semantics
Streaming Engine Architecture: Traditional stream runners store intermediate window state and shuffle data directly on worker VM disks. Dataflow Streaming Engine decouples state storage and shuffle processing from worker VMs, moving state into a dedicated, highly optimized Google Cloud backend service. This drastically reduces worker VM CPU and memory overhead, accelerates autoscaling responsiveness, and provides smoother throughput.
End-to-End Exactly-Once Processing: Dataflow ensures that every record in an unbounded stream is processed exactly once by coordinating checkpoints, tracking message IDs, and integrating with transactional sinks (such as BigQuery Storage Write API with exactly-once stream semantics, Cloud Spanner, and Cloud Storage with metadata tracking).
Windowing, Watermarks & Handling Late Data
Real-time stream processing must resolve the fundamental disconnect between Event Time (when the event actually occurred on the client device) and Processing Time (when the event reaches the processing engine over the network). EVENT TIME VS. PROCESSING TIME & WATERMARKS
- Event Time Clock: 12:00 ---- 12:05 ---- 12:10 ---- 12:15 ---- 12:20
- | | | |
- | (Network Delays) | |
- v v v v
- Processing Time: 12:02 ---- 12:09 ---- 12:14 ---- 12:26
- ^
- [ Watermark Advances to 12:05 ]
- (System assumes all <= 12:05 events have arrived)
- 1. Windowing Strategies
Windows partition unbounded streams into finite temporal chunks for aggregation.
Window Type Description Behavioral Mechanics Typical Use Cases
Fixed (Tumbling) Windows Non-overlapping, consistent time intervals (e.g., 5 minutes). Every event belongs to exactly one window (e.g., [12:00-12:05), [12:05-12:10)). Calculating 5-minute metric rollups, hourly revenue totals.
Sliding (Hopping) Windows Overlapping time intervals defined by duration and period (e.g., 10-min duration, 1-min slide). An individual event belongs to multiple overlapping windows simultaneously. Calculating 10-minute moving averages refreshed every 60 seconds.
Session Windows Dynamic, data-driven windows based on periods of user activity separated by a gap duration (e.g., 30 min of inactivity). Windows expand dynamically per user/key; closes when no events arrive within the gap duration. Tracking user web sessions, gaming play sessions, fraud burst patterns.
Global Windows Single default window spanning all time across the entire dataset. Requires explicit non-default triggers (e.g., element count) to emit results on unbounded streams. Continuous running totals, anomaly detection with custom count triggers.
2. Watermarks, Allowed Lateness & Panes
Watermarks: A watermark is a dynamically computed temporal threshold that represents the pipeline's confidence that all data up to event time T T T has arrived. Once the watermark passes the end of a window, the window is evaluated and emitted.
Allowed Lateness ( withAllowedLateness ): Mobile devices and IoT sensors frequently experience offline network disconnections, delivering data hours after the watermark has passed. Allowed lateness specifies how long Dataflow keeps window state open after the watermark has closed the window.
Accumulating vs. Discarding Panes: When late data arrives for an already-emitted window:
Accumulating Mode: Dataflow re-emits the complete window total including both original and newly arrived late records (e.g., emits 100, then emits 105).
Discarding Mode: Dataflow emits only the incremental delta brought by the late data (e.g., emits 100, then emits +5). WINDOW EMISSION & LATE DATA TIMELINE
- Window [12:00 - 12:05)
- |--- Normal Events Arrive ---> Watermark passes 12:05 ---> [ Pane 1 Emitted: Total = 100 ]
- |
- |--- (Window closed, but within Allowed Lateness = 10m) ------------+
- |--- Late Event (EventTime 12:03) arrives at ProcessingTime 12:12
- | |
- +---> [ Accumulating Mode ] ----------------------------------------> [ Pane 2 Emitted: Total = 105 ]
- +---> [ Discarding Mode ] ----------------------------------------> [ Pane 2 Emitted: Delta = +5 ]
- Dataflow Prime: Next-Generation Serverless Optimization
Dataflow Prime enhances standard Cloud Dataflow with intelligent resource management and automated infrastructure optimization.
Resource-Based Autoscaling: Traditional Dataflow scales by adding identical worker VMs (homogeneous scaling). Dataflow Prime provisions compute resources (CPU vs. Memory) dynamically per pipeline step. If a transform is memory-heavy (e.g., large cache lookup) but another is CPU-heavy (e.g., encryption), Prime assigns right-sized resources to individual stages without over-provisioning the entire cluster.
Vertical Autoscaling for Memory: Automatically detects impending Out-Of-Memory (OOM) conditions during large aggregations and dynamically increases worker memory allocations on the fly, preventing costly pipeline crashes and retries.
Right-Fitting Recommendations: Analyzes historical pipeline metrics in Cloud Logging/Monitoring to provide actionable machine configuration recommendations.
Cloud Dataproc: Architecture & Cluster Topologies
Cloud Dataproc is Google Cloud's fully managed service for open-source big data processing engines, including Apache Spark , Apache Hadoop , Hive , Presto/Trino , Pig , and Flink . +-----------------------------------------------------------------------------------------+
- | CLOUD DATAPROC CLUSTER TOPOLOGY |
- +-----------------------------------------------------------------------------------------+
- | MASTER / PRIMARY WORKERS (Standard Persistent VMs) |
- | - Runs HDFS NameNode, YARN ResourceManager, Spark Driver, Hive Metastore |
- | - State & cluster metadata maintained; NEVER run on Spot VMs |
- +-----------------------------------------------------------------------------------------+
- | PRIMARY WORKERS (Standard VMs) |
- | - Runs YARN NodeManager & HDFS DataNode; maintains block storage |
- +-----------------------------------------------------------------------------------------+
- | SECONDARY WORKERS (Spot / Preemptible VMs) |
- | - Compute-only (YARN NodeManagers); DO NOT store HDFS data |
- | - Scaled up/down aggressively; up to 80% cost reduction; zero HDFS corruption on loss |
- +-----------------------------------------------------------------------------------------+
- | PERSISTENCE TIER: Cloud Storage (gs://data-lake-bucket) via Cloud Storage Connector |
- +-----------------------------------------------------------------------------------------+
- Modern Dataproc Deployment Models
Ephemeral (Job-Scoped) Clusters:
Rather than maintaining an expensive 24/7 static Hadoop cluster, architects deploy an ephemeral cluster via Cloud SDK/Airflow, execute a single Spark/Hive job, and immediately delete the cluster upon job completion.
Rapid Provisioning: Dataproc clusters initialize in under 90 seconds , making ephemeral job execution operationally efficient.
Decoupled Storage: Persistent cluster storage is completely replaced by Google Cloud Storage (GCS) using the open-source Cloud Storage Connector ( gs:// connector instead of hdfs:// ). Cluster termination causes zero data loss.
Primary vs. Secondary (Spot/Preemptible) Workers:
Primary Workers: Run both YARN NodeManagers and HDFS DataNodes. Primary workers must be standard Compute Engine instances to preserve cluster stability and HDFS replication integrity.
Secondary Workers (Compute-Only): Run only YARN NodeManagers and execute stateless Spark executors. They do not participate in HDFS storage. Secondary workers can be configured as Spot VMs (or Preemptible VMs). If Google Cloud reclaims Spot VMs, Dataproc reschedules lost executor tasks onto remaining nodes without risking HDFS data corruption or cluster failure.
Dataproc Serverless:
Enables developers to submit Spark batch workloads (PySpark, Spark SQL, Spark R, Java/Scala) directly without creating, configuring, sizing, or maintaining any cluster infrastructure.
Google Cloud provisions and auto-tunes Spark runtime containers on demand, billing strictly for the fractional CPU cores and memory utilized per second of job execution.
Dataproc on GKE:
Deploys Spark workloads directly onto existing Google Kubernetes Engine (GKE) clusters, unifying containerized microservices and big data batch processing on a single shared compute plane.
Cloud Dataflow vs. Cloud Dataproc Decision Matrix
Evaluation Dimension Cloud Dataflow Cloud Dataproc
Core Framework Apache Beam (Java, Python, Go, SQL). Apache Spark, Hadoop, Hive, Presto/Trino, Flink.
Infrastructure Management 100% Serverless ; zero VM, OS, or cluster administration. Managed clusters (Ephemeral or Long-lived) or Dataproc Serverless for Spark.
Scaling Mechanics Horizontal autoscaling + Dynamic Work Rebalancing + Dataflow Prime vertical scaling. Cluster Autoscaler (YARN metrics) + Spot VM secondary worker pools.
Streaming / Batch Paradigm Unified programming model; identical primitives for bounded & unbounded data. Separate APIs (Spark Streaming / Structured Streaming vs Spark Core RDD/DataFrames).
Windowing & Late Data Native, sophisticated event-time windowing, watermarks, and allowed lateness. Basic micro-batch / structured streaming windowing; requires manual watermarking logic.
Migration Strategy Ideal for greenfield cloud-native stream/batch architectures. Ideal for lift-and-shift of existing Hadoop/Spark codebases with minimal rewrites.
Startup Latency Pipeline worker provisioning: ~3-5 minutes. Cluster provisioning: < 90 seconds ; Serverless job startup: ~30-60s.
Pricing Model Sized by vCPU, memory, and Streaming Engine data processed. Underlying Compute Engine VM pricing + Dataproc management fee ($0.01/vCPU/hr) or Serverless DCU.
- Concrete Architectural Scenario: Connected Vehicle IoT Fleet Analytics
- Scenario Profile
Client: Automotive OEM collecting real-time telemetry (speed, engine temperature, GPS) from 2 million connected vehicles.
Requirements:
Ingest 500,000 telemetry messages/sec from Cloud Pub/Sub.
Compute 1-minute moving average speeds and detect engine overheating alerts within 3 seconds of event occurrence.
Handle delayed telemetry uploads from vehicles traveling through cellular dead zones (up to 30 minutes late).
Train a daily fleet maintenance prediction model using an existing complex Apache Spark MLlib library. [ 2M Connected Vehicles ] ---> [ Cloud Pub/Sub Topic ]
- |
- v
- [ Cloud Dataflow Pipeline (Streaming Engine) ]
- Sliding Windows (1-min duration, 10s slide)
- Watermarks + Allowed Lateness (30 minutes)
- Dynamic Work Rebalancing & Exactly-Once Ingestion
- |
- +------------------+------------------+
- | |
- v v
- [ BigQuery Real-Time Dashboard ] [ Cloud Storage Data Lake (Parquet) ]
- (Sub-3s Overheating Anomaly Alerts) |
- v
- [ Ephemeral Dataproc Cluster ]
- Provisioned via Cloud Composer (Airflow)
- Primary: 2 Standard VMs | Secondary: 50 Spot VMs
- Runs Daily Spark MLlib Training Job
- Deletes Cluster Immediately on Completion (<90s init)
- Architecture Blueprint
Real-Time Stream Processing: Cloud Dataflow reads unbounded streams from Cloud Pub/Sub. Telemetry is grouped into Sliding Windows (1-minute window, 10-second slide) to calculate moving averages. To accommodate intermittent cellular connectivity, Dataflow sets allowedLateness(Duration.standardMinutes(30)) with accumulating panes .
Serverless Streaming Engine: Dataflow Streaming Engine offloads pipeline state, ensuring real-time anomaly detection alerts reach BigQuery and operations teams in under 3 seconds.
Batch Machine Learning Pipeline: Raw telemetry is persisted in Parquet format on Cloud Storage. A Cloud Composer (Managed Airflow) DAG provisions an ephemeral Cloud Dataproc cluster daily. The cluster uses 2 Standard Master/Worker VMs and 50 Spot VM Secondary Workers to execute the existing Spark MLlib model at minimal cost, terminating the cluster immediately upon completion.
Important
Exam Watch:
If an exam scenario describes migrating an existing Apache Spark, Hadoop, Hive, or Presto codebase to Google Cloud with minimal engineering effort and the lowest operational disruption, Cloud Dataproc is the correct answer.
To minimize Dataproc batch processing costs, configure ephemeral clusters storing persistent data in Cloud Storage ( gs:// ) and utilizing Secondary Workers on Spot / Preemptible VMs .
If the scenario demands a serverless, unified batch and stream pipeline with complex event-time windowing , watermarks , late data handling , or Dynamic Work Rebalancing , choose Cloud Dataflow (Apache Beam).