Study guide
8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing
Google Cloud Certified Professional Data Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Security, IAM, and Data Protection
1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
2 Chapter 2: Reliability and Compliance Architecture
2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack
3 Chapter 3: Cross-Cloud and Federated Governance
3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance
4 Chapter 4: Data Ingestion and Pipeline Planning
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion
5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub
5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow
6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows
7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics
7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management
8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion
8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage
9 Chapter 9: Storage Selection and Cloud Storage Architecture
9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools
10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL
10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery
11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore
11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections
12 Chapter 12: Enterprise Data Warehousing with BigQuery
12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions
13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery
13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling
14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML
14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles
15 Chapter 15: Operations, Orchestration, and Data Governance
15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services
8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing
Key Takeaways Explain This Topic AI
Dataproc Serverless runs Apache Spark batch workloads (PySpark, Spark SQL, Spark R, Spark Scala) in fully managed, dynamic execution environments without requiring users to provision, configure, or manage Compute Engine clusters.
Dataproc Serverless charges strictly for active workload execution metered in Dataproc Compute Units (DCUs) per second, with 1 DCU providing 1 vCPU and 4 GB of RAM, driving idle compute costs to zero.
Dynamic autoscaling policies on managed Dataproc clusters evaluate YARN pending memory and virtual core metrics, scaling workers horizontally between configured minimum and maximum boundaries.
Graceful Decommissioning prevents Spark FetchFailedExceptions and stage recomputations by allowing active YARN NodeManagers to finish running tasks and safely transfer shuffle blocks before instances are terminated during scale-down events.
Custom container images hosted in Google Artifact Registry allow Dataproc Serverless batch workloads to package custom Python wheels, C/C++ native libraries, and proprietary dependencies in a reproducible environment.
Last updated: September 2026
8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing
Exam Focus: The Google Cloud Professional Data Engineer exam tests both fully managed serverless data processing and elastic cluster scaling. You must understand how Dataproc Serverless for Apache Spark eliminates infrastructure management, how resource allocation is metered via Dataproc Compute Units (DCUs) , the networking and Private Google Access requirements for serverless execution, how autoscaling policies monitor YARN metrics, and why Graceful Decommissioning is mandatory to prevent shuffle data loss and pipeline failures during cluster downscaling.
While ephemeral Dataproc clusters significantly optimize cloud spend compared to 24/7 clusters, they still require data engineers to manage virtual machine sizing, cluster initialization scripts, OS patches, and teardown logic. To eliminate this operational burden, Google Cloud introduced Dataproc Serverless for Apache Spark . Dataproc Serverless abstracts the underlying infrastructure entirely, provisioning dynamic Spark runtimes on demand. Alongside dynamic Autoscaling Policies and Graceful Decommissioning for managed clusters, data engineers possess a comprehensive toolkit for executing batch workloads at any scale.
1. Dataproc Serverless for Apache Spark Architecture
Dataproc Serverless enables data engineers to submit Spark batch workloads directly to Google Cloud without creating or managing Compute Engine clusters. +───────────────────────────────────────────────────────────────────────────────────+
- | DATAPROC SERVERLESS ARCHITECTURE |
- +───────────────────────────────────────────────────────────────────────────────────+
- | |
- | [ Developer / Airflow ] ──(gcloud dataproc batches submit pyspark) |
- | │ |
- | ▼ |
- | +─────────────────────────────────────────────────────────────────────────────+ |
- | | GOOGLE-MANAGED SERVERLESS SPARK CONTROL PLANE | |
- | | - Auto-allocates Driver & Dynamic Executors in Sandboxed Runtimes | |
- | | - Enforces Subnet & Private Google Access Validation | |
- | | - Injects Custom Container from Artifact Registry (if specified) | |
- | +─────────────────────────────────────────────────────────────────────────────+ |
- | │ |
- | ▼ Provisions Dynamically |
- | +─────────────────────────────────────────────────────────────────────────────+ |
- | | CUSTOMER VPC NETWORK (Private Subnet) | |
- | | | |
- | | [ Spark Driver Pod ] <══(Internal TCP Ports 0-65535)══> [ Spark Executors ]|
- | | │ │ | |
- | +─────────┼──────────────────────────────────────────────────────┼────────────+ |
- | ▼ ▼ |
- | [ Dataproc Metastore (DPMS) ] [ Cloud Storage (gs://) ]
- +───────────────────────────────────────────────────────────────────────────────────+
- Operational Workflow
When a developer or orchestrator submits a batch job via the Dataproc API ( gcloud dataproc batches submit ):
Google Cloud dynamically provisions a dedicated, fully isolated Spark driver and an initial set of Spark executors.
The job pulls code artifacts and dependencies directly from Cloud Storage ( gs:// ) or a custom container image.
As the Spark application runs, Dataproc Serverless dynamically scales the number of executors up or down based on data volume, stage dependencies, and workload intensity.
Upon completion, the runtime environment is immediately destroyed. Logs are exported to Cloud Logging, metrics are streamed to Cloud Monitoring, and persistent Spark History is stored in Cloud Storage.
Resource Model: Dataproc Compute Units (DCUs)
Dataproc Serverless abstracts physical CPU and RAM into Dataproc Compute Units (DCUs) :
Definition: 1 DCU represents 1 vCPU and 4 GB of memory .
Allocation: Compute is allocated to Spark drivers and executors in DCU increments (e.g., standard driver size is 2 DCUs = 2 vCPUs and 8 GB RAM).
Metering: Billed strictly per second for the total DCU-hours consumed during active job execution, with a 1-minute minimum runtime. If a batch job executes for 2 minutes and 15 seconds across 10 DCUs, billing reflects exactly 135 seconds of compute.
Mandatory Networking Prerequisites
A frequent source of exam questions involves network configuration failures during Dataproc Serverless deployments:
Private Google Access (PGA): The VPC subnet where the batch executes must have Private Google Access enabled . Because serverless worker pods do not have external public IP addresses, PGA is required to route traffic internally to Google APIs (Cloud Storage, Dataproc Metastore, BigQuery, Artifact Registry).
Internal Subnet Firewall Rule: Spark drivers and executors communicate extensively across arbitrary network ports during shuffle phases. An ingress firewall rule must be configured allowing all internal TCP traffic ( tcp:0-65535 ) between instances within the target subnet tag. # Submitting a PySpark Batch Job to Dataproc Serverless
- gcloud dataproc batches submit pyspark gs://my-lake/scripts/transform.py \
- -project=my-data-project \
- -region=us-central1 \
- -subnet=projects/my-data-project/regions/us-central1/subnetworks/analytics-subnet \
- -deps-bucket=gs://my-lake/staging \
- -metastore-service=projects/my-data-project/locations/us-central1/services/central-dpms \
- -properties=spark.dynamicAllocation.maxExecutors=50,spark.executor.cores=4 \
- -version=2.1
- Custom Container Images with Artifact Registry
By default, Dataproc Serverless provides standard base container images containing Apache Spark, Java, Python, and the GCS/BigQuery connectors. However, enterprise workloads frequently require custom Python libraries, specific C dependencies, or specialized database drivers.
Mechanism: Developers create a Dockerfile based on Google's official Dataproc Serverless base image ( gcr.io/cloud-dataproc/spark-base ), install custom dependencies (e.g., PyTorch, proprietary ODBC drivers), push the image to Artifact Registry , and specify --container-image during batch submission. # Custom Container Image for Dataproc Serverless
FROM gcr.io/cloud-dataproc/spark-base:2.1
- # Install system dependencies
- RUN apt-get update && apt-get install -y libpq-dev gcc && rm -rf /var/lib/apt/lists/*
- # Install specialized Python wheels
- RUN pip install --no-cache-dir \
- pyarrow==14.0.1 \
- fastparquet==2023.10.1 \
- snowflake-connector-python==3.5.0
- 2. Serverless vs. Managed Cluster Decision Matrix
Understanding when to use Dataproc Serverless versus managed Dataproc clusters is critical for architecture design:
Evaluation Criteria Dataproc Serverless Managed Dataproc (Ephemeral) Managed Dataproc (Persistent)
Infrastructure Management Zero (no VMs, no OS patching, no cluster scripts) Low (declarative cluster creation and teardown in DAGs) High (ongoing VM management, OS updates, disk sizing)
Startup Latency 30 to 60 seconds 60 to 90 seconds (VM provisioning and init actions) Instantaneous (cluster is already running)
Workload Scope Individual batch applications (Spark only) Single batch pipeline or orchestrated sequence Multi-tenant batch, streaming, and interactive queries
Supported Engines Apache Spark (PySpark, SQL, Scala, R) Spark, Hadoop MapReduce, Hive, Presto/Trino, Flink Spark, Hive, Presto/Trino, HBase, ZooKeeper
Interactive Notebooks & Daemons Not supported (no 24/7 daemons or Thrift servers) Not supported for interactive use Supported (Zeppelin, Jupyter, HiveServer2, Thrift Server)
Hardware Customization Standard CPU/RAM DCU allocations Extreme (Local NVMe SSDs, GPUs, custom vCPU ratios) Extreme (Local NVMe SSDs, GPUs, custom shapes)
Cost Profile Billed per second of execution; zero idle cost Billed for cluster duration; zero idle cost between jobs Continuous 24/7 billing; high risk of idle resource waste
Exam Rule of Thumb:
Choose Dataproc Serverless for standard Apache Spark batch ETL jobs where you want zero operational overhead and no VM management.
Choose Managed Ephemeral Dataproc when you need non-Spark tools (MapReduce, Hive, Flink, Trino), specialized hardware (local NVMe SSDs for massive shuffle, GPUs for ML training), or custom OS-level kernel tuning.
Choose Managed Persistent Dataproc only when hosting long-running interactive daemons (e.g., Spark Thrift Server or HiveServer2 serving corporate BI dashboards 24/7).
3. Dynamic Autoscaling Policies on Managed Clusters
For managed Dataproc clusters handling variable workloads, Autoscaling Policies automate the horizontal scaling of worker nodes based on real-time resource pressure. +───────────────────────────────────────────────────────────────────────────────────+
- | DATAPROC AUTOSCALING DECISION LOOP |
- +───────────────────────────────────────────────────────────────────────────────────+
- | |
- | YARN CapacityScheduler reports: |
- | - Allocated Memory vs Total Memory |
- | - Pending Memory Containers (Requests queued due to lack of RAM/vCores) |
- | │ |
- | ▼ |
- | [ Autoscaling Algorithm Evaluates ] |
- | - Pending Memory > 0 for Cooldown Period? |
- | - Scale-Up Factor (0.0 to 1.0) applied |
- | │ |
- | ┌─────────────────────────────┴─────────────────────────────┐ |
- | ▼ ▼ |
- | [ SCALE UP ] [ SCALE DOWN ] |
- | Provisions Secondary Spot Workers Enters Graceful |
- | up to maxInstances limit. Decommissioning; |
- | Instantly expands YARN capacity. migrates shuffle. |
- +───────────────────────────────────────────────────────────────────────────────────+
- Autoscaling Metrics and Algorithms
Dataproc autoscaling relies on metrics exposed by the Apache Hadoop YARN ResourceManager :
Primary Metric: yarn-memory-allocated-percentage and pending memory containers . If applications submit Spark tasks that exceed current cluster RAM, YARN places containers into a PENDING queue.
Scale-Up Mechanics: When pending memory exceeds zero, the autoscaler computes the required capacity and scales up worker nodes by scaleUpFactor (a fractional value between 0.0 and 1.0) up to the configured maxInstances .
Scale-Down Mechanics: When cluster utilization drops below target thresholds, the autoscaler computes excess capacity and scales down by scaleDownFactor .
Cooldown Period: Configures the minimum quiet period (e.g., cooldownPeriod = 120s ) between consecutive scaling actions to prevent rapid oscillation (thrashing).
Autoscaling Policy YAML Configuration # Production Autoscaling Policy: burst-policy.yaml
basicAlgorithm:
yarnConfig:
- scaleUpFactor: 0.5 # Aggressively scale up by 50% of needed capacity
- scaleDownFactor: 0.2 # Conservatively scale down by 20% to prevent churn
- scaleUpMinWorkerNum: 2 # Add at least 2 workers per scale-up event
- scaleDownMinWorkerNum: 1 # Remove at least 1 worker per scale-down event
- gracefulDecommissionTimeout: 3600s # Allow 1 hour for shuffle migration
- cooldownPeriod: 180s # Wait 3 minutes between scaling decisions
workerConfig:
- minInstances: 2 # Keep exactly 2 primary workers for stability
- maxInstances: 2 # Primary workers never scale; fixed baseline
secondaryWorkerConfig:
minInstances: 0 # Scale down to 0 secondary workers when idle
maxInstances: 50 # Burst up to 50 Spot VMs during peak load # Create and apply the autoscaling policy to an existing cluster
- gcloud dataproc autoscaling-policies import burst-policy \
- -source=burst-policy.yaml \
- -region=us-central1
- gcloud dataproc clusters update my-dataproc-cluster \
- -autoscaling-policy=burst-policy \
- -region=us-central1
- 4. Graceful Decommissioning Mechanics
In distributed data processing, worker nodes host two vital assets:
Active task threads: CPU operations executing maps, filters, or reductions.
Intermediate shuffle blocks: Data partitions written to local persistent disks during wide transformations (e.g., groupByKey , reduceByKey , SQL joins) that downstream stages must fetch over the network.
The Disaster of Ungraceful Downscaling
If an autoscaler abruptly deletes a Compute Engine VM the instant its CPU becomes idle:
Intermediate shuffle blocks stored on the deleted VM's local disk vanish permanently.
Downstream Spark executors attempting to fetch those blocks encounter an org.apache.spark.shuffle.FetchFailedException .
Spark aborts the current stage and forces the cluster to re-compute all parent stages from scratch on remaining workers.
If multiple nodes are terminated in rapid succession, the cascading FetchFailedException limit is exceeded (default: 4 attempts), causing the entire Spark batch job to crash .
How Graceful Decommissioning Solves This
When Dataproc initiates a scale-down event under an autoscaling policy with gracefulDecommissionTimeout :
Traffic Draining: YARN marks the target NodeManagers as DECOMMISSIONING . The ResourceManager stops assigning new task containers to these nodes.
Task Completion: Currently executing tasks are permitted to run to completion.
Shuffle Preservation: If using the Spark External Shuffle Service or Cloud Storage shuffle tracking, Dataproc ensures shuffle blocks are preserved or streamed to surviving nodes before VM termination.
Timeout Boundary: If running tasks do not complete before gracefulDecommissionTimeout expires, Dataproc forcibly deletes the VM instances to prevent indefinite budget overruns. For heavy batch workloads, configure gracefulDecommissionTimeout between 1800s (30 min) and 3600s (1 hour).
5. Enterprise Migration Strategy: On-Premises Hadoop/Spark to GCP
Migrating enterprise Hadoop/Spark environments (Cloudera, Hortonworks, MapR) to Google Cloud requires a phased methodology that minimizes business risk and eliminates technical debt: +───────────────────────────────────────────────────────────────────────────────────+
- | FOUR-PHASE HADOOP MIGRATION ROADMAP |
- +───────────────────────────────────────────────────────────────────────────────────+
- | PHASE 1: STORAGE DECOUPLING |
- | - Replicate HDFS datasets to Google Cloud Storage (gs://) |
- | - Leverage Storage Transfer Service (STS) or Hadoop distcp over Cloud Interconnect|
- | │ |
- | PHASE 2: METADATA EXTERNALIZATION ▼ |
- | - Export Hive Metastore MySQL/Oracle schemas to Dataproc Metastore (DPMS) |
- | - Establish unified central catalog accessible across all future compute runtimes |
- | │ |
- | PHASE 3: LIFT-AND-SHIFT TO MANAGED DATAPROC |
- | - Deploy Dataproc clusters configured with GCS connector and DPMS |
- | - Run existing Spark/Hive scripts with zero code refactoring |
- | │ |
- | PHASE 4: MODERNIZE TO SERVERLESS & EPHEMERAL ARCHITECTURES |
- | - Replace static clusters with Cloud Composer ephemeral DAGs |
- | - Convert standalone PySpark / Spark SQL jobs to Dataproc Serverless |
- | - Transition legacy Hive SQL queries to BigQuery BigLake tables |
- +───────────────────────────────────────────────────────────────────────────────────+
- Step-by-Step Migration Execution
Data Migration with distcp or Storage Transfer Service:
Establish a Dedicated Interconnect or Partner Interconnect connection between the on-premises datacenter and Google Cloud VPC.
Use Apache Hadoop's distributed copy tool ( distcp ) configured with the Cloud Storage connector, or deploy on-premises Storage Transfer Service (STS) agents to mirror petabytes of HDFS data to gs:// buckets with automated data integrity checksums.
Metadata Migration to Dataproc Metastore:
Export relational schema DDL from the on-premises Hive metastore database (MySQL, PostgreSQL, Oracle).
Import the dump into Dataproc Metastore (DPMS) . This ensures that all existing table definitions, partition keys, and SerDe properties are instantly available in GCP.
Compute Migration (Lift-and-Shift):
Spin up Dataproc clusters linked to the DPMS instance.
Update Hadoop configuration files to point default filesystems from hdfs:// to gs://my-lake/ .
Validate that existing Spark JARs, PySpark scripts, and Hive queries produce bit-for-bit identical analytical outputs.
Cloud-Native Modernization:
Decommission persistent clusters. Wrap batch jobs in Cloud Composer DAGs running ephemeral clusters with Spot VM secondary workers.
Migrate stateless Spark batch pipelines directly to Dataproc Serverless to eliminate VM infrastructure management altogether.
6. Architecture Scenarios & Realistic Exam Pitfalls
Problem Scenario Architectural Anti-Pattern Correct Google Cloud Architecture
Dataproc Serverless Network Timeout ; A data engineer submits a PySpark batch to Dataproc Serverless. The job fails immediately during initialization with network timeout errors connecting to Google Cloud Storage. Attempting to assign an external public IP address to the Dataproc Serverless batch job. Enable Private Google Access (PGA) on the VPC subnetwork. Dataproc Serverless instances never possess public IPs; they require PGA to reach Google API endpoints internally.
Cascading Stage Failures on Downscaling ; A managed Dataproc cluster scales down 20 workers during an active Spark job. Downstream stages fail with FetchFailedException , forcing the entire job to restart from stage 0. Disabling autoscaling and running fixed-size clusters 24/7. Configure Graceful Decommissioning ( gracefulDecommissionTimeout = 3600s ) in the autoscaling policy. This prevents nodes from being terminated until running tasks complete and shuffle blocks are safely preserved.
Inter-Executor Communication Blocked ; A serverless Spark batch launches successfully but hangs indefinitely during the first shuffle operation, eventually failing with executor heartbeat timeout errors. Granting broader IAM permissions to the Dataproc service account. Configure a VPC ingress firewall rule allowing internal TCP traffic on all ports ( tcp:0-65535 ) between instances within the target subnetwork. Spark executors must communicate directly with one another during shuffle phases.
Interactive BI Dashboard on Serverless ; A business intelligence team wants to connect Tableau to Dataproc Serverless to run ad-hoc SQL queries throughout the business day. Submitting each ad-hoc SQL query as an independent Dataproc Serverless batch job. Dataproc Serverless does not support long-running interactive daemons. Deploy a managed persistent Dataproc cluster running Spark Thrift Server or HiveServer2, or modernize the reporting layer to BigQuery .