Study guide
5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python
Google Cloud Professional Machine Learning Engineer PMLE
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & the Agent Platform Map
1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map
2 Chapter 2: Low-Code ML with BigQuery ML
2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery
3 Chapter 3: AutoML & Google Cloud AI APIs
3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs
4 Chapter 4: Foundation Models in Model Garden
4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability
5 Chapter 5: Exploring, Preprocessing & Protecting Data
5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows
6 Chapter 6: Notebooks, Prototyping & Experiment Tracking
6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
7 Chapter 7: Evaluating Predictive & Generative AI Solutions
7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
8 Chapter 8: Choosing the Modeling Approach
8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements
9 Chapter 9: Training Data & Training SDKs
9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices
10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning
10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune
11 Chapter 11: Training Hardware & Distributed Training
11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism
12 Chapter 12: Deploying Models for Inference
12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers
13 Chapter 13: Model Versions, Rollouts & Inference Logic
13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing
14 Chapter 14: Scaling Online Serving
14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving
15 Chapter 15: End-to-End ML Pipelines
15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving
16 Chapter 16: Orchestration with Managed Airflow & Ray
16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads
17 Chapter 17: Automating Retraining & CI/CD/CT
17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build
18 Chapter 18: Securing AI Systems, Responsible AI & Explainability
18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform
19 Chapter 19: Monitoring AI Solutions
19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python
Key Takeaways Explain This Topic AI
BigQuery SQL is the lowest-overhead choice for large structured-data transformations when data already lives in BigQuery.
BigQuery DataFrames provides pandas and scikit-learn style APIs that push processing down to BigQuery through SQL.
Dataflow runs Apache Beam pipelines for both batch and streaming preprocessing, and its MLTransform class chains ML data transformations such as scaling, vocabularies, and embeddings.
Managed Service for Apache Spark, formerly Dataproc, is the natural choice for existing Spark or Hadoop preprocessing code, in serverless or cluster deployments.
In-memory Python libraries such as pandas and scikit-learn fit only when the data fits comfortably in a single machine's memory.
Last updated: September 2026
The exam guide lists choosing the right tool for data preprocessing based on scale and complexity , with examples: BigQuery (SQL), Dataflow, Apache Spark, and in-memory Python frameworks. Questions usually give you three signals: data size , batch or streaming , and what code or skills the team already has .
The Options at a Glance
Tool Model Best for Watch out for
BigQuery SQL Serverless SQL engine Structured data already in BigQuery: joins, aggregations, window features, filtering at terabyte scale Complex per-record logic or unstructured data processing
BigQuery ML TRANSFORM / preprocessing functions SQL functions stored with the model Scaling, bucketizing, and encoding that must match at prediction (Chapter 2) Only applies to BigQuery ML models, or exported models that support the transforms
BigQuery DataFrames ( bigframes ) pandas and scikit-learn APIs that compile to BigQuery SQL Python-first teams working on big BigQuery tables without loading data into memory Not every pandas feature is available
Dataflow (Apache Beam) Serverless, autoscaling batch and streaming pipelines Streaming feature computation from Pub/Sub, large unstructured transforms, the same code for batch and streaming, MLTransform , RunInference Beam programming model has a learning curve
Managed Service for Apache Spark (formerly Dataproc) Managed Spark in serverless or cluster deployments Existing PySpark or Spark jobs, Spark MLlib, lift-and-shift from Hadoop Cluster tuning if not serverless
Ray on Agent Platform Managed Ray clusters Distributed Python for data processing and training in one framework Cluster sizing, team Ray skills
In-memory Python (pandas, NumPy, scikit-learn) Single machine Small to medium data, prototyping, complex custom Python logic Memory limits. A 200 GB table won't fit on a notebook VM
Decision Rules
Is the data structured and already in BigQuery? Start with BigQuery SQL , or BigQuery DataFrames if the team prefers pandas syntax. Moving terabytes out of BigQuery to preprocess them elsewhere is a common wrong answer.
Is the data streaming, or must the same logic run on batch and stream? Use Dataflow . Apache Beam lets one pipeline definition handle both, which helps keep features consistent between training (batch backfill) and real-time serving.
Does the organization have a large existing Spark codebase? Use Managed Service for Apache Spark and reuse the code instead of rewriting it.
Is the data small enough for memory and the logic heavily custom? Use in-memory Python in a notebook or a single training job.
Do you need distributed Python across both data processing and training? Consider Ray on Agent Platform (Chapter 16).
Dataflow ML Details Worth Knowing
MLTransform wraps several preprocessing operations in one class: computing vocabularies, scaling, and generating embeddings with Agent Platform or Hugging Face models. It needs the Apache Beam Python SDK 2.53.0 or later and default windowing. Transforms built on TensorFlow Transform (TFT) run in batch pipelines.
Full-pass transforms (such as scaling with the dataset mean and standard deviation) compute statistics once and save artifacts that can be reapplied later. That's how you avoid recomputing different statistics at serving time.
RunInference runs local models (PyTorch, scikit-learn, TensorFlow) or calls a remote Agent Platform endpoint from inside a pipeline, for batch or streaming scoring.
Worked Scenarios
Scenario Best tool Why
3 TB of clickstream in BigQuery, need 30-day rolling aggregates per user for a churn model BigQuery SQL window functions Data is structured and already in BigQuery. Serverless with no data movement
IoT sensor events in Pub/Sub must become features within seconds, and the same logic must backfill history Dataflow Streaming plus batch with one Beam pipeline
2,000 existing PySpark jobs from an on-premises Hadoop cluster Managed Service for Apache Spark Reuse Spark code with managed infrastructure
50 MB CSV of lab results with complex domain-specific cleaning in Python pandas in a notebook Fits in memory, fastest to iterate
Python-savvy analysts want pandas syntax over a 1 TB BigQuery table BigQuery DataFrames pandas API with BigQuery execution
Generate embeddings for 100 million product descriptions as part of an ingestion pipeline Dataflow MLTransform with an embedding model, or BigQuery AI.GENERATE_EMBEDDING if data is in BigQuery Scalable embedding generation near the data
Keeping Preprocessing Consistent
Choosing a tool isn't only about speed. The exam links preprocessing to training-serving skew :
Put transformations where both training and serving can reuse them: BigQuery ML TRANSFORM , saved transformation artifacts, a shared feature pipeline feeding Feature Store, or preprocessing packaged in the serving container (Chapter 13).
Avoid writing the same feature logic twice in two languages, such as SQL for training and Java for serving.
Version preprocessing code alongside the model and record it in pipeline metadata.
Common Wrong Answers
"Export BigQuery to CSV, then preprocess with pandas" for terabyte-scale data. It is slow, fails on memory, and adds a security risk from copied data.
"Rewrite existing Spark jobs in Beam" when the requirement is minimal change. Reuse Spark on the managed Spark service.
"Use scheduled SQL for real-time features" when the scenario needs seconds-level freshness. That calls for a streaming pipeline.
"Run a long-lived self-managed cluster" when a serverless option meets the need. The exam favors managed, Google Cloud-native services when they fit.
Cost and Operations Considerations
Serverless first: BigQuery and Dataflow scale automatically and bill for what you use. Serverless Spark removes cluster management.
Ephemeral clusters: if you run Spark clusters, create them per job and delete them afterward, and consider Spot VMs for fault-tolerant work.
Data locality: process data in the region where it's stored.
Orchestrate preprocessing as a pipeline step (Agent Platform Pipelines components exist for BigQuery, Dataflow, and Managed Spark jobs) so runs are repeatable and tracked.