Study guide
15.3 Ensuring Consistent Preprocessing Between Training & Serving
Google Cloud Professional Machine Learning Engineer PMLE
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & the Agent Platform Map
1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map
2 Chapter 2: Low-Code ML with BigQuery ML
2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery
3 Chapter 3: AutoML & Google Cloud AI APIs
3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs
4 Chapter 4: Foundation Models in Model Garden
4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability
5 Chapter 5: Exploring, Preprocessing & Protecting Data
5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows
6 Chapter 6: Notebooks, Prototyping & Experiment Tracking
6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
7 Chapter 7: Evaluating Predictive & Generative AI Solutions
7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
8 Chapter 8: Choosing the Modeling Approach
8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements
9 Chapter 9: Training Data & Training SDKs
9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices
10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning
10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune
11 Chapter 11: Training Hardware & Distributed Training
11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism
12 Chapter 12: Deploying Models for Inference
12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers
13 Chapter 13: Model Versions, Rollouts & Inference Logic
13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing
14 Chapter 14: Scaling Online Serving
14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving
15 Chapter 15: End-to-End ML Pipelines
15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving
16 Chapter 16: Orchestration with Managed Airflow & Ray
16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads
17 Chapter 17: Automating Retraining & CI/CD/CT
17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build
18 Chapter 18: Securing AI Systems, Responsible AI & Explainability
18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform
19 Chapter 19: Monitoring AI Solutions
19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
15.3 Ensuring Consistent Preprocessing Between Training & Serving
Key Takeaways Explain This Topic AI
Training-serving skew happens when features are computed differently, or from different data, at training time and at serving time.
Embedding preprocessing in the model, through BigQuery ML TRANSFORM, Keras preprocessing layers, or an exported TensorFlow Transform graph, makes serving apply identical logic.
Apache Beam MLTransform can write transformation artifacts during training and read the same artifacts during inference.
A feature store serves the same governed feature definitions for training (offline, point in time) and for online inference.
Logging served feature values and comparing them with training distributions detects skew that code reviews miss.
Last updated: September 2026
The exam guide lists ensuring consistent data preprocessing between training and serving . The consequence of getting it wrong is training-serving skew : the model sees inputs in production that differ systematically from what it learned on, so accuracy drops even though nothing looks broken.
Sources of Training-Serving Skew
Source Example
Duplicated logic Standardization in a SQL training query, re-implemented in Java for serving with a different mean
Different statistics Serving code computes normalization statistics from the current request batch instead of the training set
Vocabulary mismatch Categories encoded with a training vocabulary, while serving builds a new mapping
Different data sources Training uses a cleaned warehouse table, while serving reads raw operational data with different nulls or units
- Time leakage Training uses end-of-day aggregates, while serving only has partial-day data
- Feature availability A feature populated in historical data is missing or delayed at request time
- Library or version differences A different tokenizer version in the serving container
- Pattern 1: Put Preprocessing Inside the Model
If the model artifact contains the transformations, every consumer gets the same logic.
Tool How
BigQuery ML TRANSFORM Preprocessing and its training statistics are saved with the model and applied by ML.PREDICT . Exported models include supported transformations (Chapter 2)
Keras preprocessing layers Normalization, lookup, and text vectorization layers are part of the saved model
TensorFlow Transform (tf.Transform) Full-pass analyzers compute statistics over training data (on Dataflow at scale) and export a transform_fn graph that is attached to the serving signature
Best when transformations can be expressed in the framework and latency allows running them in the model graph.
Pattern 2: Share Transformation Artifacts
When preprocessing runs outside the model, fit once and reuse the fitted artifacts :
Apache Beam MLTransform writes artifacts (vocabularies, scaling statistics) to a location during training and reads the same artifacts at inference in Dataflow pipelines.
scikit-learn Pipelines (scaler plus model saved together as model.joblib ) served by the prebuilt scikit-learn container.
A custom prediction routine that loads saved scalers or tokenizers in load() and applies them in preprocess() (Chapter 13).
Pattern 3: One Feature Definition for Offline and Online
For aggregated features, such as 30-day spend, compute them once in a governed pipeline and serve them from Feature Store (Chapters 5 and 14):
Training reads point-in-time values (offline serving or ML.FEATURES_AT_TIME ) as of each label's timestamp.
Serving reads the latest synced values from the online store.
No application team re-implements the aggregation.
Pattern 4: Same Code Path for Batch and Streaming
Apache Beam lets the same transformation code backfill historical training data in batch and compute real-time features in streaming on Dataflow. That removes the classic "Spark for training, custom Java service for serving" split.
Pattern 5: Package and Version Together
Put shared preprocessing code in one versioned library imported by the training job and the serving container.
Build training and serving images from the same base image and dependency lock file.
Record preprocessing versions in ML Metadata , and register the model version with them.
Verifying Consistency
Check How
Parity tests Run training-time and serving-time transformations on the same raw records and require identical outputs (within float tolerance)
Golden requests Fixed inputs with known predictions, checked after every deployment
Served-feature logging Log post-preprocessing features for sampled requests (request-response logging to BigQuery)
Skew monitoring Model Monitoring compares serving feature distributions with training data (Chapter 19). BigQuery ML ML.VALIDATE_DATA_SKEW does the same for BigQuery ML models
Gen AI Consistency
Consistency matters for gen AI too:
Prompt templates used in tuning data must match production templates (Chapter 2).
Chunking and embedding models used to index a RAG corpus must match those used to embed queries. Changing the embedding model means re-embedding the corpus.
System instructions and tool definitions should be versioned with evaluation results.
Where Skew Checks Belong in the Lifecycle
- Stage Consistency control
- Design Choose one feature definition and one owner per feature
- Development Shared preprocessing library, parity unit tests
- Pipeline Transformation artifacts versioned and passed between training and deployment steps
- Deployment Golden-request tests against the staging endpoint
- Production Request-response logging plus training-serving skew monitoring with alerts
- Worked Scenario
A delivery-time model trained on BigQuery data with distance_km computed by a SQL UDF performs well offline. In production, the mobile backend computes distance with a different formula, and MAE doubles.
Fix options, from strongest to weakest:
Move the distance computation into a shared feature pipeline that feeds Feature Store , or into the model's TRANSFORM clause if the model is BigQuery ML, so clients send raw coordinates.
Put the calculation in a CPR preprocess() that uses the same library as training.
As an interim step, add parity tests and skew monitoring on distance_km to catch divergence immediately.