Study guide

15.3 Ensuring Consistent Preprocessing Between Training & Serving

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

15.3 Ensuring Consistent Preprocessing Between Training & Serving

Key Takeaways Explain This Topic AI

Training-serving skew happens when features are computed differently, or from different data, at training time and at serving time.

Embedding preprocessing in the model, through BigQuery ML TRANSFORM, Keras preprocessing layers, or an exported TensorFlow Transform graph, makes serving apply identical logic.

Apache Beam MLTransform can write transformation artifacts during training and read the same artifacts during inference.

A feature store serves the same governed feature definitions for training (offline, point in time) and for online inference.

Logging served feature values and comparing them with training distributions detects skew that code reviews miss.

Last updated: September 2026

The exam guide lists ensuring consistent data preprocessing between training and serving . The consequence of getting it wrong is training-serving skew : the model sees inputs in production that differ systematically from what it learned on, so accuracy drops even though nothing looks broken.

Sources of Training-Serving Skew

Source Example

Duplicated logic Standardization in a SQL training query, re-implemented in Java for serving with a different mean

Different statistics Serving code computes normalization statistics from the current request batch instead of the training set

Vocabulary mismatch Categories encoded with a training vocabulary, while serving builds a new mapping

Different data sources Training uses a cleaned warehouse table, while serving reads raw operational data with different nulls or units

  • Time leakage Training uses end-of-day aggregates, while serving only has partial-day data
  • Feature availability A feature populated in historical data is missing or delayed at request time
  • Library or version differences A different tokenizer version in the serving container
  • Pattern 1: Put Preprocessing Inside the Model

If the model artifact contains the transformations, every consumer gets the same logic.

Tool How

BigQuery ML TRANSFORM Preprocessing and its training statistics are saved with the model and applied by ML.PREDICT . Exported models include supported transformations (Chapter 2)

Keras preprocessing layers Normalization, lookup, and text vectorization layers are part of the saved model

TensorFlow Transform (tf.Transform) Full-pass analyzers compute statistics over training data (on Dataflow at scale) and export a transform_fn graph that is attached to the serving signature

Best when transformations can be expressed in the framework and latency allows running them in the model graph.

Pattern 2: Share Transformation Artifacts

When preprocessing runs outside the model, fit once and reuse the fitted artifacts :

Apache Beam MLTransform writes artifacts (vocabularies, scaling statistics) to a location during training and reads the same artifacts at inference in Dataflow pipelines.

scikit-learn Pipelines (scaler plus model saved together as model.joblib ) served by the prebuilt scikit-learn container.

A custom prediction routine that loads saved scalers or tokenizers in load() and applies them in preprocess() (Chapter 13).

Pattern 3: One Feature Definition for Offline and Online

For aggregated features, such as 30-day spend, compute them once in a governed pipeline and serve them from Feature Store (Chapters 5 and 14):

Training reads point-in-time values (offline serving or ML.FEATURES_AT_TIME ) as of each label's timestamp.

Serving reads the latest synced values from the online store.

No application team re-implements the aggregation.

Pattern 4: Same Code Path for Batch and Streaming

Apache Beam lets the same transformation code backfill historical training data in batch and compute real-time features in streaming on Dataflow. That removes the classic "Spark for training, custom Java service for serving" split.

Pattern 5: Package and Version Together

Put shared preprocessing code in one versioned library imported by the training job and the serving container.

Build training and serving images from the same base image and dependency lock file.

Record preprocessing versions in ML Metadata , and register the model version with them.

Verifying Consistency

Check How

Parity tests Run training-time and serving-time transformations on the same raw records and require identical outputs (within float tolerance)

Golden requests Fixed inputs with known predictions, checked after every deployment

Served-feature logging Log post-preprocessing features for sampled requests (request-response logging to BigQuery)

Skew monitoring Model Monitoring compares serving feature distributions with training data (Chapter 19). BigQuery ML ML.VALIDATE_DATA_SKEW does the same for BigQuery ML models

Gen AI Consistency

Consistency matters for gen AI too:

Prompt templates used in tuning data must match production templates (Chapter 2).

Chunking and embedding models used to index a RAG corpus must match those used to embed queries. Changing the embedding model means re-embedding the corpus.

System instructions and tool definitions should be versioned with evaluation results.

Where Skew Checks Belong in the Lifecycle

  • Stage Consistency control
  • Design Choose one feature definition and one owner per feature
  • Development Shared preprocessing library, parity unit tests
  • Pipeline Transformation artifacts versioned and passed between training and deployment steps
  • Deployment Golden-request tests against the staging endpoint
  • Production Request-response logging plus training-serving skew monitoring with alerts
  • Worked Scenario

A delivery-time model trained on BigQuery data with distance_km computed by a SQL UDF performs well offline. In production, the mobile backend computes distance with a different formula, and MAE doubles.

Fix options, from strongest to weakest:

Move the distance computation into a shared feature pipeline that feeds Feature Store , or into the model's TRANSFORM clause if the model is BigQuery ML, so clients send raw coordinates.

Put the calculation in a CPR preprocess() that uses the same library as training.

As an interim step, add parity tests and skew monitoring on distance_km to catch divergence immediately.