Study guide

19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift

Key Takeaways Explain This Topic AI

Training-serving skew is a difference between the training data distribution and production serving data, often caused by pipeline or preprocessing differences.

Data drift is a change in production input distributions over time, while concept drift is a change in the relationship between inputs and the target.

Concept drift often can't be seen in input distributions alone; it shows up in model performance against ground truth or in prediction and attribution changes.

Feature attribution drift tracks changes in how much each feature contributes to predictions, and it can reveal problems when input distributions look stable.

L-infinity distance measures the largest difference in any category's share, and Jensen-Shannon divergence measures the overall difference between two distributions.

Last updated: September 2026

The exam guide lists monitoring for common issues (for example, training-serving skew, data drift, concept drift, and feature attribution drift) . Scenario questions describe symptoms, so you need to tell these apart.

Definitions

Issue What changes Compared against Typical cause

Training-serving skew Serving feature distribution differs from training Training data Different preprocessing, different data sources, bugs, stale features

Data (feature) drift Serving feature distribution changes over time An earlier serving window Changing users, seasons, markets, products

Prediction (output) drift Distribution of predictions changes Training predictions or an earlier window Input drift, concept change, upstream issues

Concept drift The relationship between inputs and label changes Ground-truth performance over time Fraud tactics evolve, customer preferences shift, policy changes

Feature attribution drift How much each feature drives predictions changes Baseline attributions (SHAP) Broken or redefined features, interactions changing, concept shift

Skew vs. drift in one sentence

Skew means "production doesn't look like training." Drift means "production today doesn't look like production before."

How Distributions Are Compared

Agent Platform monitoring and BigQuery ML validation use:

  • Metric Applies to Intuition
  • L-infinity distance Categorical features The largest absolute difference in any single category's share

Jensen-Shannon (JS) divergence Numerical (and optionally categorical) features A symmetric, bounded measure of overall distribution difference

Worked L-infinity example: in training, a payment_method feature was 60% card, 30% wallet, and 10% bank transfer. In serving it's 35% card, 55% wallet, and 10% bank. Differences are |0.60 − 0.35| = 0.25, |0.30 − 0.55| = 0.25, and 0. L-infinity is 0.25 , below a default threshold of 0.3 , so there's no alert. If wallet rises to 65% and card falls to 25%, L-infinity is 0.35 and an alert fires.

Threshold tuning: thresholds that are too low cause alert fatigue on harmless variation. Thresholds that are too high miss real shifts. Tune per feature by importance and natural volatility, and use larger time windows for low-traffic models.

Diagnosing by Symptom

Symptom Most likely issue Next step

Right after launch, several features differ sharply from training Training-serving skew Compare preprocessing code and data sources. Check units, defaults, and null handling

Gradual shift in age and device_type over months, accuracy still fine Data drift without harm Keep monitoring. Consider retraining on the next schedule

Inputs stable, but ground-truth precision falling Concept drift Retrain on recent labeled data. Consider new features

Inputs stable, predictions suddenly shift toward one class Upstream bug, model version mix-up, or concept change Check deployments, feature pipelines, and attribution drift

A minor feature suddenly dominates attributions Feature attribution drift from a broken feature (for example, constant or scaled wrongly) Inspect that feature's pipeline, and roll back if needed

Accuracy drops only for one region Segment-specific drift or bias Slice-level monitoring and fairness checks (Chapter 18)

Why Concept Drift Needs Ground Truth

Concept drift changes P(label | features) . Inputs can look identical while their meaning changes. For example, the same transaction pattern that used to be legitimate is now typical of a new fraud ring. Distribution monitors on inputs can miss it entirely. Detect it with:

Continuous evaluation : join predictions with delayed labels and track precision, recall, and calibration over time (Chapter 7).

Prediction drift and attribution drift as indirect signals.

Business KPIs , such as chargeback rate or return rate.

Feature Attribution Drift

Attribution monitoring watches which features the model relies on :

If the most important feature's attribution drops , that feature may have become constant, missing, or wrongly scaled.

If a low-importance ID-like feature's attribution rises , the model may be picking up leakage or a pipeline bug.

Attribution drift can appear when input distributions stay within thresholds , because interactions between features changed.

Model Monitoring v2 tracks attributions with SHAP values . With the Explainable AI deprecation (Chapter 18), plan SHAP-based batch jobs as a long-term way to compute attributions.

Drift Isn't Always Harmful

A drift alert means "look," not "retrain now":

Harmless drift: a feature the model barely uses shifts, or the shift stays within ranges the model handles well.

Harmful drift: important features shift into regions with little training data, or performance on recent labels declines.

Expected drift: known seasonal patterns. Compare against the same period last year with a window and offset baseline.

Combine drift magnitude with feature importance and performance evidence before deciding to retrain.

Prevention vs. Detection

Prevent Detect

Shared preprocessing (TRANSFORM, tf.Transform, Feature Store) → less skew Skew monitoring against training data

  • Point-in-time training data → fewer leakage surprises Attribution drift monitoring
  • Regular retraining policy → less impact from drift Drift monitoring plus continuous evaluation
  • Worked Scenario

A ride-hailing ETA model's MAE jumps 30% overnight. Monitoring shows distance_km skew with L-infinity-style alerts on bucketed distance, while other features are stable. Attribution for distance_km fell sharply.

Diagnosis: training-serving skew. A backend release changed distance from kilometers to meters. Fix: roll back or correct the unit conversion, add a parity test, and add a range check on distance_km in preprocessing.