Study guide

6.3 Model Monitoring: Feature Skew and Drift Detection

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

6.3 Model Monitoring: Feature Skew and Drift Detection

Key Takeaways Explain This Topic AI

Training-serving skew compares live serving requests against the training baseline; prediction drift compares a current serving window against an earlier one.

Jensen-Shannon divergence is used for categorical features and Chebyshev or Kolmogorov-Smirnov distance for numerical features, with alert thresholds typically set around 0.05 to 0.10.

Request-response payload logging must be enabled before monitoring can compare serving traffic to anything.

Concept drift changes the relationship between inputs and outcomes and is invisible to input-distribution monitoring — detecting it needs delayed labels or a business KPI proxy.

Feature attribution drift detects a shift in how much the model relies on each feature, which catches a silently broken upstream feature that distribution thresholds can miss.

Last updated: September 2026

6.3 Model Monitoring: Feature Skew and Drift Detection

Deploying a high-performing machine learning model to a production endpoint is not the conclusion of the machine learning lifecycle—it is the beginning of its operational phase. In real-world environments, machine learning models inevitably degrade over time. Changes in consumer behavior, macroeconomic shifts, upstream data pipeline modifications, sensor degradation, and seasonal variations all cause the statistical distributions of incoming production data to diverge from the distributions on which the model was originally trained.

Traditional software systems fail loudly with stack traces, HTTP 500 error codes, or crash dumps. Machine learning models, however, fail silently : a model experiencing severe distribution shift will continue accepting inputs, executing matrix multiplications, and returning HTTP 200 OK responses with high confidence, even as the real-world business accuracy plummets. Vertex AI Model Monitoring provides a fully managed observability framework designed to detect statistical anomalies in live serving traffic and trigger automated remediation workflows.

1. Feature Skew vs. Prediction Drift: Core Concepts

Vertex AI Model Monitoring distinguishes between two primary categories of distribution shift: Training-Serving Skew (Feature Skew) and Prediction Drift (Covariate Shift). +---------------------------------------------------------------------------------------------------------+

  • | DISTRIBUTION SHIFT TAXONOMY IN VERTEX AI |
  • +------------------------------------+--------------------------------------------------------------------+
  • | TRAINING-SERVING FEATURE SKEW | PREDICTION DRIFT |
  • +------------------------------------+--------------------------------------------------------------------+
  • | * Comparison: Training Baseline vs | * Comparison: Historical Serving Window (t0) vs |
  • | Live Serving Traffic (t_now) | Current Serving Window (t1) |
  • | * Focus: Upstream pipeline errors, | * Focus: Temporal evolution of user behavior, real-world trends, |
  • | data transformations, bias | market shifts over days/weeks/months |
  • | * Baseline: Training dataset in | * Baseline: Rolling sliding window of logged production |
  • | BigQuery or Cloud Storage | requests (e.g., past 48 hours or past 7 days) |
  • | * Timing: Appears immediately upon | * Timing: Manifests gradually over time as the environment |
  • | deployment or traffic onboarding | evolves |
  • +------------------------------------+--------------------------------------------------------------------+
  • 1. Training-Serving Feature Skew

Feature Skew occurs when the distribution of feature values observed during live online serving differs significantly from the distribution of feature values present in the training dataset: P serving ( X ) ≠ P training ( X ) P_{\text{serving}}(X) \neq P_{\text{training}}(X) P serving ​ ( X )  = P training ​ ( X )

Common Root Causes :

Data Pipeline Inconsistencies : The feature engineering code used in the batch training pipeline (e.g., a PySpark job in Dataproc) applies slightly different normalization, null handling, or timestamp parsing logic than the real-time serving application (e.g., a Node.js API server or Python Custom Prediction Routine).

Schema and Unit Mismatches : An upstream client submits monetary amounts in cents rather than dollars, or temperature in Fahrenheit rather than Celsius.

Sampling Bias : The training dataset was collected during a promotional holiday period, whereas the model is deployed to evaluate standard baseline traffic.

Lookahead / Data Leakage : Features available during historical batch training contain information that is unavailable or computed differently at real-time inference time.

2. Prediction Drift & Covariate Shift

Prediction Drift occurs when the statistical properties of incoming features or predicted outputs change over time within the production serving environment itself : P t 1 ( X ) ≠ P t 2 ( X ) or P t 1 ( Y ^ ) ≠ P t 2 ( Y ^ ) P_{t_1}(X) \neq P_{t_2}(X) \quad \text{or} \quad P_{t_1}(\hat{Y}) \neq P_{t_2}(\hat{Y}) P t 1 ​ ​ ( X )  = P t 2 ​ ​ ( X ) or P t 1 ​ ​ ( Y ^ )  = P t 2 ​ ​ ( Y ^ )

Common Root Causes :

Consumer Behavior Shifts : A sudden shift in user preferences, purchasing habits, or economic conditions (e.g., inflation affecting credit risk profiles).

Macroeconomic & Environmental Events : Seasonal weather changes affecting utility demand, or competitor pricing actions altering customer churn dynamics.

Concept Drift : Even if feature distributions P ( X ) P(X) P ( X ) remain constant, the underlying relationship between features and the target ground truth changes: P ( Y ∣ X ) t 1 ≠ P ( Y ∣ X ) t 2 P(Y|X)_{t_1} \neq P(Y|X)_{t_2} P ( Y ∣ X ) t 1 ​ ​  = P ( Y ∣ X ) t 2 ​ ​ . (Note: Detecting pure concept drift requires ground truth label feedback, whereas feature drift can be detected entirely on unlabeled serving requests).

2. Statistical Distance Metrics in Vertex AI

To detect distribution divergence mathematically, Vertex AI Model Monitoring executes continuous statistical tests across each feature column. It selects distance metrics based on the feature's data type (categorical vs. numerical). +---------------------------------------------------------------------------------------------------------+

  • | STATISTICAL DISTANCE METRIC SELECTION |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | FEATURE TYPE | DISTANCE METRIC | CALCULATION MECHANISM |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | Categorical / Discrete Features | Jensen-Shannon Divergence (JSD) | Symmetric, smoothed relative |
  • | (e.g., device_type, region_code) | or L-infinity Norm of PMFs | entropy bounded in [0, 1] |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | Numerical / Continuous Features | Chebyshev Distance (L-inf on CDF) | Maximum vertical distance |
  • | (e.g., account_balance, age, temp) | or Kolmogorov-Smirnov (K-S) Test | between empirical CDF curves |
  • +------------------------------------+------------------------------------+-------------------------------+
  • Categorical Features: Jensen-Shannon Divergence (JSD)

For discrete and categorical variables, Vertex AI calculates the Jensen-Shannon Divergence (JSD) . JSD is a symmetric and smoothed version of the Kullback-Leibler (KL) Divergence. Given two probability mass functions P P P (baseline distribution) and Q Q Q (serving distribution), JSD is defined as: M = 1 2 ( P + Q ) M = \frac{1}{2}(P + Q) M = 2 1 ​ ( P + Q ) D J S ( P ∥ Q ) = 1 2 D K L ( P ∥ M ) + 1 2 D K L ( Q ∥ M ) D_{JS}(P \parallel Q) = \frac{1}{2} D_{KL}(P \parallel M) + \frac{1}{2} D_{KL}(Q \parallel M) D J S ​ ( P ∥ Q ) = 2 1 ​ D K L ​ ( P ∥ M ) + 2 1 ​ D K L ​ ( Q ∥ M )

where D K L ( P ∥ M ) = ∑ x P ( x ) log ⁡ ( P ( x ) M ( x ) ) D_{KL}(P \parallel M) = \sum_x P(x) \log\left(\frac{P(x)}{M(x)}\right) D K L ​ ( P ∥ M ) = ∑ x ​ P ( x ) lo g ( M ( x ) P ( x ) ​ ) .

Key Properties for ML Engineers :

JSD is strictly bounded between 0.0 0.0 0.0 (identical distributions) and 1.0 1.0 1.0 (disjoint distributions) when using base-2 logarithm.

Unlike standard KL divergence, JSD is symmetric ( D J S ( P ∥ Q ) = D J S ( Q ∥ P ) D_{JS}(P \parallel Q) = D_{JS}(Q \parallel P) D J S ​ ( P ∥ Q ) = D J S ​ ( Q ∥ P ) ) and handles zero-probability categories without producing infinite values ( + ∞ +\infty + ∞ ).

Alerting Threshold : Typical production anomaly thresholds are set between 0.05 and 0.10 . A threshold of 0.1 indicates moderate divergence, while 0.02 is ultra-sensitive (and prone to false alarms).

Numerical Features: Chebyshev Distance & Kolmogorov-Smirnov (K-S) Distance

For continuous numerical features, Vertex AI compares the empirical cumulative distribution functions (eCDFs) of the baseline dataset F 0 ( x ) F_0(x) F 0 ​ ( x ) and the target serving dataset F t ( x ) F_t(x) F t ​ ( x ) :

Chebyshev Distance ( L ∞ L_\infty L ∞ ​ Distance) : Computes the maximum absolute difference between the probability densities or quantile histograms across all bins: D Chebyshev ( P , Q ) = max ⁡ i ∣ P i − Q i ∣ D_{\text{Chebyshev}}(P, Q) = \max_i |P_i - Q_i| D Chebyshev ​ ( P , Q ) = i max ​ ∣ P i ​ − Q i ​ ∣

Kolmogorov-Smirnov (K-S) Test Statistic : Measures the maximum vertical distance between two cumulative distribution functions: D K S = sup ⁡ x ∣ F baseline ( x ) − F serving ( x ) ∣ D_{KS} = \sup_x |F_{\text{baseline}}(x) - F_{\text{serving}}(x)| D K S ​ = x sup ​ ∣ F baseline ​ ( x ) − F serving ​ ( x ) ∣

If the calculated D D D statistic exceeds the user-configured alert threshold (e.g., 0.05 ), Vertex AI flags the feature as experiencing anomalous drift.

3. Configuring Vertex AI Model Monitoring

Setting up Model Monitoring on a deployed Vertex AI Endpoint involves four foundational configuration components: Payload Logging , Baseline Specification , Objective Configuration , and Alerting Channels . +---------------------------------------------------------------------------------------------------------+

  • | VERTEX AI MODEL MONITORING ARCHITECTURE |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | [ Client Requests ] =====> [ Vertex AI Online Endpoint ] =====> [ Predictions Returned ] |
  • | | |
  • | (Async Payload Logging) |
  • | v |
  • | [ BigQuery Logging Sink ] |
  • | (Request & Response Tables) |
  • | | |
  • | v |
  • | +---------------------------------+ |
  • | | Vertex AI Model Monitoring Job | <===== [ Training Baseline Dataset ] |
  • | | (Scheduled Cron / Hourly Eval) | (BigQuery Table or GCS Data) |
  • | +---------------------------------+ |
  • | | |
  • | (If Distance Metric > Threshold) |
  • | v |
  • | +---------------------------------------------------+ |
  • | | Cloud Monitoring / Cloud Logging / Pub/Sub Alert | |
  • | +---------------------------------------------------+ |
  • | | |
  • | v |
  • | +---------------------------------------------------+ |
  • | | Automated Retraining (Vertex AI Pipelines / KFP) | |
  • | +---------------------------------------------------+ |
  • +---------------------------------------------------------------------------------------------------------+
  • Step 1: Enable Request-Response Payload Logging

Before Model Monitoring can analyze incoming inference traffic, the endpoint must log raw feature payloads to BigQuery. Engineers configure the sampling rate (from 0.01 to 1.0 ): # Enabling request-response logging via Vertex AI SDK

from google.cloud import aiplatform

endpoint = aiplatform.Endpoint("projects/my-project/locations/us-central1/endpoints/1234567890")

  • # Configure 20% random sampling of live requests to BigQuery
  • endpoint.set_request_response_logging_sampling_rate(0.20)

Note

Sampling Rate Trade-off : A 100% sampling rate ( 1.0 ) provides exact statistical precision but increases BigQuery storage and streaming ingestion costs under high QPS loads. For high-volume endpoints (>1,000 QPS), sampling rates between 5% and 20% provide robust statistical power for hypothesis testing while drastically reducing operational logging costs.

Step 2: Establish the Baseline Dataset

For Feature Skew Detection : Specify the original training dataset URI (either a BigQuery table bq://project.dataset.training_table or Cloud Storage CSV/TFRecord path gs://my-bucket/train.csv ) along with the target feature schema.

For Prediction Drift Detection : Vertex AI automatically uses the previous sliding window of serving requests logged in BigQuery (e.g., comparing the last 24 hours against the previous 7 days) as the dynamic historical baseline.

Step 3: Instantiate the Model Deployment Monitoring Job

Using the Vertex AI Python SDK, ML engineers define feature-specific alerting thresholds and monitoring schedules: from google.cloud import aiplatform

from google.cloud.aiplatform import model_monitoring

  • # Define alerting thresholds for specific features
  • skew_thresholds = {
  • "income": model_monitoring.ThresholdConfig(value=0.05), # Numerical (Chebyshev/KS)
  • "occupation": model_monitoring.ThresholdConfig(value=0.08), # Categorical (JSD)
  • "credit_score": model_monitoring.ThresholdConfig(value=0.05) # Numerical
  • }
  • drift_thresholds = {
  • "income": model_monitoring.ThresholdConfig(value=0.03),
  • "occupation": model_monitoring.ThresholdConfig(value=0.05)
  • }
  • # Configure Skew and Drift objectives
  • skew_config = model_monitoring.SkewDetectionConfig(
  • data_source="bq://my-project.ml_datasets.training_data_baseline",
  • skew_thresholds=skew_thresholds,
  • attribute_skew_thresholds=skew_thresholds
  • )
  • drift_config = model_monitoring.DriftDetectionConfig(
  • drift_thresholds=drift_thresholds,
  • attribute_drift_thresholds=drift_thresholds
  • )
  • objective_config = model_monitoring.ObjectiveConfig(
  • skew_detection_config=skew_config,
  • drift_detection_config=drift_config,
  • explanation_config=None
  • )
  • # Configure alerting channel (Email and Cloud Monitoring / Pub/Sub)
  • alert_config = model_monitoring.EmailAlertConfig(
  • user_emails=["mlops-oncall@enterprise.com"],
  • enable_logging=True
  • )
  • # Schedule hourly monitoring job with a 24-hour lookback window
  • monitoring_job = aiplatform.ModelDeploymentMonitoringJob.create(
  • display_name="credit_risk_model_monitor",
  • endpoint=endpoint,
  • model_deployment_monitoring_objective_configs=objective_config,
  • logging_sampling_strategy=model_monitoring.RandomSampleConfig(sample_rate=0.2),
  • model_deployment_monitoring_schedule_config=model_monitoring.ScheduleConfig(monitor_interval=1), # 1 hour
  • model_deployment_monitoring_alert_config=alert_config
  • )
  • 4. Closed-Loop Automated Retraining Architecture

Detecting drift is only half the battle; enterprise MLOps architectures must act upon detection alerts automatically to close the operational loop. +---------------------------------------------------------------------------------------------------------+

  • | CLOSED-LOOP RETRAINING WORKFLOW |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | 1. Drift / Skew Detected (Vertex AI Model Monitoring) |
  • | | |
  • | v (Threshold Breached) |
  • | 2. Alert Dispatched to Cloud Monitoring & Cloud Pub/Sub Topic (`projects/p/topics/model-alerts`) |
  • | | |
  • | v (Push Subscription / Eventarc) |
  • | 3. Cloud Function / Cloud Run Invoked |
  • | | |
  • | v (Executes Vertex AI SDK) |
  • | 4. Vertex AI Pipeline Triggered (KFP / TFX) |
  • | - Data Extraction: Ingest recent 30-day production data + ground truth labels |
  • | - Data Validation: TFDV schema and anomaly verification |
  • | - Custom Training: Train candidate model v_new on GPU worker pool |
  • | - Model Evaluation: Compare candidate vs production champion on holdout test set |
  • | | |
  • | v (Evaluation Quality Gate Passed) |
  • | 5. Model Registered in Vertex Model Registry (@canary alias) |
  • | | |
  • | v |
  • | 6. Canary Deployment on Endpoint (10% Traffic Split -> Automatic Ramp to 100%) |
  • +---------------------------------------------------------------------------------------------------------+

Event Dispatch : When a statistical distance threshold is exceeded, Vertex AI Model Monitoring writes an anomaly log to Cloud Logging and publishes an event to a dedicated Cloud Pub/Sub topic.

Orchestration Trigger : A serverless Cloud Function (or Cloud Run service via Eventarc) subscribes to the Pub/Sub topic, parses the JSON anomaly payload (identifying the specific drifted feature and endpoint ID), and initiates a Vertex AI Pipeline run.

Data Refresh & Ingestion : The pipeline queries BigQuery to extract the most recent window of labeled data, combining historical training data with newly curated records.

Automated Evaluation Gate : The pipeline trains a new candidate model and runs ML.EVALUATE or custom evaluation components. If the candidate model's ROC-AUC / F1-score surpasses the current production model by a configured margin, the pipeline uploads the artifact to Vertex AI Model Registry and initiates a canary rollout on the serving endpoint.

4b. The Four Signals the Blueprint Names

The exam guide lists four monitoring targets by name — training-serving skew, data drift, concept drift , and feature attribution drift — and they are genuinely four different things with four different remedies.

Signal What changed How it is detected Remedy

Training-serving skew Serving inputs differ from the training data Live requests compared against the training baseline Fix the preprocessing path, or retrain on current data

Data drift (covariate shift) P(X) moved; inputs today differ from inputs last month Current serving window compared against an earlier serving window Retrain on recent data

Concept drift P(Y|X) moved; the same inputs now imply a different outcome Input distributions can look completely stable. Requires outcome labels or a proxy: measured accuracy against delayed ground truth, or a business KPI Retrain; the old relationship no longer holds

Feature attribution drift The model's reliance on features shifted, even where input distributions did not Compare current feature attribution magnitudes against the baseline attribution profile Investigate — often an upstream feature broke and the model silently re-weighted onto substitutes

Two consequences worth carrying into the exam.

Concept drift is invisible to input monitoring. A fraud model whose feature distributions are unchanged can still collapse because fraudsters changed tactics: the same inputs now mean something different. No distributional distance metric on X will see it. Detecting it requires labels — which arrive late, if at all — or a downstream business signal such as chargeback rate. When a scenario says "the input distributions look normal but accuracy has fallen," concept drift is the answer.

Feature attribution drift catches what distribution monitoring misses. Suppose an upstream join breaks and a strong feature silently becomes constant. Its own distribution shifts, but if the alerting threshold is tuned loosely, the change can pass. Attribution monitoring shows the model's dependence collapsing on that feature and rising on its correlates, which is a much louder signal and points directly at the cause. It also flags the inverse case: a feature whose distribution is unchanged but whose influence on predictions has grown, which usually means the rest of the input space moved around it.

5. Summary Comparison: Skew vs. Drift & Distance Metrics

Attribute / Dimension Training-Serving Feature Skew Prediction Drift / Covariate Shift

Comparison Target Live serving requests vs. Training baseline dataset Current serving window vs. Historical serving window

Baseline Source BigQuery table or GCS CSV/TFRecord used during training BigQuery request-response payload logging table

Primary Detection Objective Detect pipeline bugs, missing values, preprocessing errors Detect real-world trend shifts, seasonal patterns

Categorical Distance Metric Jensen-Shannon Divergence (JSD) / L ∞ L_\infty L ∞ ​ on PMF Jensen-Shannon Divergence (JSD) / L ∞ L_\infty L ∞ ​ on PMF

Numerical Distance Metric Chebyshev Distance / Kolmogorov-Smirnov (K-S) Chebyshev Distance / Kolmogorov-Smirnov (K-S)

Default Alert Threshold 0.05 – 0.10 0.05 – 0.10

Remediation Action Fix feature transform pipelines or retrain with new data Trigger automated retraining pipeline with recent data

Neither column detects concept drift , because both compare distributions of inputs and predictions rather than the relationship between them, and neither detects feature attribution drift , which requires comparing attribution profiles rather than input distributions. Both are covered in section 4b above.

Loading diagram... Vertex AI Model Monitoring Architecture and Closed-Loop Retraining Flow