Study guide
15.1 Validating Data & Models in Pipelines
Google Cloud Professional Machine Learning Engineer PMLE
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & the Agent Platform Map
1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map
2 Chapter 2: Low-Code ML with BigQuery ML
2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery
3 Chapter 3: AutoML & Google Cloud AI APIs
3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs
4 Chapter 4: Foundation Models in Model Garden
4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability
5 Chapter 5: Exploring, Preprocessing & Protecting Data
5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows
6 Chapter 6: Notebooks, Prototyping & Experiment Tracking
6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
7 Chapter 7: Evaluating Predictive & Generative AI Solutions
7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
8 Chapter 8: Choosing the Modeling Approach
8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements
9 Chapter 9: Training Data & Training SDKs
9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices
10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning
10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune
11 Chapter 11: Training Hardware & Distributed Training
11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism
12 Chapter 12: Deploying Models for Inference
12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers
13 Chapter 13: Model Versions, Rollouts & Inference Logic
13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing
14 Chapter 14: Scaling Online Serving
14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving
15 Chapter 15: End-to-End ML Pipelines
15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving
16 Chapter 16: Orchestration with Managed Airflow & Ray
16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads
17 Chapter 17: Automating Retraining & CI/CD/CT
17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build
18 Chapter 18: Securing AI Systems, Responsible AI & Explainability
18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform
19 Chapter 19: Monitoring AI Solutions
19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
15.1 Validating Data & Models in Pipelines
Key Takeaways Explain This Topic AI
BigQuery ML's ML.VALIDATE_DATA_SKEW compares serving data with statistics saved at training, and ML.VALIDATE_DATA_DRIFT compares two sets of serving data.
BigQuery ML data validation uses L-infinity distance by default for categorical features, and both categorical and numerical thresholds default to 0.3.
TFX pipelines validate data with StatisticsGen, SchemaGen, and ExampleValidator, and they gate models with Evaluator blessings and InfraValidator checks.
Google Cloud Pipeline Components include model evaluation components for classification, regression, and forecasting that can feed conditional deployment steps.
A validation gate should compare the candidate model with the current production model on the same held-out data before any deployment step runs.
Last updated: September 2026
The first consideration in the exam guide's pipeline section is validating data and models . In an automated pipeline, no human looks at every run, so validation steps decide whether training proceeds and whether a model deploys.
Why Pipelines Need Validation Gates
Failure Without a gate With a gate
Upstream schema change (a column renamed or a type changed) Training crashes or silently uses nulls Pipeline stops at data validation with a clear error
Distribution shift in new data The model trains on corrupted or unrepresentative data Drift check alerts. The run pauses for review
New model worse than production A worse model deploys automatically Evaluation gate blocks deployment
Model artifact can't load in the serving container Deployment fails in production Infrastructure validation catches it first
- Data Validation
- What to check
Schema: expected columns, types, required fields, allowed categorical values.
Statistics: row counts, missing-value rates, min, max, mean, and distinct counts within expected ranges.
Skew: do new training or serving data differ from the data the current model was trained on?
Drift: did this period's data change significantly from the previous period?
Freshness and completeness: did today's partition arrive, and is it complete?
BigQuery ML validation functions
- Function Purpose
- ML.DESCRIBE_DATA Descriptive statistics for training or serving data
ML.VALIDATE_DATA_SKEW Compares serving data with the training statistics saved when the BigQuery ML model was created , so the original training data isn't needed
- ML.VALIDATE_DATA_DRIFT Compares two sets of serving data , such as yesterday vs. today
- ML.TFDV_DESCRIBE Fine-grained statistics equivalent to TensorFlow Data Validation's statistics generation
- ML.TFDV_VALIDATE Compares statistics sets to find anomalies, like TFDV validate_statistics
Thresholds: categorical features use L-infinity distance by default (Jensen-Shannon divergence is an option), numerical features use Jensen-Shannon divergence , and both default thresholds are 0.3 . You can set per-feature thresholds. SELECT * FROM ML.VALIDATE_DATA_SKEW(
- MODEL `risk.credit_model`,
- TABLE `risk.applications_2026_09_16`,
- STRUCT(0.2 AS categorical_default_threshold,
- 0.2 AS numerical_default_threshold));
Run a query like this as a pipeline step ( BigqueryQueryJobOp ). If it reports anomalies, a condition stops the run or sends a notification.
TensorFlow Data Validation in TFX pipelines
Agent Platform Pipelines runs TFX pipelines as well as KFP pipelines. Google suggests TFX for workflows that process terabytes of structured or text data with TensorFlow. The TFX data validation chain:
ExampleGen ingests data.
StatisticsGen computes statistics.
SchemaGen infers a schema, which you curate and version.
ExampleValidator flags anomalies such as missing features, unexpected values, and skew or drift against the schema and previous statistics.
Other validation options
Knowledge Catalog data quality scans on BigQuery tables for rule-based checks.
Custom Python components (for example, pandas or Great Expectations style assertions) for domain rules such as "claim_amount must be positive."
- Model Validation
- Evaluation gates
Evaluate the candidate on a held-out test set with the pipeline evaluation components ( ModelEvaluationClassificationOp , ModelEvaluationRegressionOp , ModelEvaluationForecastingOp ), usually after ModelBatchPredictOp .
Compare with the champion on the same data. Absolute thresholds alone can pass a model that's worse than production.
Check slices such as region and demographic groups, so an average gain doesn't hide a regression for one group. Preview GCPC components can detect data and model bias (Chapter 18).
Gate deployment with a pipeline condition: # A small custom component reads the evaluation artifact and returns True or False
- gate = compare_to_champion(candidate_metrics=eval_task.outputs["evaluation_metrics"],
- champion_auc_pr=champion_auc_pr, min_gain=0.005)
with dsl.If(gate.output == True):
ModelDeployOp(model=upload_task.outputs["model"], endpoint=endpoint, ...)
In TFX, the Evaluator computes metrics, often per slice, and blesses the model only when thresholds and comparison with the baseline pass. The Pusher deploys only blessed models.
Infrastructure validation
A model can be accurate but unservable: wrong artifact name, missing dependency, too slow to load. Infrastructure validation loads the model in a serving-like environment and sends test requests before deployment. Examples include TFX's InfraValidator, the AutoML Tabular workflow's InfraValidatorOp , or a staging endpoint smoke test.
Gen AI validation gates
For prompt or model changes in gen AI pipelines, run Gen AI evals (Chapter 7) against a fixed evaluation set, and gate on rubric pass rates, grounding, and safety scores.
Choosing What Happens on Failure
Not every anomaly should stop a pipeline. Decide per check:
- Check result Typical policy
- Schema broken (missing column, wrong type) Hard stop. Training on broken data is never useful
- Row count far below normal Hard stop. A partial load is likely
- Moderate drift on a few features Soft gate: continue training but require human approval before deployment
- Candidate slightly worse than champion Register the version without the champion alias, and don't deploy
- Fairness slice regression Block deployment and escalate to the responsible AI review
Record every gate decision as pipeline metadata so audits can show why a model was or wasn't deployed.
Putting It Together
- Stage Gate Action on failure
- After ingestion Schema and statistics validation Stop the run, notify data owners
- Before training Skew and drift vs. training baseline Stop, or continue with an alert, depending on policy
- After training Evaluation thresholds + champion comparison + slices Don't upload as default and don't deploy
- Before deployment Infrastructure validation Don't deploy
- After deployment Canary guardrails (Chapter 13) Roll back traffic