Study guide
5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
Google Cloud Professional Machine Learning Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap
2 Section 1: Architecting Low-Code AI Solutions (~13%)
1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability
3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)
2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance
4 Section 3: Scaling Prototypes into ML Models (~21%)
3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization
5 Section 4: Serving and Scaling Models (~20%)
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput
6 Section 5: Automating and Orchestrating ML Pipelines (~18%)
5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
7 Section 6: Monitoring AI Solutions (~13%)
6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production
5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
Key Takeaways Explain This Topic AI
MLOps Maturity Level 0 relies on manual, script-driven processes; Level 1 introduces automated Continuous Training (CT) pipelines; Level 2 achieves full CI/CD automation for pipeline code, testing, and continuous model delivery.
CI/CD pipelines on Google Cloud leverage GitHub / Cloud Source Repositories with Cloud Build triggers to automate unit testing, container compilation, pipeline YAML generation, and staging deployment.
Continuous Training (CT) architectures can be driven by time-based triggers (Cloud Scheduler), event-based triggers (Cloud Storage / Eventarc / Cloud Functions), or data/model drift alerts (Vertex AI Model Monitoring / Pub/Sub).
Champion-Challenger validation gates compare candidate models against active production models on identical evaluation datasets, ensuring automated deployment occurs only when candidates exceed baseline quality and latency thresholds.
Training-serving skew is prevented by sharing transformation logic between training and inference using tf.transform, Apache Beam pipelines, BigQuery ML TRANSFORM clauses, or Vertex AI Feature Store.
Last updated: September 2026
5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
Operationalizing machine learning requires moving beyond one-off model development to establishing automated, continuous engineering lifecycles. Traditional DevOps focuses on Continuous Integration (CI) and Continuous Delivery (CD) of software code. MLOps extends these principles to incorporate the unique dynamics of machine learning: continuous integration of data and models, Continuous Training (CT) of ML pipelines, and continuous monitoring of production inference streams.
Achieving end-to-end MLOps on Google Cloud requires integrating developer tools (Cloud Source Repositories, GitHub, Cloud Build), orchestration engines (Vertex AI Pipelines), event routing services (Eventarc, Pub/Sub, Cloud Functions, Cloud Scheduler), and governance registries (Vertex AI Model Registry).
1. MLOps Maturity Levels (Google Cloud MLOps Framework)
Google Cloud categorizes machine learning operational maturity into three distinct tiers: +---------------------------------------------------------------------------------------------------------+
- | MLOps MATURITY LEVEL TAXONOMY |
- +------------------------------------+------------------------------------+-------------------------------+
- | LEVEL 0: MANUAL PROCESS | LEVEL 1: CONTINUOUS TRAINING (CT) | LEVEL 2: FULL CI/CD AUTOMATION|
- +------------------------------------+------------------------------------+-------------------------------+
- | * Ad-hoc Jupyter notebooks | * Automated pipeline orchestration | * Automated CI/CD for pipeline|
- | * Manual data extraction & prep | * Continuous Training (CT) on data | code, components, & tests |
- | * Manual model training & tuning | * Model validation gating | * Automated testing & staging |
- | * Disconnected training & serving | * Vertex ML Metadata tracking | * Automated deployment of new |
- | * No versioning or audit trail | * Triggered by time, data, drift | pipeline definitions to prod|
- | * Rapid training-serving skew | * Pipeline code updated manually | * Zero-touch continuous MLOps |
- +------------------------------------+------------------------------------+-------------------------------+
- Level 0: Manual Process
Characteristics : Data scientists explore data and train models locally in standalone notebooks. Code is rarely modularized; weights are manually copied to serving servers; feature engineering code is manually re-implemented in production serving applications.
Pain Points : Frequent model degradation, severe training-serving skew, lack of reproducibility, and inability to scale to multiple models.
Level 1: ML Pipeline Automation (Continuous Training / CT)
Characteristics : The model training process is orchestrated as an automated DAG (using Vertex AI Pipelines with KFP or TFX). When new data arrives or performance drops, the pipeline automatically executes data extraction, validation, training, and evaluation.
Gaps : While pipeline execution is automated, the pipeline code itself is authored and deployed manually. Pipeline updates require manual intervention.
Level 2: CI/CD Pipeline Automation (Full MLOps)
Characteristics : Full automation of both pipeline execution and pipeline code delivery. When engineers push new code to a Git repository, automated CI/CD pipelines (Cloud Build) test components, build containers, compile pipeline YAMLs, test pipelines in staging, and deploy pipeline definitions to production.
2. CI/CD Architecture with Cloud Build and Vertex AI Pipelines
A production-grade Level 2 MLOps pipeline on Google Cloud implements an automated multi-stage Cloud Build workflow: +---------------------------------------------------------------------------------------------------------+
- | LEVEL 2 CI/CD/CT ARCHITECTURE ON GOOGLE CLOUD |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | [ Developer Git Push ] (GitHub / Cloud Source Repositories) |
- | | |
- | v |
- | [ Cloud Build CI Trigger ] |
- | 1. Unit Tests & Linting (pytest, flake8) |
- | 2. Component Container Builds (Docker build -> Google Artifact Registry) |
- | 3. Pipeline Compilation (kfp.compiler -> pipeline_spec.json) |
- | 4. Staging Integration Test (Executes pipeline in test Vertex AI environment) |
- | | |
- | v (Promote to Production) |
- | [ Production Pipeline Definition Storage ] (Cloud Storage Pipeline Template Registry) |
- | ^ |
- | | Triggers Pipeline Execution (CT) |
- | +---------+-----------------------------------------+ |
- | | | |
- | [ Cloud Scheduler ] [ Eventarc / GCS Upload ] [ Model Monitoring Alert / PubSub ] |
- | (Time-based: Cron) (Event-based: New Data) (Drift-based: Performance Drop) |
- | | |
- | v |
- | [ Vertex AI Pipelines Execution ] |
- | - Extract -> Validate -> Train |
- | - Evaluate vs. Champion Model |
- | | |
- | v |
- | [ Candidate Model Blessed? ] |
- | / \ |
- | (YES) (NO) |
- | / \ |
- | v v |
- | [ Vertex AI Model Registry ] [ Alert ML Team ] |
- | - Set Alias: @champion - Pipeline Halts |
- | - Deploy to Endpoint / Canary |
- +---------------------------------------------------------------------------------------------------------+
- Cloud Build CI/CD Workflow Steps
Continuous Integration (CI) :
Step 1: Unit & Quality Tests : Cloud Build runs pytest on all custom component code, data transformation functions, and schema validation rules.
Step 2: Container Image Builds : Cloud Build builds updated Docker container images for custom components and pushes them with immutable SHA digests to Google Artifact Registry .
Step 3: Pipeline Compilation : Executes a compilation script to verify DAG topology and output a compiled pipeline_spec.yaml .
Step 4: Staging Smoke Test : Submits a test run of the pipeline in a staging Google Cloud project using a synthetic data sample to verify end-to-end execution without errors.
Continuous Delivery (CD) :
Uploads the validated pipeline template to a centralized Cloud Storage bucket (or Vertex AI Pipeline Template Registry) accessible by production trigger mechanisms.
3. Automated Retraining Trigger Architectures (Continuous Training / CT)
In an automated Level 1 / Level 2 MLOps system, pipeline execution is triggered dynamically through three primary architectural patterns: +---------------------------------------------------------------------------------------------------------+
- | AUTOMATED RETRAINING TRIGGER ARCHITECTURES |
- +------------------------------------+------------------------------------+-------------------------------+
- | TRIGGER PATTERN | MECHANISM / GCP SERVICES | BEST SUITED FOR |
- +------------------------------------+------------------------------------+-------------------------------+
- | 1. Time-Based (Scheduled) | Cloud Scheduler -> Cloud Function/ | Periodic retraining (nightly, |
- | | Cloud Run -> Vertex AI Pipelines | weekly) on seasonal datasets |
- | 2. Event-Based (Data Driven) | GCS Upload -> Eventarc -> | Immediate retraining when new |
- | | Cloud Function -> Vertex Pipelines | labeled batches/shards land |
- | 3. Drift/Performance-Based | Model Monitoring -> Cloud Logging | Reactive retraining when data |
- | | -> Pub/Sub -> Cloud Function | drift or accuracy drops occur |
- +------------------------------------+------------------------------------+-------------------------------+
- 1. Time-Based Scheduled Retraining
Architecture : A Cloud Scheduler cron job (e.g., 0 2 * * 0 for every Sunday at 2:00 AM) emits an HTTP request or Pub/Sub message to a Cloud Function (or Cloud Run microservice). The Cloud Function uses the google-cloud-aiplatform Python SDK to submit a PipelineJob to Vertex AI Pipelines with updated date partition parameters.
2. Event-Based Data Ingestion Triggers
Architecture : When upstream data pipelines complete and dump new training data files (e.g., TFRecords, Parquet) into a Cloud Storage bucket, Eventarc captures the google.cloud.storage.object.v1.finalized audit event and triggers a Cloud Function to execute the retraining pipeline immediately.
3. Drift-Driven Reactive Retraining
Architecture : Vertex AI Model Monitoring continuously monitors real-time endpoint prediction requests, computing drift distances against training baselines. When feature drift (e.g., Jensen-Shannon divergence > 0.1) or prediction drift is detected, Model Monitoring writes an alert to Cloud Monitoring / Cloud Logging . A Log-based alert routes a message to a Pub/Sub topic , which invokes a Cloud Function to trigger the Vertex AI Retraining Pipeline.
4. Automated Model Validation and Promotion Gates (Champion vs. Challenger)
An automated pipeline must never push a newly trained model directly to production without comparative validation. Automated pipelines implement Champion vs. Challenger gating: +---------------------------------------------------------------------------------------------------------+
- | CHAMPION VS. CHALLENGER VALIDATION GATE |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | [ Candidate Model (Challenger) ] [ Current Production Model (Champion) ] |
- | \ / |
- | \ / |
- | v v |
- | +---------------------------------------------------------------------------------------------+ |
- | | Model Evaluator Component (TFMA / KFP) | |
- | | - Evaluates both models against identical Golden Holdout Dataset | |
- | | - Computes global metrics: AUC-ROC, PR-AUC, F1, Log-Loss | |
- | | - Computes sliced fairness metrics across customer demographics | |
- | +---------------------------------------------------------------------------------------------+ |
- | | |
- | v |
- | +---------------------------------------------------------------------------------------------+ |
- | | Automated Promotion Gate Logic | |
- | | Gating Criteria: |
- | | 1. Challenger AUC >= Champion AUC + 0.005 (Statistically significant improvement) | |
- | | 2. Challenger slice disparity <= 0.02 (Fairness constraint satisfied) | |
- | | 3. Challenger P99 Latency <= 30ms (SLA constraint satisfied) | |
- | +---------------------------------------------------------------------------------------------+ |
- | / \ |
- | (PASS) (FAIL) |
- | / \ |
- | v v |
- | [ Update Model Registry Alias: @champion ] [ Reject Challenger; Retain Champion ] |
- | [ Trigger Canary Rollout to Endpoint ] [ Alert ML Team via Cloud Monitoring ] |
- +---------------------------------------------------------------------------------------------------------+
In KFP v2, the gate is implemented via a comparison component that queries Vertex AI Model Registry for the active @champion model, downloads the baseline evaluation dataset, runs inference on both models, and outputs a boolean parameter promote_candidate . A dsl.Condition(promote_candidate == True) block gates downstream deployment tasks.
5. Preventing Training-Serving Skew
Training-serving skew occurs when the feature values or data distributions presented to a model during training differ from the feature values presented during online serving. This is one of the most common causes of silent ML production failures. +---------------------------------------------------------------------------------------------------------+
- | CAUSES AND MITIGATION OF TRAINING-SERVING SKEW |
- +------------------------------------+------------------------------------+-------------------------------+
- | ROOT CAUSE | MECHANISM | RECOMMENDED GCP SOLUTION |
- +------------------------------------+------------------------------------+-------------------------------+
- | 1. Dual-Codebase Discrepancy | Feature logic written in Python | Export preprocessing graph |
- | | for training, re-implemented in | using `tf.transform` directly |
- | | Java/Go/C++ for online serving | inside the model artifact |
- | 2. Time-Travel Feature Leakage | Training data uses future feature | Use Vertex AI Feature Store |
- | | state that was unavailable at the | Point-in-Time (PIT) lookups |
- | | moment of prediction | for training data generation |
- | 3. Data Distribution Drift | Upstream data schema or consumer | Vertex AI Model Monitoring & |
- | | behavior changes silently | TFDV schema validation |
- | 4. Inconsistent Aggregation Logic | SQL batch aggregation logic differs| Apache Beam (Dataflow) for |
- | | from streaming feature processing | unified batch/stream logic |
- +------------------------------------+------------------------------------+-------------------------------+
- Best Practice Implementations on Google Cloud
tf.transform Preprocessing Graphs : In TensorFlow/TFX pipelines, tf.transform generates a TensorFlow computation graph containing all tokenization, scaling, and embedding logic. This graph is saved directly inside the exported SavedModel artifact , ensuring that raw client JSON payloads pass through the exact same mathematical transformation graph during online serving.
BigQuery ML TRANSFORM Clause : When training models in BigQuery ML, encapsulating feature engineering in the CREATE MODEL ... TRANSFORM(...) clause bakes all preprocessing transformations into the model object for both batch and real-time prediction.
Vertex AI Feature Store : Centralizes feature definitions across the enterprise, serving online features via low-latency Redis/Bigtable APIs while ensuring point-in-time correct historical feature joins for training.