Study guide

4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green

Key Takeaways Explain This Topic AI

Vertex AI Endpoints natively support multi-model traffic splitting, enabling granular integer percentage traffic distribution (e.g., 90% to Model A, 10% to Model B) across deployed models behind a single invariant endpoint URI.

Canary deployments minimize production release risk by exposing candidate models to a controlled initial fraction of live traffic (e.g., 5% -> 25% -> 50% -> 100%) while continuously monitoring latency, error rates, and prediction drift.

Blue-Green deployments provide zero-downtime, instantaneous cutover between an active production environment (Blue) and an identical staging environment (Green) with immediate 100% traffic reallocation.

Shadow deployments (dark launching) mirror live production inference requests to a candidate model asynchronously, validating real-world performance, memory stability, and prediction fidelity without affecting client responses.

Vertex AI Model Registry version aliases (such as @production, @staging, @champion) combined with instant traffic split adjustments provide zero-downtime rollback capabilities in the event of performance degradation.

Last updated: September 2026

4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green

Deploying a newly trained machine learning model into production is rarely an all-or-nothing event. Machine learning models exhibit non-deterministic behaviors, subtle data dependencies, and sensitivity to distribution shifts that traditional unit testing cannot fully catch. In enterprise Google Cloud environments, deploying new models requires sophisticated rollout strategies to validate latency, numerical stability, inference accuracy, and business metrics under real-world production conditions without jeopardizing user experience or system availability.

1. Vertex AI Endpoint Multi-Model Architecture & Native Traffic Splitting

A core architectural capability of Vertex AI Endpoints is the ability to host multiple deployed models (or multiple versions of the same model) behind a single HTTPS endpoint URI with fine-grained, dynamic traffic splitting. +---------------------------------------------------------------------------------------------------------+

  • | VERTEX AI NATIVE TRAFFIC SPLITTING |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | Inference Request: POST https://{REGION}-aiplatform.googleapis.com/v1/projects/{P}/endpoints/{E}:predict
  • | | |
  • | v |
  • | +---------------------------------------------------+ |
  • | | Vertex AI Managed Endpoint | |
  • | | Traffic Split Engine (100%) | |
  • | +---------------------------------------------------+ |
  • | / \ |
  • | 90% Live Traffic 10% Canary Traffic |
  • | / \ |
  • | v v |
  • | +-------------------------------+ +-------------------------------+ |
  • | | Deployed Model ID: 101101 | | Deployed Model ID: 202202 | |
  • | | Model: Churn-Classifier:v1 | | Model: Churn-Classifier:v2 | |
  • | | Status: Production Champion | | Status: Candidate Canary | |
  • | | Replicas: [3 .. 10 nodes] | | Replicas: [1 .. 3 nodes] | |
  • | +-------------------------------+ +-------------------------------+ |
  • +---------------------------------------------------------------------------------------------------------+
  • How Traffic Splitting Operates

When an endpoint receives an inference request, the Vertex AI routing engine inspects the endpoint's configured traffic_split dictionary. The dictionary maps each deployed_model_id to an integer percentage, where the sum of all values must equal 100 : # Configuring Traffic Split via Vertex AI Python SDK

from google.cloud import aiplatform

endpoint = aiplatform.Endpoint("projects/my-project/locations/us-central1/endpoints/1234567890")

  • # Deploy candidate model (v2) with 10% traffic while champion (v1) retains 90%
  • endpoint.deploy(
  • model=candidate_model_v2,
  • deployed_model_display_name="churn_v2_canary",
  • machine_type="n1-standard-4",
  • accelerator_type="NVIDIA_TESLA_T4",
  • accelerator_count=1,
  • min_replica_count=1,
  • max_replica_count=3,
  • traffic_percentage=10 # Automatically adjusts existing deployed models to total 100%
  • )
  • # Or explicitly mutate traffic split across active deployed model IDs
  • endpoint.set_traffic_split(
  • traffic_split={
  • "deployed_model_id_v1": 90,
  • "deployed_model_id_v2": 10
  • }
  • )

Because client applications continue sending requests to the invariant Endpoint resource URI, routing adjustments occur entirely server-side with zero application downtime and zero client SDK modifications.

2. Canary Deployment Strategy

A Canary Deployment is an incremental rollout strategy where a new model version is initially exposed to a tiny fraction of live production traffic (e.g., 5%). Over time, as telemetry confirms health and accuracy, traffic is monotonically increased until the canary replaces the legacy model completely. +---------------------------------------------------------------------------------------------------------+

  • | CANARY ROLLOUT PROGRESSION |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | [ Stage 1: Initial Ingress ] ===> Model v1 (95%) | Model v2 [Canary] (5%) |
  • | * Health Gate 1: Cloud Monitoring (Latency P99 < 30ms, Errors = 0)|
  • | |
  • | [ Stage 2: Expanded Load ] ===> Model v1 (75%) | Model v2 [Canary] (25%) |
  • | * Health Gate 2: Numerical Drift & Prediction Calibration Check |
  • | |
  • | [ Stage 3: Split Majority ] ===> Model v1 (50%) | Model v2 [Canary] (50%) |
  • | * Health Gate 3: Resource Saturation & GPU Memory Stability |
  • | |
  • | [ Stage 4: Full Promotion ] ===> Model v1 (0%) | Model v2 [Promoted] (100%) |
  • | * Action: Undeploy Model v1 from Endpoint to stop compute billing |
  • +---------------------------------------------------------------------------------------------------------+
  • Automated Gating Criteria for Canary Promotion

During a canary rollout, automated CI/CD pipelines (e.g., Cloud Build, Vertex AI Pipelines) evaluate critical telemetry metrics before advancing to subsequent stages:

System Health Metrics : HTTP 4xx/5xx error rates, gRPC error codes, P95 and P99 latency percentiles, container restart counts, and CPU/GPU memory saturation.

Statistical Data & Prediction Drift : Vertex AI Model Monitoring compares the empirical distribution of incoming feature vectors and predicted probabilities against the training baseline. If Jensen-Shannon divergence or Wasserstein distance exceeds preset thresholds, the rollout halts.

Fast Rollback Mechanism : If any health gate fails, the pipeline immediately resets the traffic split ( {"deployed_model_id_v1": 100, "deployed_model_id_v2": 0} ) and undeploys the faulty canary model in seconds.

3. Blue-Green Deployment Strategy

A Blue-Green Deployment provides an instantaneous, atomic cutover between two identical environments: the currently serving production model ( Blue ) and the newly staged candidate model ( Green ). +---------------------------------------------------------------------------------------------------------+

  • | BLUE-GREEN ATOMIC CUTOVER |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | PHASE 1: STAGING & WARM-UP |
  • | [ Production Traffic ] ======> (100%) ===> [ Blue Model (v1) - Live Production ] |
  • | [ Synthetic Smoke Tests ] ===> (0%) ===> [ Green Model (v2) - Staging / Warm-Up ] |
  • | |
  • | PHASE 2: ATOMIC CUTOVER (Single API Call) |
  • | [ Production Traffic ] ======> (0%) ===> [ Blue Model (v1) - Idle Standby ] |
  • | ======> (100%) ===> [ Green Model (v2) - Promoted Active ] |
  • | |
  • | PHASE 3: DECOMMISSIONING |
  • | [ Undeploy Blue Model from Endpoint to eliminate idle VM billing ] |
  • +---------------------------------------------------------------------------------------------------------+
  • Implementation Patterns in Google Cloud

Pattern A: Single Endpoint with Dual Deployed Models (Recommended) : Both Blue and Green models are deployed to the same Vertex AI Endpoint. Green is initially deployed with traffic_percentage=0 . Synthetic smoke tests and warm-up requests are sent to Green by specifying the deployed_model_id header. Once Green is validated, endpoint.set_traffic_split({"green_id": 100, "blue_id": 0}) executes an instantaneous cutover. Blue is retained as warm standby for 1–2 hours before being undeployed.

Pattern B: Dual Endpoints with Cloud Load Balancing : Two distinct Vertex AI Endpoints are provisioned behind an HTTP(S) Cloud Load Balancer or API Gateway. Cutover is achieved by updating the Load Balancer backend service URL map.

4. Shadow Deployment (Dark Launch) & Production A/B Testing

When deploying high-risk models (e.g., algorithmic trading, medical diagnosis, critical fraud scoring), exposing even 1% of live users to an unvalidated candidate model is unacceptable. In these scenarios, Shadow Deployments and A/B Testing provide necessary validation. +---------------------------------------------------------------------------------------------------------+

  • | SHADOW DEPLOYMENT (DARK LAUNCH) |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | +-----------------------------------+ |
  • | | Client Application | |
  • | +-----------------------------------+ |
  • | | |
  • | (Synchronous Request) |
  • | v |
  • | +-----------------------------------+ |
  • | | API Gateway / Cloud Run | |
  • | +-----------------------------------+ |
  • | / \ |
  • | (Sync Production Call) (Async Shadow Mirror) |
  • | / \ |
  • | v v |
  • | +-------------------------------+ +-------------------------------+ |
  • | | Primary Production Model | | Shadow Candidate Model | |
  • | | (Vertex AI Endpoint Blue) | | (Vertex AI Endpoint Shadow) | |
  • | +-------------------------------+ +-------------------------------+ |
  • | | | |
  • | (Returns Live Response) (Discards Response Body / |
  • | | Logs to BigQuery for Analysis) |
  • | v | |
  • | [ User Receives Prediction ] [ Offline Parity & Latency Eval ] |
  • +---------------------------------------------------------------------------------------------------------+
  • Shadow Deployments (Dark Launching)

Mechanism : The API gateway or routing proxy receives the client request, forwards it synchronously to the primary production model (whose response is returned to the user), and asynchronously forks a duplicate request to the shadow model.

Key Benefits : The shadow model experiences 100% of real production traffic, concurrency spikes, and payload anomalies with zero risk to users . Shadow predictions are logged to BigQuery or Cloud Storage, allowing data scientists to evaluate numerical parity, P99 latency, and memory consumption offline.

Trade-off : Requires running double compute infrastructure during the shadow evaluation window.

A/B Testing in Production

Unlike canary rollouts (which evaluate technical health metrics), A/B Testing evaluates business KPIs (e.g., click-through rate, conversion rate, revenue per user) across competing models.

Deterministic Partitioning : Users are segmented based on deterministic hashing (e.g., hash(user_id) % 100 ). Users in Group A always hit Model A, while users in Group B always hit Model B. This ensures consistent user experience across multiple sessions while collecting statistically valid conversion data.

5. Model Registry Governance and Rollback Mechanics

Vertex AI Model Registry provides centralized version governance to support continuous delivery and instant rollbacks: +---------------------------------------------------------------------------------------------------------+

  • | MODEL REGISTRY VERSION ALIASING |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | Model: `customer-risk-classifier` |
  • | +-----------+-----------------------+-------------------------------------------------------------+ |
  • | | Version | Aliases | Description | |
  • | +-----------+-----------------------+-------------------------------------------------------------+ |
  • | | v1 | `@champion`, `@backup`| Proven production baseline model (Deployed to Endpoint) | |
  • | | v2 | `@candidate` | New XGBoost model currently passing Canary stage 2 (25%) | |
  • | | v3 | `@experimental` | Retrained model undergoing offline evaluation in staging | |
  • | +-----------+-----------------------+-------------------------------------------------------------+ |
  • +---------------------------------------------------------------------------------------------------------+

Version Aliases : Instead of hardcoding explicit version numbers (e.g., v1 , v2 ) in deployment scripts, pipelines reference mutable aliases (e.g., @production , @challenger ). Swapping an alias pointer immediately promotes or demotes a model version across automated deployment pipelines.

Zero-Downtime Rollback Protocol : If a newly promoted model experiences silent accuracy degradation or memory leakage, engineers execute a one-line SDK command to restore 100% traffic to the standby model ID, restoring full operational stability in seconds.

6. Deployment Strategy Decision Matrix

Deployment Strategy Traffic Allocation Downtime Blast Radius Risk Infrastructure Cost Primary Validation Focus

Canary Deployment Incremental (e.g., 5% -> 25% -> 100%) Zero Low (impact limited to canary fraction) Low-Medium (small additional canary node pool) Technical health, error rates, latency, prediction drift

Blue-Green Cutover All-or-Nothing (0% to 100% switch) Zero Medium (affects all users immediately upon cutover) Medium-High (requires dual full-capacity environments temporarily) Zero-downtime cutover, instant rollback capability

Shadow Deployment Dual-write (100% sync / 100% async) Zero Zero (shadow responses are discarded) High (2x compute cost during test window) High-concurrency load testing, numerical parity, memory leak detection

A/B Testing User-segmented split (e.g., 50% / 50%) Zero Medium (controlled across user cohort) Medium (shared or dual node pools) Long-term business KPIs, conversion rates, user engagement

Loading diagram... Decision Flowchart: Selecting Production ML Deployment Strategies