Study guide

17.2 CI/CD/CT for ML with Cloud Build

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

17.2 CI/CD/CT for ML with Cloud Build

Key Takeaways Explain This Topic AI

Cloud Build runs builds as a series of steps, each in a Docker container, and build triggers start them automatically from repositories such as GitHub, Bitbucket, and Cloud Source Repositories.

ML continuous integration tests feature engineering logic, model methods, training convergence, absence of NaN values, expected component artifacts, and component integration.

ML continuous delivery verifies infrastructure compatibility, tests the prediction service API, load-tests latency and QPS, and checks that models meet performance targets before deployment.

Google's pattern deploys automatically to test, semi-automatically to pre-production after approved merges, and manually to production after several successful pre-production pipeline runs.

Cloud Build features meet the requirements of Supply-chain Levels for Software Artifacts (SLSA) level 3.

Last updated: September 2026

The exam guide lists deploying models in continuous integration, continuous delivery, and continuous training (CI/CD/CT) pipelines (for example, Cloud Build) . ML systems ship code (preprocessing, training, pipeline definitions, serving containers), data-dependent artifacts (trained models), and configuration . Each needs automation.

The Three Loops

Loop Input Output Google Cloud services

CI Code commits Tested, versioned components: container images, packages, compiled pipeline templates Cloud Build , Artifact Registry

CD (of pipelines and serving) Tested artifacts Pipeline templates and serving containers deployed to test, pre-production, and production Cloud Build, Artifact Registry (KFP templates), Agent Platform Pipelines

CT New data or triggers Newly trained, validated model versions Agent Platform Pipelines, Model Registry, endpoints

Cloud Build Essentials

A build config ( cloudbuild.yaml ) lists steps . Each step runs in a Docker container , and steps share a workspace and the cloudbuild Docker network.

Steps can use Google-provided builders, community builders, or custom builder images .

Build triggers start builds on pushes, pull requests, or tags from GitHub , Bitbucket , Cloud Source Repositories , and others. Builds can also be started manually with gcloud or the API.

Builds produce artifacts such as container images, which you push to Artifact Registry .

Cloud Build meets SLSA level 3 supply-chain requirements. Pair it with vulnerability scanning and build provenance for secure ML containers.

Approval gates on triggers and private pools (builds with private network access) support regulated environments.

What ML CI Tests

Google's MLOps guidance lists CI tests beyond normal unit tests:

Unit tests for feature engineering logic (for example, one-hot encoding handles unseen categories).

Unit tests for model methods.

Training converges: loss goes down, and the model can overfit a few sample records.

No NaN values from division by zero or extreme values.

Each pipeline component produces its expected artifacts.

Integration between pipeline components.

What ML CD Verifies

Before models or pipelines reach production:

Infrastructure compatibility: required packages are in the serving environment, and memory, compute, and accelerators are available.

Prediction service API tests with expected inputs and responses, which catch input-schema changes between versions.

Load tests for QPS and latency.

Data validation for retraining or batch prediction.

Model performance targets met before deployment.

Environment promotion (Google's recommended pattern)

  • Environment Deployment trigger
  • Test Automated on push to the development branch
  • Pre-production Semi-automated on merge to main after reviewer approval
  • Production Manual , after several successful pipeline runs in pre-production
  • A Reference CI/CD/CT Flow on Google Cloud

A data scientist opens a pull request that changes feature code and the training component.

Cloud Build (CI) runs unit tests, builds the training and serving container images, scans them, pushes them to Artifact Registry , compiles the KFP pipeline , and runs a small integration pipeline on sample data.

On merge, Cloud Build (CD) uploads the compiled pipeline as a versioned template in Artifact Registry and deploys it to pre-production (for example, updating the pipeline schedule).

CT: schedules or events run the pipeline. It validates data, trains, evaluates, and uploads a model version to Model Registry with a challenger alias.

Model CD: if validation gates pass, the pipeline (or a Cloud Build job triggered by the pipeline's result) deploys the challenger as a canary on the endpoint. After approval and guardrail checks, the champion alias moves and traffic shifts to 100%.

Monitoring (Chapter 19) feeds alerts back as retraining triggers or new experiment cycles. # cloudbuild.yaml (simplified)

steps:

  • name: python:3.11
  • entrypoint: bash
  • args: ["-c", "pip install -r requirements.txt && pytest tests/"]
  • name: gcr.io/cloud-builders/docker
  • args: ["build", "-t", "us-central1-docker.pkg.dev/$PROJECT_ID/ml/trainer:$SHORT_SHA", "trainer/"]
  • name: gcr.io/cloud-builders/docker
  • args: ["push", "us-central1-docker.pkg.dev/$PROJECT_ID/ml/trainer:$SHORT_SHA"]
  • name: python:3.11
  • entrypoint: bash
  • args: ["-c", "pip install kfp google-cloud-aiplatform && python compile_and_upload.py --image-tag $SHORT_SHA"]
  • Versioning Everything
  • Artifact Version with
  • Code Git commit SHA
  • Containers Image tag and digest in Artifact Registry
  • Pipeline definitions KFP template versions and tags
  • Data Managed dataset versions, BigQuery snapshots, partitioned tables
  • Models Model Registry versions and aliases
  • Lineage across all of them ML Metadata (automatic in pipeline runs)
  • CI/CD for Gen AI Applications

Prompt templates, system instructions, and tool definitions live in source control and pass through CI.

CI runs Gen AI evals on a fixed evaluation set and fails the build if rubric pass rates or safety scores fall below thresholds (Chapter 7).

Model version upgrades, such as moving before a retirement date, run through the same evaluation and canary process.

Exam Traps

Treating CI/CD as only "build the container." ML CI adds data and model tests, and CD adds validation against infrastructure and performance targets.

Deploying models directly from a notebook to production and skipping registry, approvals, and canaries.

Retraining automatically in production with no human approval for regulated models.

Confusing CT (new model from new data, same code) with CI/CD (new code or pipeline).

Loading diagram... CI/CD/CT for ML on Google Cloud