12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE

Key Takeaways Explain This Topic AI

Model Garden serves Google models through managed APIs, partner and open models through MaaS, and open or custom-weight models through one-click self-deployment to dedicated endpoints.

Cloud Run services can attach one NVIDIA L4 (24 GB) or RTX PRO 6000 Blackwell (96 GB) GPU per instance, and GPU instances can scale to zero.

Cloud Run GPU instances with preinstalled drivers start in about 5 seconds, which suits spiky or low-traffic inference.

GKE inference supports model servers such as vLLM, TensorFlow Serving, and Triton, with GKE Inference Gateway, Horizontal Pod Autoscaling, and Inference Quickstart.

Choose GKE when you need Kubernetes-level control, portability, or shared accelerator pools, and Agent Platform endpoints when you want managed ML serving features such as Model Registry versions and Model Monitoring.

Last updated: September 2026

The exam guide names Agent Platform, Model Garden, Cloud Run, and GKE as serving options. Picking among them depends on the model source, traffic pattern, operations appetite, and the ML-specific features you need.

Serving Through Model Garden

• Model source Serving path Operations
• Google models (Gemini, Veo, embeddings) Managed API with PayGo, Provisioned Throughput, or batch None
• Partner and open models via MaaS Serverless API ("API Service" model cards) None

Open models with your weights or tuning One-click self-deployment to a dedicated endpoint in your project, using prebuilt serving containers (vLLM, Hex-LLM, SGLang, TGI, TensorRT-LLM) or a custom vLLM container You choose machine type and accelerators and pay for compute while deployed

Self-deployed endpoints support Agent Platform features such as dedicated or private networking and the Chat Completions API for Model Garden models. Model Garden also offers batch inference for supported models. Before deploying, use the SDK's list_deploy_options() to see verified machine and accelerator configurations.

Serving on Cloud Run

Cloud Run runs stateless containers with request-based autoscaling, and it can scale to zero .

• Capability Detail
• GPUs NVIDIA L4 (24 GB) or RTX PRO 6000 Blackwell (96 GB). One GPU per instance
• Minimum resources L4: at least 4 CPU and 16 GiB memory. RTX PRO 6000: at least 20 CPU and 80 GiB
• Startup Instances with GPU drivers preinstalled start in about 5 seconds
• Scaling On-demand GPUs with no reservation needed, and scale to zero when idle
• Regions Limited to specific regions for each GPU type

Cloud Run fits well when:

Traffic is spiky or low , and idle cost should be near zero.

The model fits on one GPU or runs on CPU (small LLMs, embedding models, scikit-learn, XGBoost).

The team already deploys containerized web services and wants the same workflow for models.

You need a custom HTTP API around the model (authentication, business logic).

Trade-offs: you build and maintain the serving container, and cold starts include loading model weights (so keep images and weights small, or load weights from Cloud Storage efficiently). Cloud Run can split traffic between revisions, but ML-specific features such as built-in Model Monitoring , Model Registry version deployment, and explanations aren't provided the way Agent Platform endpoints provide them. You manage model versioning yourself.

Serving on GKE

GKE runs model servers as Kubernetes Deployments.

• Component Role
• Model servers vLLM, JetStream, TensorFlow Serving, Triton Inference Server, or a custom Python server
• GKE Inference Gateway Load balancing and routing tailored to gen AI inference
• Horizontal Pod Autoscaler (HPA) Scale pods on CPU, latency, or custom metrics

GKE Inference Quickstart Recommended accelerators, model servers, scaling, and storage configurations based on performance and cost goals

Autopilot vs. Standard Autopilot manages nodes, including accelerators. Standard gives full node-pool control

Ecosystem Kueue for queueing, Ray for distributed compute, LeaderWorkerSet for multi-host serving, LoRA adapters

GKE supports models from about 8 billion to 671 billion parameters on GPUs and TPUs, with dashboards for popular servers such as vLLM.

GKE fits well when:

The organization runs Kubernetes already and wants one platform for apps and models.

Portability across environments matters.

You need fine-grained control : custom schedulers, multi-host serving of very large models, or sharing accelerator pools across many models.

Very high, steady volume makes hand-tuned serving cost-effective.

Trade-offs: you operate clusters, upgrades, security, and scaling policies.

Choosing a Serving Platform

Requirement Best fit

Managed ML serving with model versions, traffic splits, Model Monitoring, and explanations Agent Platform endpoints

• Use Gemini or a partner/open model with no infrastructure Model Garden API or MaaS
• Serve your own weights for an open LLM with managed endpoint features Model Garden self-deployment
• Spiky traffic, scale to zero, single-GPU or CPU models, container-native team Cloud Run
• Kubernetes-standardized org, portability, shared GPU/TPU pools, multi-host giant models GKE
• Batch scoring of data in BigQuery BigQuery ML or Agent Platform batch inference
• Operational Checklist for Any Serving Platform

Concern Agent Platform endpoint Cloud Run GKE

Versioned rollout Traffic split across deployed models Revision traffic split Deployment strategies or Gateway routing

Autoscaling Min/max replicas with CPU, GPU, and request metrics Request concurrency, scale to zero HPA on custom metrics

Private access Private Service Connect or private endpoints Internal ingress, VPC connectors Internal load balancers

• Prediction logging for monitoring Built-in request-response logging Build it yourself Build it yourself
• Model Monitoring Built in Custom Custom
• Worked Scenarios

An internal document classifier gets 200 requests a day during business hours, with a small fine-tuned transformer on one L4. Cloud Run with an L4 GPU scales to zero overnight, so idle cost is minimal.

A bank serving a fraud model needs canary rollouts between versions, drift monitoring, and private networking. Agent Platform endpoint (Private Service Connect) with traffic splitting and Model Monitoring.

A platform team serves 40 open LLMs and adapters to many product teams on shared H100 pools with custom routing. GKE with vLLM, Inference Gateway, and HPA.

A startup wants a Llama chat model with no GPU management. MaaS from Model Garden.