Study guide

4.1 Evaluating & Selecting Models in Model Garden

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

4.1 Evaluating & Selecting Models in Model Garden

Key Takeaways Explain This Topic AI

Model Garden groups models into foundation models, fine-tunable models, and task-specific solutions from Google, partners, and open-model providers.

Model as a Service (MaaS) serves partner and open models through serverless APIs, while self-deployment runs model weights on dedicated endpoints in your project.

Google recommends MaaS for rapid prototyping and minimal operations, and self-deployment for custom weights, predictable high volume, and strict infrastructure control.

An organization policy for Model Garden can allow only vetted models and deny access to all others at the organization, folder, or project level.

Gemini model versions follow a published lifecycle with retirement dates, such as October 20, 2026 for gemini-2.5-pro, so model selection includes planning migrations.

Last updated: September 2026

The exam guide asks you to evaluate and select the appropriate model for a given task from Model Garden . Model Garden is a single catalog of models from Google, Google partners, and open-model providers. It offers a consistent deployment pattern and built-in links to tuning, evaluation, and serving. Picking well means matching the task, the data, and the operating constraints, not reaching for the biggest model.

What Model Garden Contains

Category Description Examples of use

Foundation models Pretrained multitask large models you can prompt, ground, and tune through Agent Studio, the API, or the SDK Gemini for text, code, and multimodal reasoning. Gemini image models and Veo for media generation. Embedding models for retrieval

Fine-tunable models Models you tune with a notebook or pipeline Open models such as Gemma or Llama tuned with LoRA

Task-specific solutions Prebuilt models, mostly ready to use, often customizable Specialized vision or language task models

Filters include task , model collection (managed by Google, partners, or you), provider , and features . Each model has a model card that describes capabilities, supported regions, limits, and license.

How a Model Is Served

Serving option What it is Choose it when

Google models via API (Gemini, Veo) Fully managed, pay per use or Provisioned Throughput Default for Google's first-party models

Model as a Service (MaaS) Serverless managed APIs for selected partner and open models. Enable the model's "API Service" card first Rapid prototyping, minimal operations, bursty or variable traffic, standard non-customized model

Self-deployed from Model Garden One-click deployment of model weights (including custom or fine-tuned weights) to a dedicated endpoint in your project Custom weights, predictable high volume (lower cost per token at scale), exact hardware choice, strict residency, private VPC data path

Prebuilt serving containers Optimized vLLM, Hex-LLM, SGLang, TGI, or TensorRT-LLM images You want a tuned serving framework without maintaining images

Custom vLLM container Your own image Serving logic or dependencies the prebuilt containers don't support

For self-deployed open models, you pay for the compute the endpoint uses (at prediction pricing) and for tuning compute (at custom training rates), not per token.

A Repeatable Selection Process

Define the task and modality. Classification, extraction, summarization, code, chat with tools, image generation, video generation, or embeddings? Text-only or multimodal input?

List the hard constraints. Latency (time to first token), throughput, cost per request, context length, data residency, whether customization is required, and license terms.

Shortlist by tier. Within the Gemini family, Pro tiers give the most capability for complex reasoning, Flash balances quality and speed, and Flash-Lite is the fast, budget option for high-volume simple tasks. Add a partner or open model when a requirement calls for it, such as self-hosting in your VPC or a specific open license.

Build an evaluation set from your own data. Public benchmarks don't reflect your prompts, documents, or edge cases. Use 50-200 representative prompts with reference answers or rubrics.

Evaluate side by side. Run Gen AI evals with computation metrics, rubric-based metrics, or pairwise comparison with a judge model (Chapter 7). Compare quality, latency, and token cost together.

Pick the smallest model that meets the quality bar. Then check that it supports the features you need: tuning, context caching, Provisioned Throughput, batch inference, and the global endpoint.

Plan the lifecycle. Note the model version's retirement date and schedule re-evaluation against newer versions.

Worked example

A bank needs to extract 20 fields from loan documents, running 300,000 documents a night. Latency doesn't matter, but cost and accuracy do. Evaluation on 150 labeled documents shows Flash-Lite at 94% field accuracy, Flash at 97%, and Pro at 97.5%. The accuracy target is 96%. Flash meets the target at a fraction of Pro's cost. Because the job runs overnight, it should use batch inference for the discount (Section 4.3).

Governance and Safety in Model Selection

Restrict models with organization policy. A Model Garden organization policy can allow only vetted models and deny all others. It applies at the organization, folder, or project level.

Check provenance. Google tests and scans the serving and tuning containers it provides. Featured partner checkpoints are scanned for authenticity. Hugging Face models are scanned by Hugging Face. Unsafe models are blocked, and suspicious ones (for example, able to run remote code) are flagged but can still be deployed, so review them first.

Licenses and terms differ between Google, partner, and open models. Include legal review in the shortlist.

Data path: if regulations forbid multi-tenant managed services, self-deploy inside your project and VPC.

Model Lifecycle Awareness

Model versions follow a published lifecycle. For example, gemini-2.5-pro and gemini-2.5-flash list an October 20, 2026 retirement date, and several open-model MaaS endpoints were deprecated in July 2026 with retirement on October 21, 2026 (self-deployment from Model Garden remains an option). Good designs:

Put the model ID in configuration, not scattered through code.

Keep the evaluation set as a regression suite for testing replacement models.

Re-run evaluations and re-tune tuned models before the base version retires.

Signals That You Picked the Wrong Model

Symptom in production Likely selection issue Next step

Good quality but cost far over budget Tier too large for the task Evaluate Flash or Flash-Lite, or tune a smaller model

Fast and cheap but frequent reasoning errors Tier too small Evaluate a larger tier or add grounding and examples

  • Frequent 429s during business hours Capacity plan, not model quality Review consumption options (Section 4.3)
  • Can't meet residency or no-multi-tenancy rules Wrong serving option Self-deploy an open model in your project
  • Embedding Models Are Selections Too

RAG and semantic search depend on the embedding model . Gemini embedding models ( gemini-embedding-001 , gemini-embedding-2 ) are current. Older text embedding models such as text-embedding-005 and multimodalembedding@001 list an April 1, 2027 retirement date. Changing embedding models means re-embedding the corpus , because vectors from different models aren't comparable.