1.6 Model Garden: Evaluating and Selecting the Right Model

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

1.6 Model Garden: Evaluating and Selecting the Right Model

Key Takeaways Explain This Topic AI

Model Garden catalogues three families with different operational contracts: Google first-party models, third-party partner models served as a managed API, and open-weight models you deploy onto your own endpoint.

Managed API models are billed per token and require no infrastructure; self-deployed open models are billed for the accelerator hours the endpoint holds, whether or not traffic arrives.

Model selection is a four-way trade-off among task capability, latency, cost per unit of work, and deployment constraints such as data residency and VPC isolation.

A model card records the intended use, training data characteristics, evaluation results, and known limitations, and is the artifact a governance review reads before approval.

Selection should be settled with a measured evaluation on representative data — an offline eval set scored with task metrics or an LLM-as-a-judge rubric — rather than a leaderboard.

Last updated: September 2026

1.6 Model Garden: Evaluating and Selecting the Right Model

Blueprint reference: Section 1.2, "Evaluating and selecting the appropriate model for a given task from Gemini Enterprise Agent Platform Model Garden."

Model Garden is the single catalogue of models available on the Agent Platform, and the exam treats "which model would you choose" as an architecture question rather than a benchmark question. The differentiator between a good and a bad answer is almost never raw quality — it is the operational contract attached to each family.

Three Families, Three Contracts

Family Examples How it is served How it is billed Who patches it

Google first-party Gemini, Imagen, Veo, embedding models Managed API endpoint, no infrastructure Per input/output token or per generated asset Google

Partner models as a service Third-party frontier models offered through Model Garden Managed API endpoint Per token, billed through Google Cloud The partner, via Google

Open-weight models Gemma, Llama-family, and other open checkpoints You deploy to an Agent Platform Inference endpoint, GKE, or Cloud Run Accelerator + node hours for as long as the endpoint exists You

That last column is the exam's favourite. A managed API model costs nothing when idle. A self-deployed open model on a GPU-backed endpoint bills for every hour that endpoint is up, whether it serves ten requests or ten million. For spiky, low-volume workloads a managed API is dramatically cheaper; for sustained high-throughput workloads with a fixed, predictable request rate, a reserved self-deployed endpoint can win on unit cost.

The Four Selection Axes

1. Task capability. Does the model do the job at all? Multimodal input, long context, tool calling, structured output, and non-English language coverage are hard capability gates. A cheaper model that cannot accept a PDF is not cheaper; it is disqualified.

2. Latency. Model families ship in tiers — a fast, small variant and a slower, stronger variant. Interactive chat and autocomplete need the fast tier. Nightly document summarization does not, and paying frontier prices for a batch job is pure waste.

3. Cost per unit of work. Compare cost per completed task , not per token. A stronger model that answers correctly on the first attempt can beat a cheaper model that needs three retries plus a validation pass.

4. Deployment constraints. This is where architecture decisions get made:

Data residency. A regulated workload confined to a specific region can only use models available in that region.

Network isolation. If inference traffic may not traverse a public endpoint, a self-deployed open model behind Private Service Connect may be the only viable option.

Weight control. Some governance regimes require the organization to hold the weights, which forces the open-weight path.

Customization depth. Managed APIs support supervised tuning and adapters; full-parameter modification of a proprietary model is not on offer.

Reading a Model Card

Every Model Garden entry carries a model card. Governance reviews read four parts of it:

Intended use and out-of-scope use. Deploying outside the stated intent is the fastest way to fail an internal review.

Training data characteristics. Drives fairness questions and jurisdictional risk.

Evaluation results. Benchmarks reported by the publisher, on their chosen data.

Known limitations. Documented failure modes, language gaps, and safety caveats.

Licence. Open weights do not mean unrestricted commercial use; licences vary and some carry usage restrictions.

The exam-relevant instinct: a model card is evidence for a decision, not proof of fitness for your task. Publisher benchmarks are run on public datasets that rarely resemble enterprise data.

Selecting by Measurement, Not Leaderboard

The defensible selection process is short:

Build a representative eval set. 100–300 real inputs with expected outputs or grading criteria, drawn from actual traffic and deliberately including the hard cases.

Shortlist by hard constraints. Region, modality, context length, isolation. This usually eliminates most of the catalogue immediately.

Score the shortlist. Use task metrics where a ground truth exists (exact match, F1, ROUGE for summarization) and an LLM-as-a-judge rubric where it does not.

Measure latency and cost on the same runs. Record p50 and p95 latency and actual token spend per task, not list prices.

Record the decision. Which models were compared, on what data, with what result — this is what an audit asks for. Candidate Task score p95 latency Cost / 1k tasks Region OK

• ------------ ----------- ------------ ---------------- ---------
• Fast tier 0.81 640 ms low yes
• Strong tier 0.93 2.4 s ~8x fast tier yes
• Open-weight 0.86 410 ms* endpoint hours yes
• * self-deployed on a dedicated accelerator; cost is fixed, not per call

A frequent exam pattern presents exactly this table in prose and asks for the choice given a stated constraint. Anchor on the constraint: an interactive assistant with a strict latency budget takes the fast tier or the self-deployed model; a nightly batch job with a quality bar takes the strong tier; a workload that must not leave a VPC takes the self-deployed model regardless of score.

Exam Traps

Choosing the highest-benchmark model by default. Nearly always wrong when latency or cost is stated.

Ignoring idle cost of self-deployed endpoints. Low-volume plus dedicated accelerator equals waste.

Treating "open model" as "free." The weights are free; the accelerators are not.

Assuming every Model Garden model is in every region. Availability varies and residency requirements bind.

Selecting on a public benchmark instead of your own eval set.