Study guide

4.3 Optimizing Gemini Applications for Cost, Latency & Availability

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

4.3 Optimizing Gemini Applications for Cost, Latency & Availability

Key Takeaways Explain This Topic AI

Agent Platform offers five consumption options for generative models: Provisioned Throughput, Standard PayGo, Priority PayGo, Flex PayGo, and batch inference.

Provisioned Throughput plans come in 1-week, 1-month, 3-month, and 1-year terms, and a 1-year commitment costs 26% less than a 1-month commitment.

Flex PayGo and batch inference each cost about 50% less than Standard PayGo, in exchange for slower or asynchronous responses.

Implicit context caching is on by default and gives a 90% discount on cached input tokens, and explicit caches default to a 60-minute TTL.

Google recommends exponential backoff for 429 errors and the global endpoint to reduce throttling from regional capacity limits.

Last updated: September 2026

The exam guide lists optimizing Gemini-based applications for cost, latency, and availability as its own consideration. Most questions describe a workload pattern (interactive chat, nightly bulk processing, a launch-day spike) and ask which consumption option or optimization fits.

The Five Consumption Options

Option What you get Ideal for Pricing

Provisioned Throughput (PT) Guaranteed throughput for a commitment period Critical, steady, always-on workloads needing an SLA Commitment: 1 week, 1 month, 3 months, or 1 year

Standard PayGo Flexible pay-per-use, no commitment Default for everyday and variable traffic Standard per-token rate

Priority PayGo Higher reliability through priority processing, still pay per use Important workloads needing more reliability than Standard Premium per-token rate

Flex PayGo Lower priority, longer latency, more throttling Latency-tolerant tasks: offline analysis, annotation, evaluation, catalog building About 50% off Standard

Batch inference Asynchronous processing of large request files Large backlogs where results can wait About 50% off Standard

Details worth remembering:

Traffic above your PT baseline ( spillover ) is handled by Standard PayGo by default. With Priority PayGo, spillover performance is more reliable at a premium.

A 1-year PT commitment costs 26% less than 1-month PT, and you can switch the model tied to your PT units to newer models.

Flex PayGo (preview at the time of writing) is used through the global endpoint only. You select it with a request header, and request timeouts can be as long as 30 minutes.

For very large jobs that can wait up to a day, Google recommends batch inference over Flex for throughput and success rate. Batch input can come from Cloud Storage (JSONL) or BigQuery.

A 429 on PayGo reads "Resource exhausted, please try again later." Under PT it reads "Too many requests. Exceeded the Provisioned Throughput."

Combining Options

Google's recommended pattern for latency-sensitive production traffic:

Analyze traffic at minute or second granularity.

Cover the baseline (your highest-priority, steady traffic) with Provisioned Throughput. Don't buy PT for peak, because unused reserved capacity wastes money.

Let spillover go to Standard PayGo, or Priority PayGo if spillover needs steadier performance.

Move latency-tolerant background work to Flex PayGo , and bulk backlogs to batch inference .

Latency Levers

Latency has two measures. Time to first token (TTFT) matters for streaming chat. Time to last token (TTLT) is the total generation time.

Lever Effect

Choose a smaller tier (Flash or Flash-Lite instead of Pro) Faster generation and lower cost, if quality holds on your eval set

Reduce prompt size Lower TTFT

Limit output : concise instructions and max_output_tokens Lower TTLT, since latency scales with tokens generated. Too low a limit cuts answers off

  • Lower the thinking level or budget Fewer internal reasoning tokens. Keep enough for hard tasks
  • Stream responses Users see text sooner, which improves perceived latency
  • Provisioned Throughput Most consistent performance, avoiding queueing during high traffic

Gemini 3 models use a thinking_level setting (such as MINIMAL, LOW, MEDIUM, HIGH, depending on the model), while Gemini 2.5 models use a numeric thinking budget. For all Gemini 3 models, Google deprecated the temperature , top_p , and top_k sampling parameters and recommends removing them from requests.

Availability Levers

Retry with exponential backoff on 429 errors, especially on Standard PayGo.

Hybrid capacity: don't rely only on PayGo for critical apps. PT plus PayGo gives the best protection against resource exhaustion.

Global endpoint: draws on Google's global capacity pool to reduce throttling from regional limits. Don't use it when data must be processed in a specific region, because you can't control where global-endpoint requests run.

Monitor PT usage with alert policies, and add PT units before known events such as product launches.

Smooth traffic and move deferrable work off-peak , since aggregate model demand follows a daily cycle.

Cost Levers

Lever How it saves

Context caching, implicit On by default. Cached input tokens get a 90% discount . Improve hit rates by putting large, shared content at the start of the prompt and sending similar prefixes close together in time

Context caching, explicit You create a cache and reference it by name. The discount on cached input tokens is guaranteed (90% on Gemini 2.5 and later). The default TTL is 60 minutes , adjustable, and storage is billed by time stored

  • Batch inference / Flex PayGo About 50% lower per-token price for work that can wait
  • Right-size PT Cover a percentile of traffic, not the peak

Smaller model or tuned model Flash-Lite pricing, or tuning that replaces long few-shot prompts with shorter ones

  • Token counting Estimate prompt tokens before sending large jobs, and trim unneeded context
  • Measuring Before Optimizing

Tie each optimization to a metric so you can tell whether it worked:

Latency: track TTFT and TTLT percentiles (p50, p95, p99) by model and route. Averages hide tail latency.

Availability: track 429 and 5xx rates by consumption option and endpoint.

Cost: track input, output, and cached tokens per request. The cachedContentTokenCount field in response metadata shows how many input tokens came from cache.

Quality: re-run the evaluation set after every model, prompt, or thinking-level change. A cheaper configuration that quietly lowers answer quality isn't a real saving.

Worked Scenario

A legal-tech app lets users chat with a 500-page contract. Every question resends the contract. Users complain about slow first answers, and costs are high.

Put the contract at the beginning of the prompt and use explicit context caching for the session. This cuts input cost and the processing of repeated content.

Stream responses and cap max_output_tokens to improve perceived and actual latency.

Evaluate whether Flash matches Pro quality on a test set of contract questions.

Run the nightly job that pre-summarizes new contracts as batch inference .

If usage is steady during business hours, cover that baseline with Provisioned Throughput and let spikes spill to PayGo.

Relative Per-Token Price vs. Standard PayGo (Standard = 100)