1.9 Optimizing Gemini Applications for Cost, Latency and Availability

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

1.9 Optimizing Gemini Applications for Cost, Latency and Availability

Key Takeaways Explain This Topic AI

Context caching stores a reusable prompt prefix and bills those cached input tokens at a 90% discount on Gemini 2.5 and later (75% on Gemini 2.0), offset by an hourly storage charge for the cache TTL.

Batch prediction for generative models trades latency for a substantially lower price and is the correct answer whenever results are not needed interactively.

Provisioned Throughput reserves dedicated generative capacity for predictable, guaranteed throughput; pay-as-you-go shared capacity is subject to dynamic shared quota.

Streaming responses cut perceived latency without reducing total token cost, because the user starts reading before generation completes.

Availability engineering for generative endpoints means handling 429 and 503 responses with exponential backoff and jitter, plus a documented fallback model or degraded path.

Last updated: September 2026

• 1.9 Optimizing Gemini Applications for Cost, Latency and Availability
• Blueprint reference: Section 1.2, "Optimizing Gemini-based applications for cost, latency, and availability."

This bullet is where a lot of otherwise-strong candidates lose points, because the levers are specific and mutually exclusive in ways that are easy to blur. A scenario will state exactly one binding constraint — a monthly budget, a p95 latency target, or a throughput guarantee — and exactly one lever is right for it.

The Cost Levers, in Order of Impact

1. Model tier. The largest single cost decision. Generative model families ship in tiers, and the price gap between a fast tier and a frontier tier is typically severalfold. Most production workloads are over-modelled: a classification or extraction task that a fast-tier model handles at 95% accuracy does not need a frontier model at eight times the price. Measure on your eval set before assuming you need the top tier.

2. Context caching. When many calls share a large, unchanging prefix — a long system instruction, a policy manual, a codebase, a video — context caching stores that prefix so it is not reprocessed and re-billed on every request. Cached input tokens are billed at a 90% discount to the standard input rate on Gemini 2.5 and later models (75% on Gemini 2.0). Creating the cache costs one ordinary input-token charge, and the cache carries an hourly storage charge for as long as its TTL keeps it alive — which is why caching pays off on a large prefix reused often, and loses money on a small one reused rarely.

The decision rule: caching pays off when the shared prefix is large and reused frequently within the cache's lifetime. A 40-token system prompt is not worth caching. A 200-page policy document queried hundreds of times an hour is the canonical win.

3. Batch mode. Generative batch prediction submits many requests as one job, returns results to Cloud Storage or BigQuery when complete, and is priced well below interactive inference. If nobody is waiting — nightly summarization, bulk enrichment, backfilling classifications over a warehouse table — batch is almost always the correct answer, and choosing interactive inference for an offline job is a recognisable wrong answer.

4. Output control. Output tokens usually cost more than input tokens. Capping max_output_tokens , asking for structured output instead of prose, and using a constrained response schema all reduce spend directly. AI.COUNT_TOKENS (or the equivalent token-counting call) lets you estimate a job's cost on a sample before running it over the full table.

Real Latency Versus Perceived Latency

These are different problems with different fixes.

• Lever Reduces real latency Reduces perceived latency Reduces cost
• Smaller model tier Yes Yes Yes
• Streaming responses No Yes No
• Context caching Yes (less prefix to process) Yes Yes
• Shorter output Yes Yes Yes
• Regional endpoint near users Yes Yes No
• Provisioned Throughput Yes (removes queueing) Yes No — costs more

Streaming is the highest-leverage fix for interactive chat. Total generation time is unchanged, but the user sees the first tokens in a few hundred milliseconds instead of staring at a spinner for several seconds. If a scenario complains that "users perceive the assistant as slow" while total latency is acceptable, streaming is the answer and switching models is over-correction.

Availability: Provisioned Throughput Versus Shared Quota

Pay-as-you-go generative capacity is served from dynamic shared quota — a shared pool with no reserved allocation. It is cost-efficient and adequate for most workloads, but under contention a request can be throttled with a 429 .

Provisioned Throughput reserves dedicated capacity for a committed term. You pay for the reservation whether or not you use it, and in return you get predictable, guaranteed throughput.

Choose Provisioned Throughput when the scenario mentions a guaranteed or predictable throughput requirement, a customer-facing SLA, a launch event with a known traffic profile, or intolerance of throttling. Choose pay-as-you-go when traffic is spiky, low-volume, or experimental — reserving capacity for a bursty prototype wastes the reservation between bursts.

A common production pattern combines both: Provisioned Throughput sized to the steady-state floor, with overflow spilling to pay-as-you-go.

Retry and Fallback Patterns

Any generative client will eventually see 429 (throttled) and 503 (unavailable). The expected engineering response: # Exponential backoff with jitter — the jitter matters, because synchronized

# retries from many clients recreate the same contention that caused the 429.

• delay = min(BASE * (2 ** attempt), MAX_DELAY)
• sleep(delay * random.uniform(0.5, 1.5))

Retry only retryable codes. A 400 from a malformed request will fail identically forever; retrying it burns quota.

Cap total attempts and surface a clean error rather than retrying indefinitely behind a user-facing request.

Define a fallback. Route to a smaller model, a cached response, or a deterministic non-AI path. A feature that degrades is better than a page that fails.

Idempotency. For expensive generations, key each request so a retry does not pay twice.

Putting It Together

• Stated constraint Correct primary lever
• "Monthly spend has tripled; results are needed by morning" Batch mode
• "Every request resends the same 300-page manual" Context caching
• "Users say the chat feels sluggish; total time is ~4 s" Streaming
• "We must guarantee 200 requests/second during the campaign" Provisioned Throughput
• "Accuracy is fine at the fast tier but we are paying frontier prices" Downgrade the model tier
• "We see intermittent 429s at peak" Backoff with jitter , then evaluate Provisioned Throughput
• "Responses are verbose and expensive" Cap max_output_tokens / structured output
• Exam Traps

Streaming to reduce cost. It does not; it reduces perceived latency only.

Provisioned Throughput for a bursty prototype. You pay for idle reservation.

Interactive inference for an offline job. Batch is cheaper and nobody is waiting.

Retrying 400-class errors. Only 429 and 5xx are retryable.

Caching a tiny prompt. The storage charge can exceed the savings.