Study guide
4.6 Scaling the Serving Backend for Throughput
Google Cloud Professional Machine Learning Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap
2 Section 1: Architecting Low-Code AI Solutions (~13%)
1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability
3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)
2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance
4 Section 3: Scaling Prototypes into ML Models (~21%)
3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization
5 Section 4: Serving and Scaling Models (~20%)
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput
6 Section 5: Automating and Orchestrating ML Pipelines (~18%)
5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
7 Section 6: Monitoring AI Solutions (~13%)
6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production
4.6 Scaling the Serving Backend for Throughput
Key Takeaways Explain This Topic AI
Autoscaling on an inference endpoint is driven by a target utilization metric between a minimum and maximum replica count, and the minimum determines both floor cost and cold-start exposure.
Load testing at the target QPS is the only way to establish replicas-per-QPS; capacity planning from assumption is the standard failure.
Scaling out adds replicas for throughput; scaling up adds resources per replica for models that are too slow or too large on the current machine type.
Endpoints do not scale to zero, so genuinely intermittent workloads belong on Cloud Run or in batch prediction rather than on an always-on endpoint.
Feature lookups, preprocessing, and downstream calls are part of the latency budget and are frequently the real bottleneck rather than model compute.
Last updated: September 2026
4.6 Scaling the Serving Backend for Throughput
Blueprint reference: Section 4.2, "Scaling the serving backend based on the throughput (e.g., Gemini Enterprise Agent Platform Inference and containerized serving)."
Scaling questions are capacity-planning questions. The exam supplies a traffic shape and a latency budget and expects a configuration that meets both without paying for idle capacity.
How Endpoint Autoscaling Works
A deployed model on an inference endpoint is configured with:
Minimum replica count — the floor. Always running, always billed. Determines resilience to sudden spikes and how often a request meets a cold replica.
Maximum replica count — the ceiling. Caps both capacity and spend.
A target metric — typically target CPU utilization, or accelerator duty cycle for accelerator-backed deployments. The service adds replicas when observed utilization exceeds the target and removes them when it falls below.
Two properties follow that questions like to test:
A normal deployment has a floor of one replica. For a standard deployment min_replica_count must be at least 1 (and defaults to 1), so the endpoint always carries a baseline cost. There is a separate Scale To Zero feature, enrolled by setting min_replica_count = 0 , which lets a deployment drop to zero replicas and stop billing when idle — but the trade is explicit: a request arriving at a scaled-down endpoint is rejected with 429 - Model is not yet ready for inference and is what triggers the scale-up back to initial_replica_count . It therefore only suits callers that retry and tolerate that first failure, and it is not available for co-hosted deployment resource pools.
Scaling is reactive and takes time. A replica must be provisioned and the model loaded before it serves. For a large model this is not instant, so a spike arriving faster than replicas can start will queue or fail regardless of the maximum.
Scaling Out Versus Scaling Up
Scale out (more replicas) Scale up (bigger machine per replica)
Fixes Insufficient throughput at acceptable per-request latency Per-request latency too high, or the model does not fit
Symptom Latency rises with QPS; each request is individually fast at low load Latency is high even at one request per second
Limit Maximum replica count, quota Largest available machine type
The diagnostic is simple: measure latency at a single request per second. If it already exceeds the budget, more replicas will not help — the model or the machine is the problem. If it is fast alone and degrades under load, the answer is more replicas.
Capacity Planning by Load Test
The only defensible way to size an endpoint:
Deploy one replica of the exact model version on the exact machine type.
Drive load at increasing QPS, measuring p50, p95, and p99 latency and error rate.
Find the QPS at which p95 crosses the budget — that is one replica's usable capacity .
Divide the target QPS by that figure, add headroom for the autoscaler's reaction time, and set the minimum from the traffic floor and the maximum from the expected peak plus margin.
Include realistic payloads. A load test with a tiny synthetic input measures nothing useful if production requests carry large images or long documents.
Handling Spikes and Cold Starts
Raise the minimum replica count to the level that absorbs a spike while the autoscaler reacts. This is the direct trade: predictable spike handling in exchange for continuous cost.
Pre-warm before known events. For a campaign launch or a scheduled batch of traffic, raise the minimum ahead of time rather than relying on reactive scaling.
Shrink the model so replicas start faster; load time is dominated by model size.
Smooth the traffic with a queue where the workload tolerates asynchrony — a Pub/Sub buffer converts a spike into a longer, flatter processing period.
Set a lower target utilization so scaling begins earlier, at the cost of running more replicas on average.
When an Endpoint Is the Wrong Shape
- Traffic shape Better fit
- Steady, high volume, latency-sensitive Inference endpoint with tuned autoscaling
Very spiky, low average, tolerant of cold starts Cloud Run — scales to zero, pay only when serving; or an endpoint enrolled in Scale To Zero if the caller retries the initial 429
Known population, results tolerate staleness Batch prediction — no serving infrastructure at all
Extremely high sustained throughput with custom routing or multi-container graphs GKE — full control of the topology
Analytical consumption inside the warehouse BigQuery ML in place
The recurring cost trap is an endpoint serving a handful of requests per hour: the minimum replica bills continuously to answer almost nothing, and either Cloud Run or a batch job would cost a fraction.
The Latency Budget Is More Than the Model
When measured latency exceeds expectations, decompose the request before touching the endpoint configuration: Total p95 = network + auth + feature lookup + preprocessing
+ model compute + postprocessing + response
Feature lookups are the usual surprise. A model that infers in 8 ms but waits 60 ms for an online feature read is not a serving-capacity problem, and adding replicas will not fix it. The remedies are on the feature side: co-locate the online store, batch the lookups, cache hot entities, or reduce the number of features fetched per request.
Exam Traps
Assuming a normal endpoint scales to zero. A standard deployment has a floor of one replica; scaling to zero requires explicit enrolment in Scale To Zero and costs the first request a 429.
Adding replicas when single-request latency already breaches the budget. Scale up or compress instead.
Sizing capacity from assumption instead of a load test with realistic payloads.
Ignoring cold start for large models during spikes.
Blaming the endpoint when the feature lookup dominates the budget.