Study guide
14.3 Serving Hardware & Autoscaling for Throughput
Google Cloud Professional Machine Learning Engineer PMLE
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & the Agent Platform Map
1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map
2 Chapter 2: Low-Code ML with BigQuery ML
2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery
3 Chapter 3: AutoML & Google Cloud AI APIs
3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs
4 Chapter 4: Foundation Models in Model Garden
4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability
5 Chapter 5: Exploring, Preprocessing & Protecting Data
5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows
6 Chapter 6: Notebooks, Prototyping & Experiment Tracking
6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
7 Chapter 7: Evaluating Predictive & Generative AI Solutions
7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
8 Chapter 8: Choosing the Modeling Approach
8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements
9 Chapter 9: Training Data & Training SDKs
9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices
10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning
10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune
11 Chapter 11: Training Hardware & Distributed Training
11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism
12 Chapter 12: Deploying Models for Inference
12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers
13 Chapter 13: Model Versions, Rollouts & Inference Logic
13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing
14 Chapter 14: Scaling Online Serving
14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving
15 Chapter 15: End-to-End ML Pipelines
15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving
16 Chapter 16: Orchestration with Managed Airflow & Ray
16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads
17 Chapter 17: Automating Retraining & CI/CD/CT
17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build
18 Chapter 18: Securing AI Systems, Responsible AI & Explainability
18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform
19 Chapter 19: Monitoring AI Solutions
19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
14.3 Serving Hardware & Autoscaling for Throughput
Key Takeaways Explain This Topic AI
By default, Agent Platform autoscales online inference so CPU usage, or the higher of CPU and GPU usage on GPU deployments, matches a 60% target.
Autoscaling can also use request count per minute, DCGM GPU utilization, vLLM KV-cache usage, vLLM waiting requests, or Pub/Sub queue size.
Replica targets are computed as ceil(current replicas × current utilization / target), evaluated every 15 seconds against the highest target from the last 5 minutes.
Scale To Zero (preview) lets minReplicaCount be 0 on endpoint types other than shared public; requests to a scaled-down model get a 429 while it scales up.
Model co-hosting in a DeploymentResourcePool shares one VM across several sparse-traffic models, but the models aren't isolated from each other.
Last updated: September 2026
The exam guide lists choosing appropriate hardware (for example, CPU, GPU, TPU, and edge) and scaling the serving backend based on throughput (for example, Agent Platform Inference and containerized serving) .
Choosing Inference Hardware
Hardware Best for Examples
CPU Tree ensembles, linear models, small neural networks, low-to-moderate QPS, latency tolerance in tens of ms XGBoost fraud scoring, scikit-learn pipelines
GPU Deep networks with heavy matrix math, large batches, vision and speech models, LLM serving NVIDIA L4 for cost-efficient inference, A100, H100, and H200 for large LLMs
TPU Large TensorFlow, JAX, or PyTorch/XLA models at high throughput Agent Platform supports single-host Cloud TPU v5e, v6e, and TPU7x for online inference (multi-host in preview). Use the optimized TensorFlow runtime (2.15+) or PyTorch TPU container (2.1+)
Edge Offline or intermittent connectivity, on-device latency, data that must stay local AutoML Edge exports (TF Lite, Edge TPU TF Lite, Core ML, TensorFlow.js, container). Agent Platform on Google Distributed Cloud for on-premises or disconnected sites
Right-size by measurement. Load-test candidate machine types at target QPS, and compare cost per 1,000 predictions at your p95 latency target. A GPU replica that serves 20× a CPU replica's throughput can be cheaper overall.
How Agent Platform Autoscaling Works
Set dedicatedResources.minReplicaCount and maxReplicaCount . If max is higher than min, the deployment autoscales.
Default targets
CPU-only deployments: scale so CPU utilization ≈ 60% .
GPU deployments: scale so the higher of CPU or GPU utilization ≈ 60% . A container with a busy unrelated CPU process can trigger unnecessary scale-up. If you configure scaling on CPU only, it won't scale up for high GPU usage .
Custom metrics ( autoscalingMetricSpecs )
- Metric Unit Default
- CPU utilization % per replica 60% target
- GPU duty cycle % per replica 60% target
- DCGM GPU utilization (preview) % per replica Disabled
- vLLM GPU KV-cache usage (preview) % per replica Disabled
- vLLM requests waiting (preview) Requests per replica Disabled
- Request count Requests per minute per replica Disabled
- Pub/Sub undelivered messages (preview) Messages per replica Disabled
For LLM serving, KV-cache usage or waiting requests reflect load better than CPU. For uniform, predictable requests, request count per replica gives a direct throughput target.
- The formula and timing
- Target replicas = ceil(current replicas × current utilization ÷ target utilization)
- 2 replicas at 100% with a 60% target → ceil(2 × 100/60) = ceil(3.33) = 4
- 10 replicas at 1% → ceil(10 × 1/60) = ceil(0.167) = 1
Every 15 seconds , the system sets replicas to the highest target from the previous 5 minutes . Scale-up happens quickly, and scale-down waits until a 5-minute window stays low. New replicas still need time to provision VMs, pull the container, and load the model , so set minimum replicas to absorb sudden bursts.
Deployment Options That Affect Scaling
Option Behavior
minReplicaCount ≥ 1 (standard) Always-on capacity. The deployment succeeds only when min replicas are ready, unless you set requiredReplicaCount
requiredReplicaCount The deployment counts as successful once this lower number of replicas is ready. The rest keep provisioning
Scale To Zero (preview) minReplicaCount = 0 . Not available on shared public endpoints. Requests while scaled down get 429 "Model is not yet ready" and trigger scale-up to initial_replica_count (default 1). idle_scaledown_period and min_scaleup_period default to 1 hour (range 5 minutes to 8 hours)
Co-hosting (DeploymentResourcePool) Several models share one VM's resources. Good for many models with sparse traffic . Models aren't isolated and can compete for CPU and memory. Up to 20 concurrent deployment requests. An empty pool uses no quota
- Spot VMs Cheaper inference capacity that can be preempted. Suitable for fault-tolerant or non-critical serving
- Reservations Guaranteed accelerator capacity for predictable peaks
- Quota
Serving quota counts CPUs and GPUs used by active replicas. For example, each a2-highgpu-2g replica counts as 24 CPUs and 2 GPUs . If the sum of maxReplicaCount across deployments exceeds quota, some deployments may fail to autoscale during peaks. Request quota for expected peak, not average.
Containerized Serving Alternatives
- Platform Scaling mechanism
- Cloud Run Request concurrency autoscaling, scale to zero, one GPU per instance
GKE Horizontal Pod Autoscaler on CPU, GPU, or custom metrics such as queue depth. Cluster autoscaler for nodes. Inference Gateway for load-aware routing
Monitoring the Serving Backend
Watch these endpoint metrics in Cloud Monitoring, with alerts:
Replica count against maxReplicaCount . Sitting at max means you need more capacity or quota.
Latency percentiles (p50, p95, p99) and error rates (429 and 5xx).
CPU, GPU duty cycle, and memory per replica.
For LLMs, KV-cache usage and queued requests .
Worked Scenario
A retailer's recommendation endpoint gets 50 QPS at night and 2,000 QPS during a daily 6 p.m. flash sale that starts instantly. Each L4 replica handles about 250 QPS at the latency target.
Peak need: 2,000 / 250 = 8 replicas at 100%. At a 60% utilization target, about 14 replicas .
Set maxReplicaCount to at least 14, and confirm GPU quota covers it.
Because the spike is instant and new replicas take minutes to be ready, use a scheduled job to raise minReplicaCount before 6 p.m. and lower it afterward, instead of relying only on reactive autoscaling.
At night, a minimum of 2 replicas keeps the service redundant.