4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge

Key Takeaways Explain This Topic AI

Serving hardware is chosen from latency at the required throughput and cost per request, not from what the model was trained on.

Most tabular and small models serve fastest and cheapest on CPU; accelerators only pay off when the model is large enough to keep them busy.

Batching raises accelerator throughput but adds queueing latency, so the batch window is a direct latency-versus-cost dial.

Edge deployment is chosen for offline operation, privacy, or network-independent latency, and forces compression through quantization, pruning, or distillation.

Serving accelerators should never run on Spot or preemptible capacity, because reclamation removes replicas from a live endpoint.

Last updated: September 2026

• 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge
• Blueprint reference: Section 4.2, "Choosing appropriate hardware (e.g., CPU, GPU, TPU, and edge)."

Training hardware selection was covered in Section 3.11. Serving is a different problem with a different cost function, and the most common error is assuming a model trained on GPUs must be served on GPUs.

Inference Economics Differ from Training

• Training Serving
• Workload shape One long job, high utilization by design Continuous, often bursty, frequently underutilized
• Batch size Large, chosen for throughput Often 1, dictated by the request
• Cost driver Job duration × device rate Replica-hours × device rate, plus idle time
• Optimization goal Time to trained model Latency at a target QPS, per-request cost
• Interruptible? Yes, with checkpointing No — a reclaimed replica is a dropped request

That last row is a hard rule: Spot and preemptible capacity is correct for training and wrong for a live endpoint , because reclamation removes serving capacity without warning.

When an Accelerator Pays Off at Inference

An accelerator only helps if the model is large enough to occupy it. A gradient boosted tree or a small MLP responds in microseconds on CPU, and putting it on a GPU adds transfer overhead and a much higher hourly rate for no latency gain — the utilization graph will show a device that is idle almost all the time.

• Model Typical serving choice Why
• Tree ensembles, linear, small MLPs CPU Sub-millisecond on CPU; accelerator adds cost, not speed

Mid-size CNNs, embedding models, small transformers CPU with enough cores, or an entry-level inference GPU Depends on QPS; measure before deciding

• Large vision models, large language models GPU Model size and compute make CPU serving impractical
• Very large models at very high sustained throughput, XLA-compatible TPU Cost per unit throughput at scale
• Anything on a device, offline, or privacy-bound Edge Network independence is the requirement

The practical procedure: serve on CPU first, load-test to the required QPS, and move to an accelerator only when CPU cannot meet the latency budget at acceptable replica count. Many teams discover that ten CPU replicas cost less than one GPU replica and meet the target comfortably.

Batching: The Latency-Throughput Dial

Accelerators are efficient on batches and wasteful on single requests. Server-side dynamic batching collects arriving requests for a short window and runs them as one batch. Batch window 0 ms → lowest latency, lowest accelerator efficiency, highest cost/request

• Batch window 10 ms → modest added latency, substantially better throughput
• Batch window 50 ms → high throughput, may breach an interactive latency budget

The window is a design parameter, not a default. Set it from the latency budget: if p95 must stay under 100 ms and inference itself takes 60 ms, a 10 ms batching window is affordable and a 50 ms one is not.

For offline scoring, batching is unambiguously right and the window can be as large as convenient — which is one reason batch prediction is so much cheaper per record than online serving.

Other Levers Before Buying Hardware

Model compression. Quantization, pruning, and distillation (covered in Section 4.3) often move a model from "needs a GPU" to "fits comfortably on CPU."

Optimized runtimes. Compiling the model for the target hardware frequently yields large speedups without changing the device.

Right-sizing the machine type. Serving replicas are often over-provisioned on memory and under-provisioned on CPU, or vice versa; measure.

Caching. If the same inputs recur, a cache in front of the model removes inference entirely for those requests.

Edge Deployment

Edge is chosen for one of three reasons, and the reason determines the constraints:

• Reason Consequence
• Offline operation The model and all its features must live on the device; no server round trip
• Privacy Raw input never leaves the device; only results or aggregates may be transmitted
• Latency independence Network round trip removed entirely; predictable even on poor connectivity

Edge imposes requirements that must be settled at design time:

Size and memory budget. Handled with quantization (typically INT8), pruning , and distillation into a smaller student model.

Runtime. A device-appropriate inference runtime, not the training framework.

Feature availability. Every feature must be computable on-device; a server-side aggregate is not available offline.

Update distribution. A mechanism to ship new model versions to a fleet, plus a way to know which version each device runs.

Monitoring. Devices must report prediction statistics back when connectivity allows, or drift on the fleet is invisible.

Hybrid designs are common and often the best answer: a small on-device model handles the common case offline, and hard cases are escalated to a larger cloud model when connectivity is available.

Exam Traps

Serving a small model on a GPU because it trained on one.

Spot capacity for a production endpoint. Reclamation drops requests.

A large batch window on an interactive latency budget.

Edge deployment with a server-side feature. It will not exist offline.

Buying hardware before trying compression or an optimized runtime.