14.4 Tuning Models for Production Training & Serving

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

14.4 Tuning Models for Production Training & Serving

Key Takeaways Explain This Topic AI

Quantization lowers numeric precision (for example, to 8-bit or 4-bit weights) to cut memory and latency at a possible accuracy cost that must be measured.

Knowledge distillation trains a smaller student model to mimic a larger teacher, lowering serving cost and latency.

For TensorFlow models, the optimized TensorFlow runtime and server-side request batching can reduce serving cost and latency without code changes.

LLM serving optimizations on Agent Platform and GKE include continuous batching, PagedAttention, prefix caching, speculative decoding, tensor parallelism, and KV-cache quantization.

Production training efficiency comes from mixed precision, efficient input pipelines, checkpointing, warm-starting from previous models, and right-sized distributed training.

Last updated: September 2026

The exam guide ends its serving section with tuning ML models for training and serving in production . Here "tuning" means performance engineering: fitting production latency, throughput, memory, and cost budgets without losing more accuracy than the business accepts.

Start with Measurements

Define service-level objectives (SLOs) : for example, p95 latency under 50 ms at 1,500 QPS, cost under $0.20 per 1,000 predictions, and AUC drop under 0.5 points.

Profile where time goes: request parsing, feature lookup, preprocessing, model forward pass, postprocessing, network.

Load-test candidates on staging endpoints with production-like payloads.

Change one thing at a time , and re-check accuracy on the evaluation set after every optimization.

Model-Level Optimizations

Technique What it does Typical gain Risk

Quantization Store and compute with lower precision (float16, int8, 4-bit) Smaller memory footprint, higher throughput, possibly fewer GPUs Accuracy loss, especially at 4-bit, so evaluate

Pruning Remove low-importance weights or neurons Smaller models, sometimes faster Needs fine-tuning. Speedups depend on hardware support

Distillation Train a small student to mimic a large teacher Large latency and cost reductions Student may lose quality on hard cases

Architecture choice Smaller variants, fewer layers, efficient architectures Fundamental latency improvement Requires retraining

Feature reduction Drop low-value, expensive-to-compute features Faster preprocessing and lookups Slight accuracy change

Compilation Graph optimization (XLA, TensorRT) for target hardware Faster kernels Build complexity, hardware-specific artifacts

Example from GKE's LLM guidance: quantizing Llama-2 13B with GPTQ, or Gemma 7B with AWQ, lets each model serve on one L4 GPU instead of two .

• Serving-Level Optimizations
• Predictive models

Optimized TensorFlow runtime: a drop-in serving image with lower cost and latency than open-source TensorFlow Serving, and no code changes.

Server-side batching: TensorFlow prebuilt containers read config/batching_parameters_config next to the SavedModel. Batching raises GPU utilization and throughput at a small latency cost.

Right hardware: trees on CPU, deep models on GPU or TPU (Section 14.3).

Caching: cache predictions for repeated inputs (such as popular product pages) and cache static lookups in the container.

Precompute: move expensive features to Feature Store or batch jobs.

Co-hosting and scale to zero for low-traffic models, to cut idle cost.

LLM serving

Technique Effect

Continuous batching (vLLM) Combines incoming requests into running batches to maximize GPU utilization and throughput

PagedAttention (vLLM) Manages KV-cache memory efficiently for high concurrency

Prefix caching Reuses computation for shared prompt prefixes, such as the same long document or system prompt, and lowers time to first token. Agent Platform's vLLM adds host-memory prefix caching

• Speculative decoding A fast draft proposes tokens that the main model verifies, lowering time per output token
• Tensor parallelism Splits a large model across GPUs so it fits and serves faster
• KV-cache quantization (for example, FP8) Shrinks KV-cache memory, allowing larger batches. Can reduce accuracy
• Weight quantization (AWQ, GPTQ) Fits models on fewer or smaller GPUs

For Google's managed Gemini models, the equivalent levers are model tier, context caching, output limits, thinking level, and consumption options (Chapter 4).

Training-Side Tuning for Production Retraining

Retraining pipelines run repeatedly, so efficiency compounds:

• Lever Benefit
• Mixed precision training Faster steps and lower memory on modern GPUs and TPUs
• Efficient input pipelines (sharded files, parallel reads, prefetch) Keeps accelerators busy

Warm starting from the previous model's weights or checkpoint Fewer epochs for incremental retraining on new data

• Right-sized distributed training and Reduction Server Shorter wall-clock time without wasted accelerators
• Spot VMs with checkpointing Lower cost for fault-tolerant retraining
• Pipeline caching (Chapter 15) Skips unchanged steps such as preprocessing on unchanged data
• Hyperparameter search reuse Narrow the search space from previous studies instead of starting fresh
• Trading Accuracy for Latency and Cost

Use a structured comparison:

Candidate AUC p95 latency Cost / 1K predictions Meets SLO?

• Baseline DNN on CPU 0.912 140 ms $0.31 No (latency)
• Same DNN on L4 with batching 0.912 38 ms $0.22 No (cost)
• Distilled student on CPU 0.906 22 ms $0.08 Yes
• int8-quantized DNN on L4 0.909 25 ms $0.15 Yes

Several options may meet the SLOs. Choose based on the accuracy the business needs and on operational simplicity. Here, the distilled student gives the biggest cost saving for a 0.006 AUC drop.

Guarding Against Regressions

Optimizations can quietly change model behavior. Build them into the release process:

Keep the unoptimized model as a reference, and compare predictions on a golden dataset. Flag rows where outputs differ beyond tolerance.

Evaluate per slice (region, device, language) as well as overall, because quantization errors can concentrate in rare segments.

Roll out optimized models with a canary (Chapter 13), and watch both latency and prediction distributions.

Record the optimization settings (precision, batch configuration, runtime image) as part of the model version in Model Registry.

Exam Traps

Buying more GPUs when profiling shows preprocessing or feature lookups dominate latency.

Deploying a quantized model without re-evaluating accuracy , including per-slice results.

Optimizing average latency when the SLO is p95 or p99.

Forgetting that batching adds queueing delay. Tune batch timeouts for your latency budget.