ExamFinch.com My Study

Study guide

12.3 Prebuilt & Custom Serving Containers

Practice this exam

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

12.3 Prebuilt & Custom Serving Containers

Key Takeaways Explain This Topic AI

Prebuilt inference containers serve TensorFlow SavedModels, PyTorch archives named model.mar, scikit-learn files named model.joblib, and XGBoost files named model.bst or model.joblib.

A custom serving container must run an HTTP server that answers health checks with 200 OK within 10 seconds and accepts prediction requests with an instances array and optional parameters.

Agent Platform tells custom containers where to listen and route through AIP_HTTP_PORT (default 8080), AIP_HEALTH_ROUTE, and AIP_PREDICT_ROUTE, and where artifacts are through AIP_STORAGE_URI.

Custom prediction routines (CPR) let you add preprocessing and postprocessing in a Predictor class without writing a model server or Dockerfile.

The optimized TensorFlow runtime is backward compatible with prebuilt TensorFlow Serving containers and can lower serving cost and latency without code changes.

Last updated: September 2026

The exam guide lists packaging and serving models from different frameworks (for example, PyTorch and XGBoost) using prebuilt and custom containers . Many questions reduce to: does a prebuilt container support this model as-is, or do I need custom code?

Prebuilt Inference Containers

Google maintains HTTP inference servers per framework and version, stored in Artifact Registry. Like training images, they follow a support schedule with end-of-patch and end-of-availability dates, and you must use the latest tag.

Framework Artifact requirement Serving technology

TensorFlow A SavedModel directory ( saved_model.pb plus variables). Optional config/batching_parameters_config enables server-side request batching TensorFlow Serving, or the optimized TensorFlow runtime

  • PyTorch A TorchServe archive named exactly model.mar (model-name set to model ) TorchServe
  • scikit-learn A file named exactly model.joblib (estimators or pipelines) Google's inference server
  • XGBoost A file named exactly model.bst (Booster save_model ) or model.joblib Google's inference server

Upload the artifact directory to Cloud Storage, register the Model with the matching prebuilt image, and deploy. The framework version in the image should match the training version.

Optimized TensorFlow runtime

The optimized TensorFlow runtime is a drop-in serving image that uses Google's proprietary and open-source technologies to cut inference cost and latency compared with open-source TensorFlow Serving containers. It's backward compatible , so you switch images with no code changes .

When Prebuilt Isn't Enough

Need Solution

Preprocessing or postprocessing around the model (tokenize text, scale features, map class IDs to labels) Custom prediction routine (CPR)

  • A framework or version with no prebuilt image, or special system libraries Custom container
  • Non-JSON request formats or custom HTTP behavior Custom container, or CPR with a custom Handler
  • Several models or frameworks in one GPU server with dynamic batching NVIDIA Triton container
  • LLM serving with high throughput Prebuilt vLLM , Hex-LLM, SGLang, TGI, or TensorRT-LLM containers
  • Custom Container Contract

A custom container must run an HTTP server that follows these rules:

  • Requirement Detail
  • Port Listen on AIP_HTTP_PORT (default 8080 )

Health route AIP_HEALTH_ROUTE , set from containerSpec.healthRoute . Return 200 OK within 10 seconds when ready to serve. Return anything else (for example, 503) while loading. After an unhealthy result, up to 3 more checks run at 10-second intervals

  • Predict route AIP_PREDICT_ROUTE , set from containerSpec.predictRoute
  • Request body {"instances": [...], "parameters": {...}} . parameters is optional
  • Response body {"predictions": [...]} , in the same order as instances
  • Model artifacts Read from AIP_STORAGE_URI , which points to a copy of your artifact directory

Size limits Requests and responses up to 1.5 MB on shared public endpoints. Dedicated and private endpoints allow up to 10 MB

Build the image, push it to Artifact Registry , upload a Model that references the image and routes, then deploy.

Custom Prediction Routines (CPR)

CPR gives you custom logic without writing a web server or Dockerfile :

Implement a Predictor with load (read artifacts), preprocess , predict , and postprocess .

Optionally implement a Handler for raw request access, custom headers, or non-JSON payloads. Google recommends keeping web logic in the Handler and ML logic in the Predictor.

Build the container with the SDK: LocalModel.build_cpr_model(src_dir, image_uri, predictor=..., handler=..., requirements_path=...) .

Test locally with deploy_to_local_endpoint(...) before pushing, which speeds up iteration.

Push, upload, and deploy like any custom container. class ChurnPredictor(Predictor):

def load(self, artifacts_uri):

  • prediction_utils.download_model_artifacts(artifacts_uri)
  • self._model = joblib.load("model.joblib")
  • self._scaler = joblib.load("scaler.joblib")

def preprocess(self, request):

return self._scaler.transform(request["instances"])

def predict(self, instances):

return self._model.predict_proba(instances)[:, 1]

def postprocess(self, scores):

return {"predictions": [{"churn_prob": float(s), "tier": "high" if s > 0.7 else "low"} for s in scores]}

Putting preprocessing in the serving container keeps training and serving transformations consistent when they share code (Chapter 15).

Testing Containers Before Deployment

Run locally with the same environment variables Agent Platform sets ( AIP_HTTP_PORT , AIP_HEALTH_ROUTE , AIP_PREDICT_ROUTE , AIP_STORAGE_URI ), or use CPR's deploy_to_local_endpoint .

Send sample requests in the exact instances format your clients will use, including edge cases such as missing fields.

Measure startup time so health checks and minimum replicas are set realistically.

Load-test a staging endpoint to choose machine type and autoscaling targets (Chapter 14).

Scan the image in Artifact Registry for vulnerabilities, and pin base image versions.

Choosing a Container Strategy

  • Scenario Choice
  • Standard XGBoost model saved as model.bst Prebuilt XGBoost container
  • TensorFlow model where cost and latency matter Optimized TensorFlow runtime
  • scikit-learn model needing input normalization and label mapping CPR with a custom Predictor

Model in a framework with no prebuilt image (for example, a custom C++ runtime) Custom container meeting the HTTP contract

  • Several TensorFlow, PyTorch, and ONNX models sharing one GPU with dynamic batching Triton container
  • Serving an open LLM with high throughput Prebuilt vLLM container through Model Garden
  • Exam Traps

Naming artifacts incorrectly, such as model.pkl for a scikit-learn prebuilt container that expects model.joblib , or a PyTorch archive not named model.mar .

Returning 200 OK on the health route before the model finishes loading. Requests then fail during startup.

Choosing a full custom container when a prebuilt container plus CPR would do.

Sending large payloads, such as base64 images, to a shared public endpoint over the 1.5 MB limit. Use dedicated or private endpoints, or pass Cloud Storage URIs instead.