Study guide
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE
Google Cloud Professional Machine Learning Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap
2 Section 1: Architecting Low-Code AI Solutions (~13%)
1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability
3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)
2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance
4 Section 3: Scaling Prototypes into ML Models (~21%)
3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization
5 Section 4: Serving and Scaling Models (~20%)
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput
6 Section 5: Automating and Orchestrating ML Pipelines (~18%)
5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
7 Section 6: Monitoring AI Solutions (~13%)
6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE
Key Takeaways Explain This Topic AI
Vertex AI Online Prediction provides synchronous, low-latency (<50ms) inference via managed Endpoints with autoscaling (min_replica_count, max_replica_count), dedicated CPU/GPU hardware, and private networking via VPC Peering or Private Service Connect.
Vertex AI Batch Prediction handles asynchronous, high-throughput offline scoring across petabyte-scale Cloud Storage (JSONL, CSV, TFRecord) and BigQuery datasets, automatically provisioning distributed workers and scaling to zero upon job completion.
Custom Prediction Routines (CPR) allow ML engineers to combine custom pre/post-processing Python code with pre-built Google serving containers without having to build and maintain full Docker containers from scratch.
Custom Serving Containers must implement an HTTP web server listening on port 8080 (or AIP_HTTP_PORT) exposing a health check endpoint (GET /health) and a prediction endpoint (POST /predict), supporting REST or high-throughput gRPC protocols.
Multi-model endpoints allow deploying multiple model artifacts to a single shared endpoint, optimizing compute utilization and reducing idle hardware costs at the expense of potential noisy-neighbor resource contention.
Last updated: September 2026
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE
In production machine learning systems, model serving represents the critical interface between trained mathematical artifacts and downstream business applications. Deploying models in Google Cloud requires ML engineers to balance stringent latency service level agreements (SLAs), throughput demands, infrastructure costs, and security perimeters. Google Cloud's Vertex AI Prediction suite offers two primary serving paradigms: Online Prediction for synchronous, real-time, low-latency inference, and Batch Prediction for asynchronous, high-throughput, offline scoring over massive datasets.
1. Vertex AI Online Prediction: Real-Time Serving Architecture
Vertex AI Online Prediction is engineered for interactive applications (e.g., e-commerce recommendation feeds, real-time fraud scoring, mobile app APIs) where client applications submit inference requests and expect immediate responses within milliseconds. +---------------------------------------------------------------------------------------------------------+
- | VERTEX AI ONLINE PREDICTION TOPOLOGY |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | [ Client App ] |
- | | |
- | | (HTTPS REST / gRPC) |
- | v |
- | +-------------------------------------------------------------------------------------------------+ |
- | | Vertex AI Managed Endpoint (Single DNS / IP) | |
- | | - SSL Termination & Cloud IAM Authentication | |
- | | - Traffic Routing & Splitting Engine (e.g., 90% Model v1, 10% Model v2) | |
- | +-------------------------------------------------------------------------------------------------+ |
- | | |
- | +-------------------------------+-------------------------------+ |
- | | | |
- | v v |
- | +---------------------------------+ +---------------------------------+ |
- | | Deployed Model 1 (e.g., v1) | | Deployed Model 2 (e.g., v2) | |
- | | - Worker Pool: n1-standard-4 | | - Worker Pool: g2-standard-8 | |
- | | - Accelerators: 1x NVIDIA T4 | | - Accelerators: 1x NVIDIA L4 | |
- | | - Autoscaler: [2 .. 10 nodes] | | - Autoscaler: [1 .. 4 nodes] | |
- | +---------------------------------+ +---------------------------------+ |
- +---------------------------------------------------------------------------------------------------------+
- Core Mechanics of Managed Endpoints
To serve online predictions, trained models must be registered in the Vertex AI Model Registry and subsequently deployed to a Vertex AI Endpoint :
Decoupled Architecture : An Endpoint is a persistent, managed HTTPS gateway with a stable resource URI. A single Endpoint can host multiple deployed models simultaneously, enabling seamless traffic splitting, canary rollouts, and multi-model routing without altering client client-side API configurations.
Hardware & Accelerator Co-location : When deploying a model to an Endpoint, engineers specify the dedicated machine type (e.g., n1-standard-4 , n1-highmem-8 , a2-highgpu-1g ) and optional hardware accelerators (e.g., NVIDIA T4, L4, V100, A100 GPUs). Vertex AI provisions dedicated compute instances where compute cores, memory, and accelerator cards are co-located in the same physical rack.
Autoscaling Configuration : Online endpoints support dynamic horizontal autoscaling governed by two mandatory parameters:
min_replica_count : The minimum number of compute nodes maintained at all times. Setting min_replica_count >= 1 is standard for production workloads to eliminate cold-start latency. (A standard deployment must keep min_replica_count >= 1 , so dedicated nodes remain provisioned and billed continuously. A deployment can instead be enrolled in the separate Scale To Zero feature by setting min_replica_count = 0 , at the cost of the first request after an idle period being rejected with 429 - Model is not yet ready for inference while the endpoint scales back up.)
max_replica_count : The maximum ceiling of compute nodes the autoscaler can provision under peak query-per-second (QPS) load.
Autoscaling Metric Target : The underlying horizontal pod autoscaler monitors average CPU utilization or GPU duty cycle (typically targeting 60%–80% utilization) and scales replicas up or down accordingly.
Private Endpoints and Enterprise Networking :
By default, Vertex AI Endpoints expose public HTTPS endpoints secured by Google Cloud IAM OAuth2 tokens.
For sensitive enterprise workloads requiring zero-trust network isolation, Vertex AI supports Private Endpoints deployed directly into a Virtual Private Cloud (VPC) via VPC Network Peering or Private Service Connect (PSC) . This prevents inference traffic from traversing the public Internet, reduces network hops for sub-10ms latency SLAs, and complies with strict corporate egress policies.
2. Vertex AI Batch Prediction: High-Throughput Offline Serving
When inference requests do not require immediate sub-second responses, Batch Prediction offers an asynchronous, serverless, and highly cost-optimized alternative.
Batch Prediction Execution Model
Batch prediction is optimized for processing millions or billions of records in scheduled offline pipelines (e.g., nightly churn scoring, weekly marketing affinity updates, bulk catalog embedding generation): +-----------------------+ Batch Prediction Job Spec +------------------------+
- | Input Data Source | ==================================> | Vertex AI Worker Pool |
- | (Cloud Storage JSONL/ | - Machine Type (e.g., c2-std-16) | (Serverless Autoscaling|
- | CSV/TFRecord or BQ) | - Optional GPUs (e.g., 4x L4) | Dynamic Data Shards) |
- +-----------------------+ - Dedicated / Preemptible VMs +------------------------+
- |
- | Direct Distributed Writes
- v
- +------------------------+
- | Output Destination |
- | (Cloud Storage Bucket |
- | or BigQuery Table) |
- +------------------------+
- Key Architectural Differences from Online Serving
Serverless Provisioning & Scale-to-Zero : Batch prediction jobs automatically provision temporary distributed worker clusters, shard input datasets across workers, execute inference in parallel, write results directly to the designated destination, and immediately terminate all compute resources. Users are billed strictly for the node-seconds consumed during execution.
Input and Output Formats : Supports reading directly from BigQuery tables (via the BigQuery Storage Read API) and Cloud Storage files formatted as JSON Lines ( .jsonl ), CSV, or TensorFlow TFRecords ( .tfrecord ). Outputs are written back to BigQuery tables or partitioned GCS bucket prefixes.
Preemptible and Spot VMs : To minimize compute costs for fault-tolerant offline jobs, batch prediction jobs can be configured to run on Preemptible or Spot VM instances, achieving up to 60%–91% cost savings compared to on-demand pricing.
No Persistent Endpoint Required : Batch prediction jobs run directly against a registered model artifact in the Model Registry or a GCS model directory path without deploying to an active Vertex AI Endpoint.
3. Custom Prediction Routines (CPR) vs. Custom Serving Containers
When deploying trained models, ML engineers must choose the runtime environment that handles HTTP serialization, request payload parsing, feature transformations, and tensor execution. +---------------------------------------------------------------------------------------------------------+
- | SERVING CONTAINER EXTENSION OPTIONS |
- +------------------------------------+------------------------------------+-------------------------------+
- | PRE-BUILT CONTAINERS | CUSTOM PREDICTION ROUTINES (CPR) | CUSTOM SERVING CONTAINERS |
- +------------------------------------+------------------------------------+-------------------------------+
- | * Standard TF, PyTorch, Scikit, | * Custom Python pre/post logic | * Complete control over OS, |
- | XGBoost serving images | * User implements Predictor class | web server, C++ runtimes |
- | * Fast deployment, zero Docker | * Google builds container image | * Supports ONNX, TensorRT, |
- | * Rigid input/output expectations | * Balance of speed and flexibility | Triton, C++ dependencies |
- +------------------------------------+------------------------------------+-------------------------------+
- 1. Pre-built Prediction Containers
Google Cloud provides fully managed, optimized container images for popular ML frameworks (TensorFlow Serving, PyTorch TorchServe, Scikit-learn, XGBoost). These containers require strict adherence to standard model serialization formats (e.g., TensorFlow SavedModel, Scikit model.joblib , XGBoost model.bst ) and expect JSON payloads matching standard signature definitions.
2. Custom Prediction Routines (CPR)
In real-world applications, raw client request payloads often require feature transformations (e.g., tokenizing text, scaling numerical inputs, looking up categorical embeddings) before reaching the model, and raw model logits require post-processing (e.g., softmax calibration, JSON formatting).
Custom Prediction Routines (CPR) bridge the gap between pre-built containers and custom Dockerfiles. CPR allows engineers to define a Python class extending the Predictor interface with custom load() , preprocess() , predict() , and postprocess() methods: # Custom Prediction Routine (CPR) Implementation
- import joblib
- import numpy as np
- from google.cloud.aiplatform.prediction.predictor import Predictor
class CustomChurnPredictor(Predictor):
def __init__(self):
- self._model = None
- self._scaler = None
def load(self, artifacts_uri: str) -> None:
- """Loads model weights and preprocessing artifacts from Cloud Storage."""
- self._model = joblib.load(f"{artifacts_uri}/model.joblib")
- self._scaler = joblib.load(f"{artifacts_uri}/scaler.joblib")
def preprocess(self, prediction_input: dict) -> np.ndarray:
- """Parses raw JSON payload and applies feature normalization."""
- raw_instances = prediction_input["instances"]
- features = np.array([[item["tenure"], item["monthly_charges"]] for item in raw_instances])
- scaled_features = self._scaler.transform(features)
- return scaled_features
def predict(self, instances: np.ndarray) -> np.ndarray:
- """Executes model inference."""
- return self._model.predict_proba(instances)
def postprocess(self, prediction_results: np.ndarray) -> dict:
- """Formats probabilities into client-ready response payload."""
- probabilities = prediction_results[:, 1].tolist()
return {"predictions": [{"churn_probability": p, "risk_level": "HIGH" if p > 0.7 else "LOW"} for p in probabilities]}
Vertex AI SDK compiles this CPR class into an optimized container image automatically, pushing it to Artifact Registry without requiring manual Dockerfile maintenance.
3. Custom Serving Containers: The HTTP Contract Specification
For complex architectures requiring non-Python runtimes, specialized C++ libraries, NVIDIA Triton Inference Server, TensorRT optimization, or gRPC streaming, engineers build Custom Serving Containers . To run reliably on Vertex AI Prediction, the custom container must strictly adhere to the Vertex AI Serving HTTP Contract :
Listening Port : The web server must listen for HTTP requests on 0.0.0.0 at the port defined by the AIP_HTTP_PORT environment variable (defaults to port 8080 ).
Health Check Endpoint ( GET /health or GET /v1/models/{model_name} ) :
Vertex AI routinely sends HTTP GET requests to verify container responsiveness.
The server must return an HTTP status code 200 OK with an empty or diagnostic body.
If the container fails to return 200 OK within the configured timeout (typically 4–10 seconds), Vertex AI restarts the container replica.
Prediction Endpoint ( POST /predict or POST /v1/models/{model_name}:predict ) :
Receives client prediction requests containing a JSON body structured with an "instances" array.
Must return an HTTP status code 200 OK containing a JSON body with a "predictions" array.
Environment Variables Injected by Vertex AI :
AIP_STORAGE_URI : The Cloud Storage path where the model artifacts reside (e.g., gs://my-bucket/models/v1 ).
AIP_HTTP_PORT : The network port on which the web server must bind (e.g., 8080 ).
AIP_HEALTH_ROUTE : The HTTP path for health checks (e.g., /health ).
AIP_PREDICT_ROUTE : The HTTP path for prediction requests (e.g., /predict ).
Important
Payload Size Constraints : Vertex AI Online Prediction enforces a maximum request payload size of 1.5 MB for standard REST predictions and 10 MB for Vertex AI standard RPC/gRPC calls. For payloads exceeding these limits (e.g., high-resolution video streams, multi-gigabyte point clouds), client applications must upload the raw data to Cloud Storage and pass the GCS URI in the prediction request, or utilize Batch Prediction.
4. Multi-Model Endpoints vs. Dedicated Endpoints
When managing a fleet of production models, ML architects must choose between deploying models to separate dedicated endpoints or co-locating them on a single multi-model endpoint:
Architectural Attribute Dedicated Endpoints (One Model Per Endpoint) Multi-Model Shared Endpoints (Multiple Models Per Endpoint)
Resource Isolation Complete hardware isolation; zero noisy-neighbor risk. Shared CPU/GPU memory; risk of resource contention.
Cost Profile Higher baseline cost (each endpoint incurs min_replica_count hardware fees). Highly cost-effective for long-tail, low-QPS models sharing idle compute.
Autoscaling Mechanics Scaled independently based on model-specific traffic spikes. Shared autoscaler scales all replicas based on aggregate endpoint load.
Failure Domain Isolated: a memory leak in Model A does not impact Model B. Blast radius includes all models sharing the container/instance.
Recommended Use Case Tier-1 mission-critical models with high QPS and strict latency SLAs. Multi-tenant SaaS with hundreds of customer-specific lightweight models.
5. Architectural Decision Matrix: GCP Model Serving Platforms
Selecting the optimal serving infrastructure across Google Cloud Platform is heavily tested on the Professional ML Engineer certification:
Serving Platform Inference Type Serving Latency Scaling Characteristics Best Suited For
Vertex AI Online Prediction Synchronous REST / gRPC Ultra-low (<20–50ms) Managed node autoscaling ( min >= 1 ); dedicated GPU/TPU co-location Real-time APIs, mobile apps, fraud scoring, low-latency microservices
Vertex AI Batch Prediction Asynchronous Offline Minutes to Hours Serverless distributed workers; scales to zero automatically Scheduled bulk scoring, catalog indexing, nightly risk scoring on GCS/BigQuery
Custom Prediction Routines (CPR) Synchronous Online Low (<30–60ms) Managed node autoscaling ( min >= 1 ); custom Python pre/post logic Models requiring custom feature scalers/tokenizers without Docker complexity
Cloud Run (with GPU) Synchronous Microservices Low (<50–100ms) True scale-to-zero serverless containers; fast cold starts Lightweight models, sporadic traffic spikes, microservice integration
BigQuery ML ( ML.PREDICT ) Asynchronous SQL Batch Seconds to Minutes Serverless analytical queries scaled by BigQuery Dremel slots In-database batch scoring on existing BigQuery data warehouses