ExamFinch.com My Study

Study guide

12.1 Batch vs. Online Inference on Agent Platform

Practice this exam

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

12.1 Batch vs. Online Inference on Agent Platform

Key Takeaways Explain This Topic AI

Batch inference reads from Cloud Storage (JSON Lines, TFRecord, CSV, or file lists) or BigQuery and writes results to Cloud Storage or BigQuery without deploying the model to an endpoint.

Batch inference jobs don't autoscale: they use starting_replica_count and ignore max_replica_count.

Google recommends sizing batch jobs to run at least 10 minutes on CPUs and at least 20 minutes on GPUs, because replicas take about 5 and 10 minutes to start.

For batch jobs, scaling out with more replicas of the smallest workable machine type improves throughput more predictably than using larger machines.

Batch inference input, output, and the model must be in the same region or multi-region.

Last updated: September 2026

The exam guide lists deploying models for batch and online inference using appropriate services (for example, Agent Platform, Model Garden, Cloud Run, and GKE) . This section covers Agent Platform Inference (formerly Vertex AI Prediction). Section 12.2 covers Model Garden, Cloud Run, and GKE.

Batch vs. Online at a Glance

  • Dimension Batch inference Online inference
  • Pattern Asynchronous job over a known dataset Synchronous request → response
  • Endpoint needed No. Request batch inference directly from the model Yes. Deploy the model to an endpoint
  • Latency Minutes to hours for the whole job Milliseconds to seconds per request
  • Scaling Fixed starting replica count for the job Autoscaling between minimum and maximum replicas
  • Cost profile Pay only while the job runs Pay for deployed replicas the whole time the model is deployed

Typical uses Nightly scoring, backfills, weekly propensity lists, offline evaluation Fraud checks at checkout, recommendations on page load, chat

  • Batch Inference
  • Inputs and outputs
  • Input format Notes
  • JSON Lines One instance per line in Cloud Storage
  • TFRecord Instances stored as TFRecords
  • CSV Header row plus instances
  • File list A list of Cloud Storage file URIs, such as images
  • BigQuery A table as input. For registered BigQuery ML models, set the instance type to object

Output goes to Cloud Storage or BigQuery . Agent Platform converts inputs to JSON instances before sending them to the container. For PyTorch prebuilt containers , each instance is wrapped in a data field because TorchServe's default handlers expect it. You can also filter or transform input fields (for example, exclude an ID column from the features while keeping it in the output).

Rules and limits

Same location: input, output, and model must be in the same region or multi-region.

No autoscaling: the data is partitioned across replicas at the start. starting_replica_count is used and max_replica_count is ignored.

Partitioning: Agent Platform automatically partitions BigQuery, file list, and JSON Lines input. It doesn't partition CSV, so Google advises against CSV for throughput-sensitive jobs. For TFRecord, split the data into at least as many files as replicas and pass a wildcard URI.

Model load timeout: loading a model for batch inference times out after about 40 minutes. Very large models may need a smaller variant.

Custom service accounts apply to the model server, not the batch client that reads and writes Cloud Storage and BigQuery.

Monitoring: Model Monitoring can analyze batch inference inputs for drift (Chapter 19).

Sizing replicas

Google's guidance:

Prefer more replicas of a small machine type over bigger machines. Throughput scales more linearly and predictably.

Aim for jobs of at least 10 minutes on CPUs, because billing is per replica node hour and each replica takes about 5 minutes to start. GPU machines take about 10 minutes to start, so target at least 20 minutes .

Rough starting points: tens of replicas for thousands of instances, hundreds for millions.

Estimate: replicas ≈ N / (T × (60 / Tb)) , where N is the number of batches, T is the target minutes, and Tb is seconds per batch.

Worked example: 1,000,000 instances at batch size 100 is 10,000 batches. With a 10-minute target and 1 second per batch, 10,000 / (10 × 60) ≈ 16.7, so about 17 replicas .

Other batch options

  • Option Best for
  • BigQuery ML ML.PREDICT Models living in BigQuery, SQL consumers
  • Gemini batch inference Large prompt backlogs at a discount (Chapter 4)
  • Dataflow RunInference Batch or streaming pipelines that combine preprocessing and scoring
  • Online Inference
  • Flow

Upload the model to Model Registry with its serving container (prebuilt or custom).

Create an endpoint (dedicated public, shared public, Private Service Connect, or private; Chapter 14).

Deploy the model to the endpoint with a machine type, optional accelerators, and minimum and maximum replica counts .

Send requests . The request body is {"instances": [...], "parameters": {...}} , and the response is {"predictions": [...]} .

APIs

  • API Use
  • predict Standard JSON instances and predictions
  • rawPredict Send an arbitrary HTTP payload to the container, such as a non-standard format
  • streamRawPredict Streaming responses (dedicated and PSC endpoints), which suits gen AI
  • explain Predictions plus feature attributions, where configured
  • What drives online latency
  • Model size and hardware (Chapter 14)
  • Feature retrieval before inference (Feature Store online serving)
  • Pre- and postprocessing inside the container
  • Network path (dedicated and private endpoints optimize it)
  • Cold capacity: set minimum replicas so traffic never waits for new replicas to start
  • Hybrid Patterns

Pattern Example

Precompute + lookup Batch-score product affinity nightly, store results in a low-latency store, and serve lookups online

Candidate generation + online ranking A batch job retrieves top 500 items per user. An online model re-ranks them with session context

  • Online with batch fallback If the endpoint is unavailable, serve yesterday's batch score
  • Explanations and Monitoring for Each Mode

Batch: you can request feature attributions in the batch job when the model is configured for explanations (keep the Explainable AI deprecation in mind, Chapter 18). Model Monitoring can analyze batch inputs for drift against training data.

Online: enable request-response logging to BigQuery on the endpoint so predictions can be joined with later ground truth and monitored for skew and drift (Chapter 19).

Decision Checklist

Are all inputs available before the prediction is needed? → Batch is possible.

Is a fresh prediction needed within seconds of an event? → Online or streaming .

Is traffic sparse and spiky? → Batch, or online with careful minimum replicas and scale-down.

Is cost the main constraint and a delay acceptable? → Batch .