Study guide

9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools

Key Takeaways Explain This Topic AI

A serverless training job (CustomJob) runs your code from either a Python training application on a prebuilt container or a custom container image.

In a distributed job, workerPoolSpecs[0] is the primary replica with replicaCount 1, [1] holds workers, [2] holds parameter servers or Reduction Server, and [3] holds evaluators.

Agent Platform sets AIP_MODEL_DIR, AIP_CHECKPOINT_DIR, and AIP_TENSORBOARD_LOG_DIR when you set a base output directory, so code can write artifacts without hard-coded paths.

A persistent resource keeps a training cluster running between jobs, which assures capacity and cuts startup time but bills the entire time it runs.

Spot VMs cut serverless training cost for fault-tolerant jobs, and Dynamic Workload Scheduler's FLEX_START strategy waits for requested GPUs to become available.

Last updated: September 2026

The exam guide lists model training using different SDKs (for example, Agent Platform custom training, Kubeflow on GKE, Agent Platform AutoML, and Tabular Workflows) and organizing training on Google Cloud . This section covers custom training. Section 9.3 covers Kubeflow on GKE, AutoML, and Tabular Workflows.

Three Custom Training Resources

Resource What it does Use when

CustomJob (serverless training job) Runs your training code on the machines you specify Standard training runs, single node or distributed

HyperparameterTuningJob Runs many trials of a CustomJob with different hyperparameters Searching for the best configuration (Chapter 10)

TrainingPipeline Orchestrates a training job plus steps such as loading a managed dataset and uploading the model to Model Registry You want dataset handling and model upload built in

Packaging Your Code

Option How Choose when

Python training application + prebuilt container Package code as a Python source distribution, and pick a Google prebuilt training image for TensorFlow, PyTorch, scikit-learn, or XGBoost Standard framework versions. Fastest path with Google-maintained dependencies

Custom container Build a Docker image with your code and dependencies, and push it to Artifact Registry Unusual libraries, system packages, non-Python code, exact reproducibility, or a framework version without a prebuilt image

Autopackaging gcloud ai custom-jobs create builds an image from local code, pushes it, and creates the job in one command Quick iteration from a laptop or notebook

Prebuilt containers follow a framework support policy with end-of-patch and end-of-availability dates. Pin a supported version and plan upgrades. Images must be referenced with their latest tag, which gets the most recent patch.

Anatomy of a CustomJob

A CustomJob includes:

One or more worker pools ( workerPoolSpecs ), each with a machine type , optional accelerators (type and count), disk , replica count , and either a PythonPackageSpec or a ContainerSpec .

Scheduling: timeout, restart behavior, and a strategy such as Spot or flex-start.

A service account , network (VPC peering or Private Service Connect interface), and environment variables .

A base output directory , from which Agent Platform sets:

  • AIP_MODEL_DIR : where to export the final model artifacts
  • AIP_CHECKPOINT_DIR : where to write checkpoints
  • AIP_TENSORBOARD_LOG_DIR : where TensorBoard logs go

Training code must export model artifacts to Cloud Storage . Local disk disappears when the job ends. If you want to upload the model to Model Registry for Agent Platform serving, export it in a format the serving containers support, such as TensorFlow SavedModel, a scikit-learn or XGBoost file, or a PyTorch model archive for a PyTorch serving container.

Worker pool order for distributed training

  • Position Role
  • workerPoolSpecs[0] Primary (chief) replica. Required. replicaCount must be 1
  • workerPoolSpecs[1] Workers
  • workerPoolSpecs[2] Parameter servers or Reduction Server
  • workerPoolSpecs[3] Evaluators (TensorFlow usually expects at most one)

Leave a position empty ( {} ) to skip it. Code finds its role through the CLUSTER_SPEC or TF_CONFIG environment variables. Distributed strategies are covered in Chapter 11.

Identity, Networking, and Security

Assign a custom service account with least privilege instead of relying on the default service agent's project-wide bucket access.

Use VPC Network Peering or a Private Service Connect interface when training code must reach private resources such as databases or on-premises systems.

Use VPC Service Controls and CMEK as required (Chapter 18).

Store secrets in Secret Manager , not in container images or code.

Capacity and Cost Options

Option What it does Trade-off

On-demand serverless training Provision resources per job and release them afterward Startup time. No guaranteed capacity

Spot VMs Big discounts on spare capacity Preemption. The job fails with a STOCKOUT error and Compute Engine retries up to six times, so code must checkpoint and resume. Not supported for TPU Pods

Dynamic Workload Scheduler ( scheduling.strategy = FLEX_START ) Waits until requested GPUs (L4, A100, H100, H200, B200) are available, and starts all nodes together Delayed start. Jobs are limited to 7 days, and all worker pools need the same machine configuration

Reservations Use Compute Engine reservations for guaranteed capacity Pay for the reservation

Persistent resource A long-running cluster that stays up between jobs Billed the entire time it runs , even when idle. Best for many short jobs, repeated jobs that benefit from image and data caching, or critical peak periods

Managed Training Clusters Reserved, dedicated high-end accelerator clusters Highest commitment. For large, long-running training

Making Jobs Resilient

Long or preemptible jobs fail in predictable ways. Build for it:

Checkpoint regularly to AIP_CHECKPOINT_DIR (Cloud Storage) and resume from the latest checkpoint at startup.

Make steps idempotent , so rerunning a partially finished job doesn't duplicate outputs.

Set a sensible timeout , so hung jobs don't burn accelerator hours.

Log to Cloud Logging and stream metrics to TensorBoard, so failures can be diagnosed after the job ends (Chapter 10).

Use the interactive shell on a running job when you need to inspect a live replica.

Worked Scenario

A team trains a PyTorch recommendation model nightly on 8 A100 GPUs. Jobs sometimes wait hours for GPUs, and the model occasionally needs an unusual CUDA extension.

Build a custom container with the extension and push it to Artifact Registry, since no prebuilt image includes it.

Enable Dynamic Workload Scheduler ( FLEX_START ) so the job starts when all A100s are available together, or buy a reservation if the nightly deadline is strict.

Write checkpoints to AIP_CHECKPOINT_DIR so a restarted job resumes instead of starting over.

Run the job as an Agent Platform Pipelines step that uploads the model to Model Registry and evaluates it (Chapter 15).