Study guide
10.1 Troubleshooting ML Training Failures
Google Cloud Professional Machine Learning Engineer PMLE
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & the Agent Platform Map
1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map
2 Chapter 2: Low-Code ML with BigQuery ML
2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery
3 Chapter 3: AutoML & Google Cloud AI APIs
3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs
4 Chapter 4: Foundation Models in Model Garden
4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability
5 Chapter 5: Exploring, Preprocessing & Protecting Data
5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows
6 Chapter 6: Notebooks, Prototyping & Experiment Tracking
6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
7 Chapter 7: Evaluating Predictive & Generative AI Solutions
7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
8 Chapter 8: Choosing the Modeling Approach
8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements
9 Chapter 9: Training Data & Training SDKs
9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices
10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning
10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune
11 Chapter 11: Training Hardware & Distributed Training
11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism
12 Chapter 12: Deploying Models for Inference
12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers
13 Chapter 13: Model Versions, Rollouts & Inference Logic
13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing
14 Chapter 14: Scaling Online Serving
14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving
15 Chapter 15: End-to-End ML Pipelines
15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving
16 Chapter 16: Orchestration with Managed Airflow & Ray
16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads
17 Chapter 17: Automating Retraining & CI/CD/CT
17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build
18 Chapter 18: Securing AI Systems, Responsible AI & Explainability
18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform
19 Chapter 19: Monitoring AI Solutions
19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
10.1 Troubleshooting ML Training Failures
Key Takeaways Explain This Topic AI
An interactive shell on a running serverless training job lets you inspect the container, run py-spy or nvidia-smi, and check permissions with gcloud auth list.
Interactive shell access ends as soon as the job or trial finishes, and files created in the container don't persist.
GPU out-of-memory errors are usually fixed by smaller batches, gradient accumulation, mixed precision, gradient checkpointing, a larger-memory accelerator, or model-parallel sharding.
A NaN or exploding loss usually points to a learning rate that is too high, bad or unnormalized input data, or numerical overflow; gradient clipping and loss scaling help.
Training that looks too good to be true on validation data often signals label leakage or overlap between training and validation sets.
Last updated: September 2026
The exam guide lists troubleshooting ML model training failures under Section 3.2. Scenario questions describe a symptom, such as a job stuck in a pending state, a crash with exit code 137, or a loss that turns to NaN, and ask for the most likely cause or the best next step.
A Systematic Approach
Classify the failure: did the job fail to start , crash while running , or finish with a bad model ?
Read the evidence: job state and error message, Cloud Logging output from each replica, Cloud Monitoring CPU, GPU, and memory utilization, and TensorBoard loss curves.
Reproduce small: run the same container locally or on a tiny data sample with one replica.
Change one thing at a time , and record each attempt as an experiment run.
Failures Before or at Startup
Symptom Likely cause Fix
Job waits long or fails to get resources Not enough accelerator quota in the region, or temporary capacity shortage Request a quota increase, try another supported region, use Dynamic Workload Scheduler ( FLEX_START ), or use reservations
STOCKOUT error with Spot VMs Spot capacity reclaimed or unavailable Checkpointing plus automatic retries, or on-demand VMs for deadline-critical jobs
Permission denied reading data or writing output The job's service account lacks bucket or table roles, or the user lacks iam.serviceAccounts.actAs on a custom service account Grant least-privilege roles to the job's service account. Grant actAs to the submitter
Image pull failure Wrong image URI, image in a repository the service agent can't read, or an unsupported tag on a prebuilt image Fix the URI, grant Artifact Registry read access, use a supported prebuilt tag
Invalid machine configuration Accelerator type not available in the region, or an incompatible machine type and GPU combination Choose a supported combination for the region
Crashes While Running
Symptom Likely cause Fix
ModuleNotFoundError or version conflicts Dependencies differ from the local environment Pin versions in the package or container. Test the container locally with the same command
GPU out-of-memory ( CUDA out of memory ) Batch too large, sequence too long, or model too big for one device Smaller batch plus gradient accumulation , mixed precision (bfloat16 or float16), gradient checkpointing , a bigger-memory GPU, or model-parallel / sharded training (Chapter 11)
Process killed (exit code 137) The process got SIGKILL, commonly from host-memory exhaustion (for example, loading a whole dataset into RAM) Stream data instead of loading it all, use a higher-memory machine, reduce loader workers or buffer sizes
Hangs or NCCL errors in multi-node jobs Mismatched framework or NCCL versions across replicas, network setup, or a replica that crashed while others wait Use the same image for every worker pool. Check each replica's logs. Enable timeouts and retries
Crash after hours of training Preemption, a bad record late in the data, or a disk filling up Checkpoint to Cloud Storage, validate data upfront, write large outputs to Cloud Storage instead of local disk
Finishes, but the Model Is Bad
Symptom in curves or metrics Likely cause Fix
Loss becomes NaN or explodes Learning rate too high, unnormalized features, invalid values (log of 0, division by zero), float16 overflow Lower the learning rate, normalize inputs, clean data, use gradient clipping , use loss scaling with mixed precision
Loss flat, not decreasing Learning rate too low, a bug (labels out of order, frozen layers), a dead activation, or data that doesn't carry signal Check the label pipeline, try to overfit a tiny batch (it should reach near-zero loss), then raise the learning rate
Training loss falls while validation loss rises (overfitting) Model too complex for the data, or too many epochs Early stopping, regularization (L1, L2, dropout), data augmentation, more data, a simpler model
Both training and validation loss stay high (underfitting) Model too simple, weak features, too little training Larger model, better features, longer training, lower regularization
Validation metrics look unrealistically high Leakage : future information, duplicates across splits, or a feature derived from the label Rebuild splits (time-based or group-based), remove leaky features
Great offline, poor in production Training-serving skew or data drift Share preprocessing, monitor skew and drift (Chapters 15 and 19)
- Tools for Diagnosis on Agent Platform
- Interactive shell
Enable web access on a CustomJob, HyperparameterTuningJob, or custom TrainingPipeline to open a shell inside the running container . With it you can:
Run nvidia-smi to check GPU utilization and memory.
Profile Python with py-spy without changing code.
Run gcloud auth list and try bq or gcloud storage commands to confirm what the job's service account can reach.
Requirements: bash in the image (prebuilt containers have it), a supported region, and permissions such as aiplatform.customJobs.get , plus iam.serviceAccounts.actAs if a custom service account is attached. When the job ends, the shell closes immediately. To debug a crash, catch the exception, log the traceback, and sleep before exiting. That time is billed.
Logs, metrics, and profiling
Cloud Logging: per-replica stdout and stderr, with severity filters.
Cloud Monitoring: CPU, memory, and accelerator utilization. Low GPU utilization usually means an input bottleneck.
TensorBoard: loss and metric curves, and profiling of training steps. Cloud Profiler supports TensorFlow 2.4 and later on custom training.
Worked Scenario
A TensorFlow image model on one A100 crashes with GPU out-of-memory at batch size 256. The team must keep the effective batch at 256 for training stability.
Use batch size 64 with 4 gradient accumulation steps , which keeps the same effective batch and cuts memory per step.
Enable mixed precision , which roughly halves activation memory on supported GPUs.
If that's still not enough, use data parallelism across 4 GPUs at batch 64 each (Chapter 11).
Confirm the fix in TensorBoard: the loss curve should track the earlier runs.