3.8 Troubleshooting ML Model Training Failures

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

3.8 Troubleshooting ML Model Training Failures

Key Takeaways Explain This Topic AI

Read the failure class first: out-of-memory, permission denied, quota exceeded, container crash, and non-convergence each have a distinct diagnostic path.

CUDA out-of-memory is resolved by reducing effective batch size, gradient accumulation, mixed precision, gradient checkpointing, or a larger accelerator — not by adding more workers.

Loss that becomes NaN points to an exploding gradient, a too-high learning rate, a log of zero, or unnormalized inputs; gradient clipping and a lower learning rate are first responses.

A job that fails only at scale is usually a quota, permission, or shared-filesystem contention problem rather than a modelling problem.

Checkpointing to Cloud Storage is what makes a long training job survivable, and it is mandatory when using preemptible or Spot resources.

Last updated: September 2026

• 3.8 Troubleshooting ML Model Training Failures
• Blueprint reference: Section 3.2, "Troubleshooting ML model training failures."

Exam questions in this area give you a symptom and expect a diagnosis. The skill is classifying the symptom correctly before proposing a fix, because the wrong class leads to a plausible answer that does nothing.

Classify the Failure First

Symptom Class First checks

Job fails immediately, no training logs Container / entrypoint Image builds and runs locally? Entrypoint correct? Args parsed?

• PermissionDenied on a bucket or table IAM Does the job's service account have the role? Not your user account
• ResourceExhausted before any step Quota Accelerator quota in that region; request an increase or change region
• CUDA out of memory mid-training Device memory Batch size, model size, activation memory

Job runs but accelerator utilization is low Input pipeline Small files, cross-region reads, insufficient prefetch

Loss becomes NaN or Inf Numerical Learning rate, gradient explosion, log of zero, unnormalized inputs

Loss plateaus far above expectation Optimization / data Learning rate too low, label problems, insufficient capacity

• Training loss falls, validation rises Overfitting Regularization, augmentation, early stopping
• Works on one worker, fails on many Distributed setup Cluster spec, ports, initialization, quota
• Job dies partway with no error Preemption Spot/preemptible reclaim; needs checkpointing
• Out-of-Memory on the Accelerator

The most common hard failure. Device memory holds parameters, gradients, optimizer state, and — usually dominant — activations , which scale with batch size.

Remedies in order of preference:

Reduce per-device batch size. Immediate and always available.

Gradient accumulation. Keep the effective batch size by accumulating gradients over several micro-batches before the optimizer step. Preserves convergence behaviour at a smaller memory footprint.

Mixed precision. Store activations in bf16/fp16 while keeping a master copy of weights in fp32. Roughly halves activation memory and usually speeds training on modern accelerators.

Gradient checkpointing. Recompute activations during the backward pass instead of storing them — trades compute for memory, often 30–40% slower but a large memory saving.

Model parallelism / sharding. When the model itself does not fit, split it across devices (see the distributed training section).

A larger-memory accelerator.

The classic wrong answer: "add more workers." Data-parallel workers each hold a full copy of the model, so adding workers does not reduce per-device memory at all.

NaN and Divergence

When the loss becomes NaN or Inf:

Lower the learning rate and add linear warmup . A large initial learning rate on a fine-tuning run is the single most common cause.

Clip gradients by global norm. Standard practice for recurrent and transformer models.

Check for log(0) and division by zero in a custom loss. Add a small epsilon.

Normalize inputs. Unscaled features with wildly different magnitudes destabilize the first steps.

Check the data for NaNs. A single NaN in a feature column propagates through every gradient.

In mixed precision, verify loss scaling is enabled; fp16 gradients underflow without it.

Underfitting Versus Overfitting

• Underfitting Overfitting
• Signature Training loss high and flat Training loss low, validation loss rising

Causes Too little capacity, learning rate too low, over-regularized, broken features Too much capacity, too little data, no regularization, leakage in reverse

Fixes Larger model, higher learning rate, train longer, better features, less regularization Regularization (L1/L2, dropout), augmentation, early stopping, more data, smaller model

A validation loss that is lower than training loss usually means dropout or augmentation is active during training and disabled during evaluation, which is expected — not a bug.

Failures That Only Appear at Scale

Quota. Accelerator quota is per region and per type. A job that runs with 2 GPUs and fails with 16 is usually quota, and the error will say ResourceExhausted before any training step.

Permissions. The training job runs as a service account , not as the submitting user. A notebook that reads a bucket successfully proves nothing about whether the job's service account can.

Distributed initialization. Every worker must agree on the cluster specification; a mismatched worker count, an unreachable chief, or a blocked port hangs the job at initialization with no useful loss output.

Stragglers. One slow worker throttles a synchronous all-reduce step. Uneven shard sizes are a frequent cause.

Checkpointing: The Non-Negotiable

Long training runs fail. Machines preempt, quotas throttle, transient errors happen. Checkpoint to Cloud Storage at regular step or time intervals, saving model weights, optimizer state, and the step counter, then resume from the latest checkpoint on restart.

This is mandatory rather than optional when using Spot or preemptible resources, whose entire value proposition — substantially lower cost — is only realizable if reclamation costs you minutes rather than the whole run. # Resume-safe: the job restarts into the same state it left

ckpt_dir = os.environ["AIP_CHECKPOINT_DIR"] # provided by the training service

if latest := find_latest_checkpoint(ckpt_dir):

• model, optimizer, start_step = load_checkpoint(latest)
• Profiling Before Guessing

When a job runs but is slow rather than failing, stop diagnosing and start profiling. The tooling — Cloud Logging, Cloud Monitoring, the TensorBoard profiler, and Cloud Profiler — and the read-the-trace workflow are covered in Section 6.4, Operational Troubleshooting, Performance Profiling and Cost Optimization . The rule that belongs in a failure triage is simply that slow and failed are different classes: a job that completes with poor utilization is a performance investigation, while a job that terminates is one of the failure classes tabulated above, and applying the wrong playbook wastes the cycle.

Exam Traps

Adding workers to fix out-of-memory. Data parallelism replicates the model.

Checking your own IAM instead of the job's service account.

Increasing the learning rate when the loss is already diverging.

Spot resources without checkpointing.

Guessing at slowness instead of profiling.