Study guide

6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization

Google Cloud Professional Machine Learning Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap

2 Section 1: Architecting Low-Code AI Solutions (~13%)

1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability

3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

4 Section 3: Scaling Prototypes into ML Models (~21%)

3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization

5 Section 4: Serving and Scaling Models (~20%)

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput

6 Section 5: Automating and Orchestrating ML Pipelines (~18%)

5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention

7 Section 6: Monitoring AI Solutions (~13%)

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production

6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization

Key Takeaways Explain This Topic AI

Cloud Logging captures structured container logs, stdout/stderr, and pipeline events, while Cloud Monitoring tracks endpoint latency percentiles (P50/P90/P99), request counts, 4xx/5xx error rates, and GPU/CPU utilization metrics.

GPU Out-Of-Memory (OOM) errors (CUDA error: out of memory) must be differentiated from Host RAM OOM (Linux kernel OOM killer terminating process with Exit Code 137); mitigations include reducing per-replica micro-batch sizes, gradient accumulation, mixed precision (fp16/bf16), and activation checkpointing.

Vertex AI TensorBoard Profiler and Cloud Profiler identify GPU/TPU kernel underutilization by breaking down step-time into input pipeline loading (I/O starvation), host-to-device memory copy, and compute kernel execution.

Input pipeline bottlenecks are resolved in TensorFlow/PyTorch using tf.data optimizations (prefetch(AUTOTUNE), interleave, cache(), parallel mapping) and storing datasets in sharded binary formats (TFRecord, WebDataset, Parquet) on co-located regional Cloud Storage buckets.

Cost optimization strategies include using Spot/Preemptible VMs with frequent Cloud Storage checkpointing for custom training, configuring endpoint autoscaling (min_replica_count), multi-model sharing, and purchasing 1-year or 3-year Committed Use Discounts (CUDs).

Last updated: September 2026

6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization

Operating production machine learning systems on Google Cloud Platform requires deep expertise in system observability, performance profiling, and infrastructure cost governance. Training large deep neural networks and serving low-latency online models involves distributed GPU/TPU clusters, high-speed networking, specialized container runtimes, and petabyte-scale storage tiers. When failures occur or infrastructure costs escalate, ML engineers must rapidly isolate whether bottlenecks originate in the application code, the hardware accelerator layer, the data I/O pipeline, or the network topology.

1. Cloud Observability: Logging and Monitoring Architecture

Google Cloud provides two core pillars of observability for Vertex AI: Cloud Logging for granular diagnostic event traces and Cloud Monitoring for real-time quantitative telemetry. +---------------------------------------------------------------------------------------------------------+

  • | VERTEX AI OBSERVABILITY ARCHITECTURE |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | +-------------------------------------------------------------------------------------------------+ |
  • | | Vertex AI Workloads (Custom Training, Prediction Endpoints, Pipelines, TensorBoard) | |
  • | +-------------------------------------------------------------------------------------------------+ |
  • | / \ |
  • | (stdout / stderr / Container Logs) (System Metrics & Telemetry) |
  • | / \ |
  • | v v |
  • | +---------------------------------------------------+ +-------------------------------------+ |
  • | | Cloud Logging | | Cloud Monitoring | |
  • | | - Resource: `aiplatform.googleapis.com/Endpoint` | | - Latency: P50, P90, P95, P99 | |
  • | | - Filter: `severity >= ERROR` | | - Predictions / Request Count | |
  • | | - Container lifecycle & crash stack traces | | - Error Rates (4xx, 5xx codes) | |
  • | | - Custom application print / logging statements | | - GPU Duty Cycle & Memory Saturation| |
  • | +---------------------------------------------------+ +-------------------------------------+ |
  • | | | |
  • | +-------------------------+-------------------------+ |
  • | | |
  • | v |
  • | +---------------------------------+ |
  • | | Cloud Monitoring Alert Policies | |
  • | | (Email, PagerDuty, Pub/Sub) | |
  • | +---------------------------------+ |
  • +---------------------------------------------------------------------------------------------------------+
  • Cloud Logging for Vertex AI

All Vertex AI custom training containers, prediction containers, and pipeline steps automatically stream stdout and stderr to Cloud Logging. Key log query filters for rapid troubleshooting include: -- Query prediction container errors on Vertex AI Endpoints

  • resource.type="aiplatform.googleapis.com/Endpoint"
  • resource.labels.endpoint_id="1234567890"
  • severity>=ERROR
  • - Query training job worker container failure stack traces
  • resource.type="ml_job"
  • resource.labels.job_id="custom_training_job_98765"
  • textPayload=~"CUDA out of memory|Traceback|Error"
  • Cloud Monitoring Metrics & SLAs

Cloud Monitoring tracks vital quantitative indicators for deployed models:

aiplatform.googleapis.com/prediction/online/response_count : Total inference requests partitioned by HTTP response code ( 200 OK , 4xx Client Error , 5xx Server Error ).

aiplatform.googleapis.com/prediction/online/prediction_latencies : End-to-end request latency distribution. Critical for tracking P90, P95, and P99 tail latency percentiles against organizational SLAs.

container.googleapis.com/container/accelerator/duty_cycle : Percentage of time the GPU/TPU accelerator is actively processing compute kernels versus idling.

container.googleapis.com/container/accelerator/memory_used : Total VRAM allocated on the GPU card.

2. Systematic Troubleshooting of Common Failure Modes

When diagnosing broken or degraded ML systems on Google Cloud, ML engineers encounter four prevalent failure patterns: +---------------------------------------------------------------------------------------------------------+

  • | COMMON ML FAILURE MODES & ROOT CAUSES |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | FAILURE MODE | SYMPTOMS | DIAGNOSTIC & FIX |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | 1. GPU Out-Of-Memory (CUDA OOM) | RuntimeError: CUDA out of memory | Reduce per-replica batch size,|
  • | | during forward/backward training | use mixed precision (fp16), |
  • | | pass or large batch inference | enable gradient accumulation |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | 2. Host RAM OOM (Exit Code 137) | Worker process killed abruptly | Increase machine memory |
  • | | (SIGKILL / OOM killer) | (n1-highmem), stream data |
  • | | without Python traceback | with tf.data / IterableDataset|
  • +------------------------------------+------------------------------------+-------------------------------+
  • | 3. Data Pipeline I/O Starvation | Low GPU duty cycle (<30%), long | tf.data prefetch(AUTOTUNE), |
  • | | step times, GPU compute idling | store sharded TFRecords in |
  • | | waiting for data transfers | same region Cloud Storage |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | 4. Serving Cold Starts / Timeouts | HTTP 504 Gateway Timeout on first | Set min_replica_count >= 1, |
  • | | requests, container restarts | pre-load model weights into |
  • | | during sudden traffic spikes | RAM in container init routine |
  • +------------------------------------+------------------------------------+-------------------------------+
  • 1. GPU VRAM Out-Of-Memory (CUDA OOM) vs. Host RAM OOM

A vital distinction tested on the exam is differentiating between GPU memory exhaustion and Host system memory exhaustion:

CUDA VRAM OOM ( RuntimeError: CUDA out of memory ) :

Occurs when the tensors allocated for model weights, optimizer states (e.g., Adam maintains 2 FP32 states per parameter), activations, and batch gradients exceed the physical memory of the GPU card (e.g., 16 GB on NVIDIA T4, 24 GB on L4, 80 GB on A100).

Immediate Mitigations :

Reduce Micro-Batch Size : Halve the per-device batch size and configure Gradient Accumulation (accumulate gradients across N N N steps before executing optimizer.step() ) to preserve the effective global batch size.

Mixed Precision Training : Enable Automatic Mixed Precision ( torch.cuda.amp.autocast() or tf.keras.mixed_precision.set_global_policy('mixed_float16') / 'bfloat16' ), cutting activation and weight tensor memory in half.

Activation Checkpointing : Recompute intermediate activation tensors during the backward pass rather than caching them in VRAM.

Distributed Sharding : Implement Fully Sharded Data Parallel (FSDP) or DeepSpeed ZeRO to shard optimizer states and parameters across multiple GPUs.

Host RAM OOM (Exit Code 137 / SIGKILL) :

The Linux kernel Out-Of-Memory (OOM) killer forcibly terminates the container process because system RAM (host memory) is exhausted. The log will show container exit code 137 with no Python traceback.

Fix : Upgrade the Compute Engine machine type from standard to high-memory (e.g., from n1-standard-8 with 30 GB RAM to n1-highmem-8 with 52 GB RAM, or a2-highgpu-1g with 85 GB RAM), and eliminate in-memory dataset loading ( pd.read_csv() loading a 50GB file into RAM) by streaming data with tf.data or PyTorch IterableDataset .

2. Data Pipeline I/O Bottlenecks vs. Compute Saturation

When training deep neural networks, hardware accelerators should ideally operate at >85% duty cycle. If GPU utilization hovers at 15%–30%, the accelerator is starving for data —spending most of each step waiting for the host CPU to fetch, decode, and transfer batches across the PCIe bus. # Optimal High-Throughput TensorFlow Input Pipeline

import tensorflow as tf

def create_optimized_dataset(file_pattern, batch_size):

  • # 1. Parallel file matching and sharding
  • files = tf.data.Dataset.list_files(file_pattern, shuffle=True)
  • # 2. Interleave reads across multiple sharded TFRecord files
  • dataset = files.interleave(
  • lambda x: tf.data.TFRecordDataset(x, num_parallel_reads=tf.data.AUTOTUNE),
  • cycle_length=tf.data.AUTOTUNE,
  • num_parallel_calls=tf.data.AUTOTUNE
  • )
  • # 3. Cache data in RAM if dataset fits, or cache to local SSD
  • # dataset = dataset.cache()
  • # 4. Parallel feature parsing and data augmentation
  • dataset = dataset.map(parse_and_augment_fn, num_parallel_calls=tf.data.AUTOTUNE)
  • # 5. Shuffle with reasonable buffer size
  • dataset = dataset.shuffle(buffer_size=10000)
  • # 6. Batch and drop remainder for static tensor shapes on TPU/GPU
  • dataset = dataset.batch(batch_size, drop_remainder=True)
  • # 7. Prefetch batches to overlap GPU compute with CPU data prep
  • dataset = dataset.prefetch(buffer_size=tf.data.AUTOTUNE)

return dataset

Key Optimization Rules :

prefetch(tf.data.AUTOTUNE) : Decouples the producer (CPU data loading) from the consumer (GPU training step). While the GPU executes step N N N , the CPU loads and augments step N + 1 N+1 N + 1 concurrently.

Storage Co-location : Store training data in a Cloud Storage bucket located in the same Google Cloud region as the Vertex AI training worker pool (e.g., bucket in us-central1 and worker pool in us-central1 ). Cross-region reads introduce high network latency and substantial data egress charges.

Sharded Binary Formats : Consolidate millions of small image/text files into sharded TFRecord, WebDataset, or Parquet files (100 MB to 200 MB per shard) to maximize Cloud Storage sequential read throughput.

3. Profiling with Vertex AI TensorBoard & Cloud Profiler

To scientifically identify performance bottlenecks, Google Cloud provides dedicated profiling tools: +---------------------------------------------------------------------------------------------------------+

  • | VERTEX AI TENSORBOARD STEP-TIME BREAKDOWN |
  • +---------------------------------------------------------------------------------------------------------+
  • | |
  • | [ STEP TIME BREAKDOWN ] |
  • | +-------------------------------------------------------------------------------------------------+ |
  • | | Input Pipeline (CPU / Disk / Network) | Host-to-Device (PCIe Transfer) | GPU Kernel Execution | |
  • | | [=========== 65% ============] | [====== 15% ======] | [==== 20% ====] | |
  • | +-------------------------------------------------------------------------------------------------+ |
  • | * DIAGNOSIS: I/O Bottleneck! Accelerators are idling 80% of step time waiting for data. |
  • | * REMEDIATION: Apply tf.data.prefetch, increase num_parallel_calls, verify GCS bucket regional locality|
  • | |
  • +---------------------------------------------------------------------------------------------------------+
  • Vertex AI TensorBoard Profiler

By attaching a TensorBoard Profiler callback to training scripts, engineers capture detailed hardware execution traces:

Overview Page : Displays overall Step-Time breakdown, highlighting percentage of time spent in Input , Compute , and Host-to-Device copying .

GPU Kernel Stats : Identifies specific tensor operations (e.g., Gemm , Conv2D , LayerNorm ) and their memory bandwidth saturation.

Trace Viewer : Displays microsecond-accurate timeline of CPU thread activity, CUDA stream kernel launches, and PCIe data transfers.

Cloud Profiler

Cloud Profiler is a continuous, low-overhead (<1% CPU/memory overhead) profiling agent that runs inside production containers. It constructs flame graphs of CPU consumption and memory allocations, allowing engineers to identify CPU-intensive loops in Custom Prediction Routines during live serving.

4. Cost Optimization Strategies for GCP ML Workloads

Machine learning infrastructure represents one of the largest compute expenditures in enterprise cloud environments. Implementing disciplined cost optimization is a core skill evaluated on the certification: +---------------------------------------------------------------------------------------------------------+

  • | GCP ML COST OPTIMIZATION PILLARS |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | SPOT / PREEMPTIBLE VMS | AUTOSCALING & SIZING | COMMITTED USE DISCOUNTS |
  • +------------------------------------+------------------------------------+-------------------------------+
  • | * Up to 91% discount on Compute | * Set min_replica_count=1 (not 10) | * 1-year (37%) or 3-year (55%)|
  • | Engine VMs and GPU accelerators | * Scale-to-zero for batch scoring | Committed Use Discounts |
  • | * Mandatory: Checkpoint to GCS | * Multi-model endpoints for long- | * Reserve steady-state baseline|
  • | every N epochs to survive preemp | tail, low-QPS models | accelerator/CPU capacity |
  • +------------------------------------+------------------------------------+-------------------------------+
  • 1. Spot and Preemptible VMs with Checkpointing

Spot VMs utilize excess Google Cloud compute capacity at discounts between 60% and 91% compared to on-demand pricing.

Preemption Handling : Spot instances can be reclaimed by Google Cloud with a 30-second warning. To safely utilize Spot VMs for custom training, ML engineers must implement automated checkpointing : # PyTorch Checkpointing to Cloud Storage for Spot VM Resilience

  • import torch
  • import os
  • from google.cloud import storage

def save_checkpoint(model, optimizer, epoch, gcs_bucket_name, checkpoint_prefix):

  • local_path = f"/tmp/checkpoint_epoch_{epoch}.pt"
  • torch.save({
  • 'epoch': epoch,
  • 'model_state_dict': model.state_dict(),
  • 'optimizer_state_dict': optimizer.state_dict(),
  • }, local_path)
  • # Upload checkpoint to persistent Cloud Storage bucket
  • client = storage.Client()
  • bucket = client.bucket(gcs_bucket_name)
  • blob = bucket.blob(f"{checkpoint_prefix}/checkpoint_epoch_{epoch}.pt")
  • blob.upload_from_filename(local_path)
  • print(f"Checkpoint successfully persisted to GCS for epoch {epoch}")

When a Spot VM is preempted and Vertex AI restarts the training job, the script inspects GCS, restores the latest checkpoint, and resumes training with zero loss of progress.

2. Right-Sizing Accelerators

Inference : Avoid over-provisioning A100 (80GB) GPUs for lightweight tabular or text models. Use NVIDIA L4 (24GB) or NVIDIA T4 (16GB), or host lightweight models on CPU machine types ( c2-standard-4 ).

Multi-Model Endpoints : Co-locate multiple low-QPS models on a single Vertex AI Endpoint to share baseline replica costs rather than maintaining separate idle endpoints with min_replica_count=1 .

Committed Use Discounts (CUDs) : For persistent, predictable baseline workloads (such as 24/7 online serving endpoints), purchase 1-year or 3-year resource-based Committed Use Discounts to secure up to 55% savings on compute and GPU resources.

5. Troubleshooting Decision Matrix

Symptom / Observed Behavior Likely Root Cause Immediate Diagnostic Action Corrective Engineering Action

CUDA out of memory during training VRAM exhaustion due to large batch size / activations Inspect GPU memory profile in TensorBoard Enable mixed precision (fp16), reduce batch size, enable gradient accumulation

Container terminated abruptly with Exit Code 137 Linux kernel Host RAM OOM killer Check Cloud Logging for system memory logs Upgrade to highmem machine type; stream data instead of in-memory caching

Low GPU utilization (<25%), high step time I/O pipeline bottleneck / GPU starvation Inspect TensorBoard Step-Time breakdown Apply tf.data.prefetch(AUTOTUNE) , interleave reads, co-locate GCS bucket region

Endpoint returns HTTP 504 Gateway Timeout on startup Container takes too long to load weights / initialize Inspect Cloud Logging endpoint startup traces Set min_replica_count >= 1 , pre-load model during container startup, optimize Docker image

Endpoint returns HTTP 413 Payload Too Large Prediction payload exceeds REST (1.5MB) or gRPC (10MB) limit Check size of incoming client JSON instances Store large files (images/audio) in Cloud Storage and pass GCS URIs in prediction payload

Loading diagram... Diagnostic Troubleshooting Flowchart for Vertex AI Machine Learning Workloads