Study guide
14.1 GKE Resources, OOM Diagnosis, and Autoscaling
Google Cloud Professional Cloud Developer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam and Compute Platform Decisions
1.1 Current Exam, Sources, and Scenario Method 1.2 Cloud Run Execution and Platform Selection 1.3 GKE Modes and Compute Engine Trade-offs
2 Chapter 2: Container Engineering and API Contracts
2.1 Stateless Microservices and Multi-stage Builds 2.2 Container Hardening, Signals, and Layer Caching 2.3 REST, gRPC, and Protobuf Contracts
3 Chapter 3: gRPC Compatibility and Geographic Routing
3.1 Schema Evolution and gRPC Routing 3.2 Geography, Network Tiers, and Load Balancers 3.3 Serverless NEGs, Sessions, and Edge Controls
4 Chapter 4: API Management and Caching
4.1 API Management & Rate Limiting with Apigee and API Gateway 4.2 Memorystore and Cache Access Patterns 4.3 Cache Invalidation, Capacity, and Failover
5 Chapter 5: Messaging, Task Dispatch, and Orchestration
5.1 Event-Driven Messaging with Cloud Pub/Sub 5.2 Task Queues & Rate Throttling with Cloud Tasks 5.3 Event Routing and Service Orchestration: Eventarc & Cloud Workflows
6 Chapter 6: Resilience and Relational Availability
6.1 Designing Resilient Systems: Circuit Breaking, Retries & Failover 6.2 Cloud SQL Availability and Recovery 6.3 Connection Budgets and Spanner Consistency
7 Chapter 7: Distributed Schemas and NoSQL Data Models
7.1 Spanner Keys, Indexes, and Transactions 7.2 Firestore Data, Indexes, and Transactions 7.3 Bigtable Row Keys and Replication
8 Chapter 8: Object Storage and Application Identity
8.1 Cloud Storage Lifecycle and Signed Access 8.2 Workload Identity and Service Authentication 8.3 Secret Delivery, Rotation, and IAP
9 Chapter 9: Security, Data Integration, and Developer Credentials
9.1 Private Communication, Key Rotation, and Vulnerability Response 9.2 AlloyDB Schemas and BigQuery Ingestion 9.3 CLI Configurations, Local ADC, and Cloud Shell
10 Chapter 10: Managed Development and Automated Tests
10.1 Cloud Workstations and the Container Inner Loop 10.2 Emulating Google Cloud Services & Database Auth Proxies 10.3 Automated Testing Strategies: Unit, Integration, and Mocking
11 Chapter 11: AI Development Tools and Build Pipelines
11.1 AI Coding Assistants, Context, MCP, and Test Review 11.2 Cloud Build Workers and Step Dependencies 11.3 Builders, Substitutions, Triggers, and IAM
12 Chapter 12: Artifact Security and Admission Policies
12.1 Artifact Management and Vulnerability Scanning with Artifact Registry 12.2 Build Provenance and Admission Concepts 12.3 Attestors, Policy Enforcement, and Break-glass
13 Chapter 13: Cloud Run and GKE Deployment Configuration
13.1 Cloud Run Deployment and Scaling Configuration 13.2 Cloud Run CPU, Private Egress, and Secrets 13.3 GKE Workload Controllers and Gateway Routing
14 Chapter 14: Kubernetes Resources, Health, and Progressive Releases
14.1 GKE Resources, OOM Diagnosis, and Autoscaling 14.2 Container Health Probes, Lifecycle Hooks & Graceful Shutdown 14.3 Progressive Delivery: Traffic Splitting, Canary, and Blue/Green Rollouts
15 Chapter 15: Source/Event Deployment and Cloud API Integration
15.1 Source Deployment, Event Receivers, and API Versions 15.2 Client Libraries, Service Enablement, and ADC 15.3 API Responses, Pagination, Caches, and Retries
16 Chapter 16: Tracing, Sampling, and Monitoring
16.1 Trace Context and OpenTelemetry Export 16.2 Custom Spans, Trace Analysis, and Sampling 16.3 Application Metrics, Health Checks & Alerting with Cloud Monitoring
17 Chapter 17: Structured Logs, Routing, and Error Management
17.1 Structured Logs, Trace Correlation, and Queries 17.2 Log Routing, Export Destinations, and Exclusions 17.3 Error Reporting Formats and Regression Handling
18 Chapter 18: Profiling, Generative AI, and Assisted Troubleshooting
18.1 Cloud Profiler Types and Flame Graph Analysis 18.2 Generative AI Integration with the Google Gen AI SDK 18.3 Evidence-Based Troubleshooting with AI Assistance
14.1 GKE Resources, OOM Diagnosis, and Autoscaling
Key Takeaways Explain This Topic AI
CPU limits throttle execution; memory exhaustion can produce OOMKilled with exit code 137.
CPU-utilization HPA compares usage to requests, not limits.
Coordinate autoscaling metrics, node capacity and downstream limits; QoS does not make a Pod immune to eviction.
Last updated: October 2026
Resource Specifications: Requests, Limits & QoS Classes
Kubernetes schedules and protects workloads based on CPU and memory declarations inside the resources block of container specifications:
CPU (Compressible): Measured in cores or millicores ( 1000m = 1 vCPU; 250m = 0.25 vCPU). If a container attempts to consume more CPU than its limit, the Linux kernel Completely Fair Scheduler (CFS) throttles execution by restricting CPU cycles, but the process is not killed .
Memory (Uncompressible): Measured in bytes or mebibytes ( Mi , Gi ). Memory cannot be throttled. If a container allocates memory beyond its configured limit, the Linux Out-Of-Memory (OOM) killer terminates the process immediately.
Quality of Service (QoS) Classes
Kubernetes classifies pods into three QoS classes based on how requests and limits are defined. This classification dictates eviction priority when a cluster node experiences resource pressure: Eviction Priority Order (First Evicted to Last Evicted):
BestEffort (First) ==> Burstable (Middle) ==> Guaranteed (Last)
QoS Class Resource Definition Criteria Scheduling & Eviction Behavior
Guaranteed Every container in the pod has CPU and Memory requests AND limits set, and requests == limits for both resources. Lower eviction risk when within requests; no QoS class is immune. Actual eviction ranking also considers whether usage exceeds requests and Pod priority.
Burstable At least one container has requests set, but requests do not equal limits (or limits are unset). Allowed to burst up to limits if node capacity permits; eviction depends on usage relative to requests, Pod priority, and resource pressure.
BestEffort No requests or limits specified for any container in the pod. No resource request reserved; vulnerable under pressure. Actual eviction is not an unconditional immediate QoS-only ordering.
Diagnosing OOMKilled (Exit Code 137)
When a container exceeds its memory limit or when the host node runs out of physical memory, the Linux kernel sends SIGKILL (signal 9) to the offending process. Kubernetes reports this status as OOMKilled with exit code 137 ( 128 + 9 = 137 ): ## Inspecting an evicted pod
kubectl describe pod order-service-6789b-f4x9q
Output shows: Containers:
order-service:
- State: Terminated
- Reason: OOMKilled
- Exit Code: 137
- Last State: Terminated
- Reason: OOMKilled
- Exit Code: 137
Troubleshooting and Remediation Steps for OOMKilled:
Differentiate Container OOM vs. Node OOM: Check kubectl describe node . If the node condition reports MemoryPressure: True , the host itself is depleted. If the node has plenty of free memory, the container exceeded its own limits.memory .
Java / JVM Cgroup Awareness: Use a supported JVM that recognizes the container's cgroup version and memory limit. Set a tested heap budget below the container limit, leaving room for native memory, threads and buffers; an arbitrary 75% setting is not universal. Inspect runtime flags and observed memory rather than assuming every older JVM has the same behavior.
Memory Leak Profiling: Use supported heap profiles and separate heap-dump tools to identify memory leaks before artificially inflating memory limits.
Autoscaling Architectures: HPA v2, VPA & Cluster Autoscaler
GKE provides three interrelated autoscaling engines that operate across different abstraction layers: +---------------------------------+
- | Workload Demand |
- +---------------------------------+
- |
- +-----------------------+-----------------------+
- | |
- v v
- +-------------------------+ +-------------------------+
- | Horizontal Pod Auto- | | Vertical Pod Auto- |
- | scaler (HPA v2) | | scaler (VPA) |
- | * Adjusts REPLICA COUNT | | * Adjusts CPU & MEMORY |
- | * Scales horizontally | | REQUESTS / LIMITS |
- +-------------------------+ +-------------------------+
- | |
- +-----------------------+-----------------------+
- |
- v
- +---------------------------------+
- | Pods in PENDING state (No Room) |
- +---------------------------------+
- |
- v
- +---------------------------------+
- | Cluster Autoscaler (CA) |
- | * Adjusts NODE COUNT in pool |
- +---------------------------------+
- Horizontal Pod Autoscaler (HPA v2)
HPA monitors resource consumption and adjusts the number of pod replicas. In autoscaling/v2 , HPA supports scaling based on CPU utilization, memory utilization, and custom Cloud Monitoring metrics: apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: order-processor-hpa
spec:
scaleTargetRef:
- apiVersion: apps/v1
- kind: Deployment
- name: order-processor
- minReplicas: 3
- maxReplicas: 50
metrics:
- # Target 70% average CPU utilization
- type: Resource
resource:
name: cpu
target:
- type: Utilization
- averageUtilization: 70
- # Target custom Cloud Monitoring metric: Pub/Sub backlog
- type: External
external:
metric:
name: pubsub.googleapis.com|subscription|num_undelivered_messages
selector:
matchLabels:
resource.labels.subscription_id: "orders-sub"
target:
- type: AverageValue
- averageValue: "50"
- HPA Requests, Targets, and Capacity
For CPU utilization targets, HPA compares usage to CPU requests , not limits. A Pod using 300m with a 500m request is at 60% utilization. The basic recommendation is ceil(current replicas * current metric / desired metric) : four replicas at 90% with a 60% target recommend ceil(4 * 90 / 60) = 6 . Stabilization, tolerance, missing metrics, readiness and min/max replicas can modify the applied result. Missing CPU requests can prevent a meaningful utilization calculation.
For external Pub/Sub backlog metrics, install and authorize a supported metrics adapter, select the exact subscription, and choose a target per replica that matches measured processing capacity. More replicas do not create database capacity. Bound concurrency/pools, monitor oldest-message age as well as count, and ensure node capacity can accommodate the requested Pods. The adapter and scaling identity need Monitoring permissions independently of the worker's Pub/Sub permissions.
Vertical Pod Autoscaler (VPA)
VPA monitors historical CPU and memory consumption over time and automatically optimizes container resource requests and limits. Common VPA modes include (additional modes depend on Kubernetes/GKE support):
Off : Generates resource recommendations only; does not alter pods.
Initial : Assigns optimal resource requests only when new pods are created.
Auto : applies resource recommendations using supported update mechanisms; recreation and in-place behavior depend on the configured mode and GKE version.
Exam Trap: Never combine HPA and VPA on the same resource metrics (CPU or memory). If HPA is configured to scale replicas when CPU exceeds 70%, and VPA is configured in Auto mode to increase CPU requests when CPU is high, the two controllers enter a destructive race condition: VPA will increase CPU requests while HPA simultaneously adds more pods, resulting in runaway resource over-allocation.
Cluster Autoscaler (CA) & GKE Autopilot
Cluster Autoscaler: Watches the Kubernetes scheduler for pods that cannot be scheduled due to insufficient CPU or memory capacity ( Pending state). When pending pods are detected, CA adds Compute Engine VM nodes to the node pool. When nodes remain underutilized for a sustained period, CA gracefully drains pods and terminates idle nodes.
GKE Autopilot: Completely eliminates manual node pool configuration and Cluster Autoscaler tuning. In Autopilot clusters, Google provisions and scales the underlying node infrastructure automatically, with Pod-based general-purpose billing and node-based billing for some specialized hardware/compute classes.