Study guide
16.3 Application Metrics, Health Checks & Alerting with Cloud Monitoring
Google Cloud Professional Cloud Developer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam and Compute Platform Decisions
1.1 Current Exam, Sources, and Scenario Method 1.2 Cloud Run Execution and Platform Selection 1.3 GKE Modes and Compute Engine Trade-offs
2 Chapter 2: Container Engineering and API Contracts
2.1 Stateless Microservices and Multi-stage Builds 2.2 Container Hardening, Signals, and Layer Caching 2.3 REST, gRPC, and Protobuf Contracts
3 Chapter 3: gRPC Compatibility and Geographic Routing
3.1 Schema Evolution and gRPC Routing 3.2 Geography, Network Tiers, and Load Balancers 3.3 Serverless NEGs, Sessions, and Edge Controls
4 Chapter 4: API Management and Caching
4.1 API Management & Rate Limiting with Apigee and API Gateway 4.2 Memorystore and Cache Access Patterns 4.3 Cache Invalidation, Capacity, and Failover
5 Chapter 5: Messaging, Task Dispatch, and Orchestration
5.1 Event-Driven Messaging with Cloud Pub/Sub 5.2 Task Queues & Rate Throttling with Cloud Tasks 5.3 Event Routing and Service Orchestration: Eventarc & Cloud Workflows
6 Chapter 6: Resilience and Relational Availability
6.1 Designing Resilient Systems: Circuit Breaking, Retries & Failover 6.2 Cloud SQL Availability and Recovery 6.3 Connection Budgets and Spanner Consistency
7 Chapter 7: Distributed Schemas and NoSQL Data Models
7.1 Spanner Keys, Indexes, and Transactions 7.2 Firestore Data, Indexes, and Transactions 7.3 Bigtable Row Keys and Replication
8 Chapter 8: Object Storage and Application Identity
8.1 Cloud Storage Lifecycle and Signed Access 8.2 Workload Identity and Service Authentication 8.3 Secret Delivery, Rotation, and IAP
9 Chapter 9: Security, Data Integration, and Developer Credentials
9.1 Private Communication, Key Rotation, and Vulnerability Response 9.2 AlloyDB Schemas and BigQuery Ingestion 9.3 CLI Configurations, Local ADC, and Cloud Shell
10 Chapter 10: Managed Development and Automated Tests
10.1 Cloud Workstations and the Container Inner Loop 10.2 Emulating Google Cloud Services & Database Auth Proxies 10.3 Automated Testing Strategies: Unit, Integration, and Mocking
11 Chapter 11: AI Development Tools and Build Pipelines
11.1 AI Coding Assistants, Context, MCP, and Test Review 11.2 Cloud Build Workers and Step Dependencies 11.3 Builders, Substitutions, Triggers, and IAM
12 Chapter 12: Artifact Security and Admission Policies
12.1 Artifact Management and Vulnerability Scanning with Artifact Registry 12.2 Build Provenance and Admission Concepts 12.3 Attestors, Policy Enforcement, and Break-glass
13 Chapter 13: Cloud Run and GKE Deployment Configuration
13.1 Cloud Run Deployment and Scaling Configuration 13.2 Cloud Run CPU, Private Egress, and Secrets 13.3 GKE Workload Controllers and Gateway Routing
14 Chapter 14: Kubernetes Resources, Health, and Progressive Releases
14.1 GKE Resources, OOM Diagnosis, and Autoscaling 14.2 Container Health Probes, Lifecycle Hooks & Graceful Shutdown 14.3 Progressive Delivery: Traffic Splitting, Canary, and Blue/Green Rollouts
15 Chapter 15: Source/Event Deployment and Cloud API Integration
15.1 Source Deployment, Event Receivers, and API Versions 15.2 Client Libraries, Service Enablement, and ADC 15.3 API Responses, Pagination, Caches, and Retries
16 Chapter 16: Tracing, Sampling, and Monitoring
16.1 Trace Context and OpenTelemetry Export 16.2 Custom Spans, Trace Analysis, and Sampling 16.3 Application Metrics, Health Checks & Alerting with Cloud Monitoring
17 Chapter 17: Structured Logs, Routing, and Error Management
17.1 Structured Logs, Trace Correlation, and Queries 17.2 Log Routing, Export Destinations, and Exclusions 17.3 Error Reporting Formats and Regression Handling
18 Chapter 18: Profiling, Generative AI, and Assisted Troubleshooting
18.1 Cloud Profiler Types and Flame Graph Analysis 18.2 Generative AI Integration with the Google Gen AI SDK 18.3 Evidence-Based Troubleshooting with AI Assistance
16.3 Application Metrics, Health Checks & Alerting with Cloud Monitoring
Key Takeaways Explain This Topic AI
Cloud Monitoring models time series data using three metric kinds (GAUGE, DELTA, CUMULATIVE); supported value types and kind/type combinations depend on the metric source.
Custom application metric labels must strictly avoid high-cardinality values (such as user IDs or order numbers) to prevent exponential time series explosion and quota exhaustion.
Latency distributions are captured as DISTRIBUTION metrics and rendered as Heatmaps to estimate percentiles from bucketed observations (p50, p95, p99) without the mathematical distortion of averaging.
Alerting policies combine temporal alignment (ALIGN_RATE, ALIGN_DELTA, ALIGN_PERCENTILE_99) over fixed alignment periods with spatial reduction (REDUCE_SUM, REDUCE_MEAN) across monitored resources.
Global Uptime checks execute synthetic probes from multiple worldwide Google Cloud regions to validate external routing, SSL certificate validity, and HTTP response codes independently of internal container health.
Last updated: October 2026
Application Metrics, Health Checks & Alerting with Cloud Monitoring
Maintaining the health, availability, and performance of cloud-native microservices requires comprehensive metric instrumentation and proactive alerting. Cloud Monitoring (formerly Stackdriver Monitoring) is Google Cloud's centralized telemetry platform, collecting system and application metrics, generating real-time dashboards, and evaluating alerting policies.
For the Google Cloud Professional Cloud Developer exam, you must master the Cloud Monitoring multidimensional data model, understand metric kinds and value types, design custom application metrics while avoiding cardinality traps, analyze latency distributions using heatmaps, configure time series alignment and reduction operations, and implement global uptime checks.
The Cloud Monitoring Data Model: Metric Kinds and Value Types
In Cloud Monitoring, telemetry is organized into Time Series . A time series is a regular or irregular sequence of timestamped data points uniquely identified by a Metric Descriptor , a Monitored Resource Descriptor , and associated label key-value pairs:
- Time Series = MetricDescriptor + MonitoredResource + MetricLabels + Points[(t0, v0), (t1, v1), ...]
- Metric Kinds ( metricKind )
The metric kind defines how data points in the time series relate to the passage of time:
Metric Kind Description & Semantics Value Behavior Practical Examples
GAUGE Instantaneous measurement taken at a specific point in time. Fluctuates freely up and down. No memory of prior points. Current CPU utilization percentage, available memory, active TCP connections, thread pool size.
DELTA Measures the change or accumulation over a specific non-overlapping time interval [t_start, t_end] . Represents the delta during the interval. Resets each window. Number of HTTP requests received in the last 60 seconds, bytes transferred during an interval.
CUMULATIVE Monotonically increasing counter measuring continuous accumulation since an initial start timestamp t0 . Strictly increases over time. A reset or new start timestamp begins a new accumulation interval. Total requests served since container boot, total CPU seconds consumed since system startup.
Exam Tip: Google Cloud natively prioritizes CUMULATIVE for counters (such as request counts). Because network packets can be delayed or arrive out of order, cumulative counters prevent data loss: if an intermediate scraping interval is missed, the next cumulative point captures the complete uninterrupted count since boot.
Value Types ( valueType )
Every metric specifies the data format of its recorded values:
BOOL : Boolean state ( true / false ), e.g., service binary health check.
INT64 : Signed 64-bit integer, e.g., exact transaction counts or queue depth.
DOUBLE : Double-precision 64-bit floating point, e.g., CPU ratio (0.0 to 1.0).
DISTRIBUTION : A complex statistical histogram containing histogram buckets, count, mean, and sum of squared deviations. Essential for recording request latency histograms , allowing post-hoc calculation of arbitrary percentiles (p50, p90, p95, p99).
System Metrics vs. Custom Application Metrics
System Metrics: Ingested automatically by Google Cloud infrastructure without requiring agent installation or developer instrumentation (e.g., run.googleapis.com/container/cpu/utilizations , compute.googleapis.com/instance/disk/read_bytes_count ). Many built-in Google Cloud metrics are non-chargeable; agent, user-defined, and Prometheus metrics follow their own pricing rules.
Custom Application Metrics: User-defined metrics published under the custom.googleapis.com/ namespace. Applications publish custom metrics via:
Google Cloud Managed Service for Prometheus (GMP): Standard /metrics Prometheus scraping using GKE PodMonitoring custom resources.
OpenTelemetry Cloud Monitoring Exporter: Streaming application metrics directly via the OTel Metrics SDK.
Cloud Monitoring API: Explicitly creating metric descriptors and publishing via projects.timeSeries.create .
The High-Cardinality Metric Anti-Pattern
A critical exam concept is avoiding high cardinality in custom metric labels. In Cloud Monitoring, a separate, persistent time series is allocated for every unique combination of label values . ## HIGH CARDINALITY DISASTER (DO NOT DO THIS):
custom.googleapis.com/orders/processed
Labels:
- environment: "production" # Cardinality: 2 (prod, staging)
- payment_method: "credit_card" # Cardinality: 4 (card, paypal, apple_pay, crypto)
- user_id: "USER-8492019" # Cardinality: 10,000,000 unique users!
Total Time Series Created = 2 * 4 * 10,000,000 = 80,000,000 Time Series!
Consequences of High Cardinality
Quota Exhaustion: Projects enforce strict quotas on the number of active time series (limits depend on metric and monitored-resource dimensions; check the current quota table). Exceeding this quota results in HTTP 429 quota exhaustion errors, dropping subsequent metric writes.
Exorbitant Monitoring Costs: Billing depends on the metric ingestion path and current pricing tiers; custom-byte and Prometheus-sample pricing are different. Creating millions of time series generates thousands of dollars in unintended metric ingestion charges.
Query Degradation: Dashboards and alerting policies time out attempting to aggregate across millions of individual series.
Developer Best Practice: Metric labels must have low, bounded cardinality (e.g., HTTP status code class 2xx / 4xx / 5xx , cloud region, deployment environment). High-cardinality attributes like user_id , order_id , ip_address , or request_url belong exclusively in Cloud Logging structured payload fields or Cloud Trace span attributes , never in Cloud Monitoring metric labels.
- Dashboards and Visualization: Heatmaps for Latency Percentiles
- The Fallacy of the Average (Mean)
A common production pitfall is monitoring API latency using arithmetic mean ( sum(duration) / count ). If 99 requests take 10ms and 1 request takes 2,000ms, the average latency is ~30ms, completely hiding the fact that 1% of users are experiencing unacceptable timeouts. Production latency must be tracked using percentiles: p50 (median) , p95 , and p99 .
Heatmaps for DISTRIBUTION Metrics
Line charts of p99 latency can still obscure multimodal latency distributions. A Heatmap chart visualizes the full underlying DISTRIBUTION metric.
The X-axis represents time , the Y-axis represents latency bucket boundaries (logarithmic or linear), and the color saturation/darkness of each pixel indicates the volume of requests falling into that specific bucket.
Diagnosing Issues via Heatmaps:
A solid dark band across the bottom indicates consistent low-latency requests.
A faint secondary band running higher up indicates a dual-path execution problem (e.g., database cache hits at 10ms vs. unindexed queries at 800ms).
Vertical streaks indicate momentary system-wide stalls, such as stop-the-world JVM garbage collection cycles or pod evictions.
Time Series Alignment and Reduction Pipeline
Raw telemetry points arrive at irregular intervals from hundreds of distributed container instances. Before Cloud Monitoring can plot a metric on a chart or evaluate an alerting condition, the time series must pass through a two-stage aggregation pipeline: [Raw Asynchronous Points from Multiple Container Instances]
- │
- ▼
- [Stage 1: Temporal Alignment (Per-Series Normalization)]
- Partitions time into uniform intervals (Alignment Period, e.g., 1m, 5m)
- Applies an Aligner function to condense points within each interval:
- ALIGN_RATE: Converts CUMULATIVE counters into per-second rates (req/sec)
- ALIGN_DELTA: Computes net change across the interval
- ALIGN_MEAN: Computes average value for GAUGE metrics (CPU, Memory)
- ALIGN_PERCENTILE_99: Extracts 99th percentile from DISTRIBUTION
- │
- ▼
- [Stage 2: Spatial Reduction (Cross-Series Aggregation)]
- Combines multiple normalized time series across instances into fewer series
- Applies a Reducer function:
- REDUCE_SUM: Adds values across instances (Total cluster requests/sec)
- REDUCE_MEAN: Computes cluster-wide average utilization
- REDUCE_MAX: Tracks peak utilization across any single node
- Group-By: Groups resulting series by specific labels (e.g., response_code)
- │
- ▼
- [Evaluation: Alerting Policy Threshold / Dashboard Line Chart]
- Alerting Policies: Thresholds, Metric Absence, MQL and PromQL
An Alerting Policy defines the conditions under which Cloud Monitoring generates incidents and dispatches notifications.
Condition Types
Metric Threshold Condition: Triggers when an aligned and reduced metric value crosses a threshold (e.g., HTTP 5xx error rate > 5%). To prevent false alarms from momentary 10-second traffic bursts, policies enforce an evaluation duration window (e.g., "condition must be met for 5 consecutive minutes").
Metric Absence Condition: Triggers when expected telemetry stops arriving entirely for a specified duration. Crucial for monitoring asynchronous background workers, cron jobs, or IoT devices: if a message processing worker fails to report a heartbeat metric for 10 minutes, an absence alert fires immediately.
PromQL and Existing MQL Assets
Use the query builder or PromQL for new Monitoring charts and alerts. MQL is no longer recommended; since July 22, 2025, new MQL charts and alert policies cannot be created through the console, although existing assets and API workflows remain supported.
For example, a PromQL condition for a bounded-cardinality application counter exported through Managed Service for Prometheus can compare the error rate to the total rate: sum(rate(checkout_requests_total{status_class="5xx"}[5m]))
- /
- sum(rate(checkout_requests_total[5m])) > 0.05
Configure the condition's retest/evaluation duration separately (for example five minutes), select the intended service/environment, and define behavior for no traffic and missing data. The five-minute rate window is not itself a five-minute sustained-breach requirement. For fleet latency, combine compatible histogram buckets before estimating a percentile; averaging instance p99 values does not produce the fleet p99.
- Notification Channels and Global Uptime Checks
- Notification Channels and Automated Remediation
When an alerting policy condition triggers an incident, Cloud Monitoring routes notifications to configured channels:
Standard Channels: Email, SMS, Slack, PagerDuty, Webhooks.
Automated Remediation via Pub/Sub: Alerts can publish directly to a Cloud Pub/Sub topic. This enables event-driven remediation: an alert message triggers a Cloud Function or Cloud Run service to restart a failing process, purge an overloaded Memorystore cache, or spin up emergency compute capacity without requiring human operator intervention.
Global Uptime Checks
Internal health checks (such as Kubernetes liveness/readiness probes) only confirm that a container process is running inside its local VPC. They cannot verify whether external customers can reach your service through public DNS, Edge Points of Presence (PoPs), and Cloud Load Balancers.
Global Uptime Checks resolve this by executing external synthetic HTTP/HTTPS/TCP probes from multiple geographic regions simultaneously (e.g., US East, US West, Europe West, Asia Pacific):
Probing Frequency: Choose a supported check interval, such as 1, 5, 10, or 15 minutes, and verify the current API options.
Validation Criteria: Checks assert expected HTTP status codes (e.g., 200 OK ), response body string/regex matching, and SSL/TLS certificate expiration (dispatching alerts e.g., 30 days before a certificate expires).
Resilience: Configure the alert aggregation/threshold to require failures from the chosen number of checker locations; this is not an unconditional default, preventing false alarms caused by localized transit ISP anomalies.
Official references checked October 18, 2026: Google Cloud 1 .