Study guide
10.2 Observability: Cloud Monitoring, Logging & Alert Policies
Google Professional Cloud Architect Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview, Strategy & Case Study Methodology
1.1 Exam Overview, Structure & Scoring Logistics 1.2 The Professional Cloud Architect Role & Exam Mindset 1.3 Deconstructing GCP Case Studies & Scenario Questions
2 Chapter 2: Designing for Business Requirements
2.1 Mapping Business Objectives, ROI & Total Cost of Ownership 2.2 Cost Optimization Strategies, Pricing Models & Trade-Offs 2.3 Business Continuity, Governance & Compliance Planning
3 Chapter 3: Designing for Technical Requirements & Availability
3.1 Scalability, Elasticity & Capacity Planning 3.2 High Availability, Disaster Recovery Patterns & RTO/RPO 3.3 Performance Optimization & Global Latency Reduction
4 Chapter 4: Compute Selection & Workload Architecture
4.1 Compute Engine Architecture, Sizing & Managed Instance Groups 4.2 Google Kubernetes Engine (GKE) Standard vs Autopilot & Microservices 4.3 Serverless Architectures: Cloud Run, Cloud Functions & Eventarc
5 Chapter 5: Storage, Database & Data Warehouse Architecture
5.1 Cloud Storage Architecture, Storage Classes & Lifecycle Management 5.2 Relational Databases: Cloud SQL, AlloyDB & Cloud Spanner 5.3 NoSQL & Big Data Storage: Firestore, Bigtable & BigQuery
6 Chapter 6: Networking, Hybrid Connectivity & Traffic Routing
6.1 VPC Design, Subnets, Shared VPC & VPC Peering 6.2 Hybrid & Multicloud Connectivity: Cloud VPN & Cloud Interconnect 6.3 Cloud Load Balancing, Cloud CDN & Cloud DNS
7 Chapter 7: Identity, Access & Resource Governance
7.1 Cloud IAM Hierarchy, Custom Roles & Service Accounts 7.2 Workload Identity Federation, Service Account Impersonation & IAP 7.3 Organization Policies, Tags & Resource Governance
8 Chapter 8: Security Architecture, Data Protection & Compliance
8.1 Data Encryption at Rest/Transit, Cloud KMS, CMEK & Cloud HSM 8.2 Network Security: Cloud Armor, VPC Service Controls & Firewalls 8.3 Compliance Frameworks, Sensitive Data Protection & Security Command Center
9 Chapter 9: DevOps, Automation & Infrastructure as Code
9.1 Infrastructure as Code (IaC) with Terraform on Google Cloud 9.2 CI/CD Pipelines: Cloud Build, Artifact Registry & Cloud Deploy 9.3 Deployment Strategies (Blue/Green, Canary) & GitOps with Config Sync 9.4 Interacting with Google Cloud Programmatically: Cloud SDK, Cloud Shell & Emulators
10 Chapter 10: Site Reliability Engineering & Operations Excellence
10.1 SRE Fundamentals: SLIs, SLOs, SLAs & Error Budgets 10.2 Observability: Cloud Monitoring, Logging & Alert Policies 10.3 Diagnosing Performance: Cloud Trace, Profiler & Error Reporting
11 Chapter 11: Data Pipelines, Integration & Modernization
11.1 Event-Driven Architectures with Cloud Pub/Sub & Eventarc 11.2 Stream and Batch Processing: Dataflow & Dataproc 11.3 API Management & Service Integration: Apigee & Cloud API Gateway 11.4 Migration Planning & Execution: Migration Center, DMS, Datastream & Transfer Options
12 Chapter 12: Well-Architected Framework & Official Case Study Solutions
12.1 Google Cloud Well-Architected Framework & FinOps Governance 12.2 Comprehensive Case Studies: EHR Healthcare & Cymbal Retail 12.3 Comprehensive Case Studies: Altostrat Media & KnightMotives Automotive
13 Chapter 13: AI, Machine Learning & Generative AI Architecture
13.1 AI/ML Platform Architecture: Vertex AI, Gemini Enterprise Agent Platform & AI Hypercomputer 13.2 Prebuilt AI APIs, Model Garden, Gemini Cloud Assist & Securing AI
10.2 Observability: Cloud Monitoring, Logging & Alert Policies
Key Takeaways Explain This Topic AI
Cloud Monitoring provides unified, real-time metrics collection across Google Cloud infrastructure: agentless metrics capture platform-level telemetry automatically, while the Google Cloud Ops Agent captures guest OS metrics (memory, disk swap, active processes) and application-level payloads via OpenTelemetry.
Log-based metrics extract quantitative signals directly from log streams, creating Counter metrics (tracking occurrence frequency such as HTTP 500 error counts) and Distribution metrics (extracting numeric values such as request latency to build histograms and percentiles).
Cloud Monitoring Alerting Policies support metric threshold, metric absence, forecast, and anomaly detection conditions, integrating with notification channels like PagerDuty, Slack, Webhooks, and Cloud Pub/Sub for automated remediation.
Cloud Logging ingests structured JSON logs into specialized log buckets ( _Required for 400-day free audit logs, _Default for 30-day customizable storage), using the Log Router to filter exclusions and route sinks to Cloud Storage, BigQuery, Pub/Sub, or other Google Cloud projects.
Log Analytics enables direct BigQuery SQL queries across log buckets without data duplication or external pipeline overhead, unlocking deep analytical investigations, compliance auditing, and security analytics directly within Cloud Logging.
Last updated: August 2026
Observability: Cloud Monitoring, Logging & Alert Policies
Architectural Objective: Observability is the prerequisite for operational excellence, high availability, and rapid incident triage. A Google Professional Cloud Architect must design comprehensive observability pipelines using Google Cloud Monitoring and Cloud Logging: selecting between agentless metrics and the Google Cloud Ops Agent, implementing custom and log-based metrics, architecting Log Router sinks and exclusion filters, enabling BigQuery-powered Log Analytics, and building proactive, anomaly-aware alerting policies.
Cloud Monitoring Architecture & Telemetry Ingestion
Google Cloud Monitoring (formerly Stackdriver) is a fully managed, hyper-scalable telemetry and observability service that ingests metrics, events, and metadata across Google Cloud, hybrid on-premises environments, and multicloud infrastructure. +-----------------------------------------------------------------------------------+
- | CLOUD MONITORING METRICS ARCHITECTURE |
- +-----------------------------------------------------------------------------------+
- | 1. AGENTLESS METRICS | Automatically collected by GCP hypervisors & APIs. |
- | (Platform Layer) | CPU utilization, network I/O, disk throughput, LB QPS. |
- +--------------------------+---------------------------------------------------------+
- | 2. GOOGLE CLOUD OPS AGENT| Installed inside VM Guest OS (Linux / Windows). |
- | (Host & Application) | Guest RAM usage, disk swap, processes, Apache, MySQL. |
- +--------------------------+---------------------------------------------------------+
- | 3. CUSTOM & OTel METRICS | Application-level metrics pushed via Monitoring API, |
- | (Software Layer) | OpenTelemetry Collector, or Managed Service for Prom. |
- +-----------------------------------------------------------------------------------+
- Time Series Data Model
Cloud Monitoring stores all metric data points as Time Series . Each time series consists of:
Monitored Resource: The specific cloud asset emitting telemetry (e.g., gce_instance , k8s_pod , cloudsql_database ).
Metric Descriptor: The metric type name (e.g., compute.googleapis.com/instance/cpu/utilization ), metric kind ( GAUGE , DELTA , CUMULATIVE ), and value type ( BOOL , INT64 , DOUBLE , DISTRIBUTION ).
Labels: Key-value metadata pairs providing dimensionality (e.g., instance_id , zone , response_code ).
Data Points: Timestamped numeric values.
Agentless Ingestion vs. Google Cloud Ops Agent
- Telemetry Capability Agentless Platform Metrics Google Cloud Ops Agent
- Installation Required None (Default, Zero Overhead) Installed on Compute Engine VM guest OS
- Underlying Technology GCP Hypervisor & API probes Fluent Bit (Logs) + OpenTelemetry Collector (Metrics)
- CPU & Network Metrics Yes (Host-level vCPU & NIC throughput) Yes (Guest-level breakdown per process)
Memory (RAM) Utilization No (Hypervisor cannot inspect guest RAM) Yes (Exact guest memory used, free, buffered, cached)
Disk Space & Swap Usage Disk I/O bytes only (no filesystem view) Exact mount-point capacity ( / , /var ) and swap usage
Third-Party App Telemetry No Built-in receivers for Nginx, Redis, Kafka, PostgreSQL, etc.
Log Collection System & GCP audit logs Guest syslog, /var/log/* , Windows Event Logs, app files
Warning
Exam Trap: Compute Engine VM instances do not report Guest OS Memory (RAM) utilization or filesystem disk space to Cloud Monitoring by default. If an exam scenario requires autoscaling Managed Instance Groups (MIGs) or triggering alerts based on memory utilization , you must install the Google Cloud Ops Agent .
Custom Metrics & Log-Based Metrics
When standard platform and Ops Agent metrics are insufficient to track business domain logic or application state, architects employ Custom Metrics and Log-Based Metrics. +-----------------------------------------------------------------------------------+
- | LOG-BASED METRICS ARCHITECTURE |
- +-----------------------------------------------------------------------------------+
- | Application writes JSON log: {"status": 500, "latency_ms": 342, "user": "u89"} |
- | │ |
- | v |
- | [ Cloud Logging Ingestion Pipeline ] ──> [ Log-Based Metric Evaluator ] |
- | │ |
- | ┌───────────────────────────────┴─────────────────────────┐ |
- | ▼ ▼ |
- | [ COUNTER METRIC ] [ DISTRIBUTION ]|
- | Counts occurrences matching filter: Extracts numerical value |
- | `jsonPayload.status = 500` `jsonPayload.latency_ms` |
- | Produces metric: `custom.googleapis.com/errors_500` Produces p50/p95/p99 dist|
- +-----------------------------------------------------------------------------------+
- 1. Custom Metrics
Ingestion: Emitted directly from application code using the Cloud Monitoring API ( MetricServiceClient ), OpenTelemetry SDKs, or the Google Cloud Managed Service for Prometheus (GMP).
Cardinality Governance: High cardinality (e.g., placing unique User IDs or UUID transaction tokens into metric labels) creates millions of independent time series, causing massive ingestion costs and degraded dashboard query performance. User IDs must be placed in structured logs, never metric labels.
2. Log-Based Metrics: Counter vs. Distribution
Log-based metrics bridge Cloud Logging and Cloud Monitoring, extracting real-time time series from log entries as they pass through the ingestion pipeline:
Counter Metrics: Increments a count each time a log entry matches a designated Cloud Logging query filter (e.g., resource.type="k8s_container" AND severity>=ERROR ). Useful for counting application errors, login attempts, or checkout events.
Distribution Metrics: Extracts a numeric field from structured JSON log payloads (e.g., extracting jsonPayload.processing_time_ms ) and records the values in a statistical distribution histogram. Enables calculating p50, p90, p95, and p99 percentiles over time for alerting and SLO tracking.
Synthetic Monitoring: Uptime Checks & Broken Link Probers
While metrics reflect internal system state, Synthetic Monitoring simulates external user traffic to evaluate external reachability, SSL certificate validity, and endpoint responsiveness. +-----------------------------------------------------------------------------------+
- | SYNTHETIC MONITORING ARCHITECTURE |
- +-----------------------------------------------------------------------------------+
- | Probers located across 6 Global Regions (USA, Europe, Asia-Pacific, etc.) |
- | │ |
- | +───> [ Global HTTPS Uptime Check ] ──> Public API Endpoint |
- | │ (Validates HTTP 200, Latency < 500ms, SSL Expiry > 14 days) |
- | │ |
- | +───> [ Private Uptime Check ] ───────> Internal VPC Endpoint |
- | (Via VPC Network / PSC) (Internal Microservice) |
- +-----------------------------------------------------------------------------------+
Global Uptime Checks: Google-managed probing servers located in multiple geographic regions periodically send HTTP, HTTPS, or TCP requests to public IP endpoints, URLs, App Engine apps, or Load Balancers. An alert triggers if multiple distinct geographic probing locations report failure simultaneously.
Private Uptime Checks: Validates internal, non-public endpoints within a customer VPC network (such as internal load balancers or private GKE clusters) routed securely without traversing the public internet.
Custom Synthetic Probers: Cloud Functions-based synthetic probers (using Puppeteer or Mocha) that execute multi-step synthetic user flows (e.g., logging in, adding an item to cart, and submitting a test order).
Cloud Monitoring Alerting Policies: Conditions & Notification
Alerting policies notify operations teams when systems exhibit abnormal behavior, risk breaching SLOs, or experience resource exhaustion. +-----------------------------------------------------------------------------------+
- | ALERTING POLICY ANATOMY |
- +-----------------------------------------------------------------------------------+
- | 1. CONDITION | What triggers the alert? (Threshold, Absence, Forecast, |
- | | Anomaly, PromQL, MQL Query). |
- +---------------------+-------------------------------------------------------------+
- | 2. AGGREGATION & | How data points are grouped and aligned over time |
- | ALIGNMENT | (Alignment Period: 1m, 5m; Reducer: MEAN, SUM, PERCENTILE). |
- +---------------------+-------------------------------------------------------------+
- | 3. TRIGGER RULE | Percentage of instances or number of time series violating |
- | | condition (e.g., "Any time series violates for 5 minutes"). |
- +---------------------+-------------------------------------------------------------+
- | 4. NOTIFICATIONS | PagerDuty, Slack, Email, Webhook (Cloud Run/Functions), |
- | | Cloud Pub/Sub Topic (Automated Event-Driven Remediation). |
- +---------------------+-------------------------------------------------------------+
- | 5. DOCUMENTATION | Markdown runbook instructions & troubleshooting links. |
- +-----------------------------------------------------------------------------------+
- Advanced Alert Condition Types
Condition Type Operational Mechanism Best Use Case
Metric Threshold Compares time series value against a static upper/lower boundary. High CPU utilization (>85%), HTTP 5xx error rate spikes.
Metric Absence Fires when expected time series stops sending data for a specified duration. Dead batch worker, stalled message consumer, terminated agent.
Metric Forecast (Predictive) Uses linear regression to forecast when a metric will cross a threshold in the future (e.g., within 24–48 hours). Disk volume filling up , quota exhaustion, database connection exhaustion.
Anomaly Detection Compares current metric against historical baselines and seasonal patterns (day-of-week, hour-of-day). Sudden drop in checkout traffic during peak business hours.
Alert Snoozing & Mute Rules
To prevent alert storms during scheduled maintenance windows, database patching, or network upgrades, architects configure Mute Rules (Alert Snoozing). Mute rules suppress notification delivery for matching alert policies during a specified maintenance window while continuing to record incidents in Cloud Monitoring for historical auditing.
Cloud Logging Architecture: Ingestion, Storage & Sinks
Google Cloud Logging is a centralized, real-time log management service capable of ingesting exabytes of structured JSON, text, and binary logs across an enterprise footprint. +-----------------------------------------------------------------------------------+
- | CLOUD LOGGING INGESTION & ROUTER FLOW |
- +-----------------------------------------------------------------------------------+
- | [ Log Producers: GKE, Compute Engine, Cloud Run, Audit Logs, Custom Apps ] |
- | │ |
- | v |
- | [ CLOUD LOG ROUTER ] |
- | │ |
- | ┌────────────────────────────┼────────────────────────────┐ |
- | ▼ ▼ ▼ |
- | [ EXCLUSION FILTERS ] [ DEFAULT SINK ] [ EXPORT SINKS ] |
- | Drops high-volume debug Routes to `_Default` Routes filtered logs |
- | logs before ingestion. Log Bucket (30-day). to enterprise targets|
- | (Zero Ingestion Cost) │ │ |
- | v │ |
- | [ LOG ANALYTICS ] │ |
- | (BigQuery SQL queries │ |
- | directly on bucket) │ |
- | │ |
- | ┌─────────────────────────────────────────────────────────┴─────────┐ |
- | ▼ ▼ ▼ ▼ |
- | [ Cloud Storage ] [ BigQuery ] [ Pub/Sub ] [Other]|
- | Long-term compliance Real-time SQL analytics SIEM export (GCP |
- | cold archive (years). & machine learning. (Splunk/Datadog)Proj|
- +-----------------------------------------------------------------------------------+
- 1. Log Buckets & Default Retention
Cloud Logging stores ingested logs in specialized Log Buckets (distinct from Cloud Storage buckets):
_Required Log Bucket: Automatically stores Admin Activity audit logs, System Event audit logs, and Access Transparency logs. Retained for 400 days at zero cost . Cannot be disabled, modified, or deleted.
_Default Log Bucket: Automatically stores all other runtime logs (Data Access audit logs, GKE stdout/stderr, Compute Engine syslog, Cloud Run logs). Retained for 30 days by default (customizable from 1 to 3,650 days). Incur ingestion and storage fees.
2. Log Router: Exclusion Filters & Cost Optimization
The Log Router evaluates every log entry at the instant of ingestion against defined rules:
Exclusion Filters: Drops unwanted, verbose logs (e.g., debug logs severity=DEBUG or high-frequency health checks httpRequest.requestUrl:"/healthz" ) before they enter log buckets. Excluded logs incur zero storage and zero ingestion costs.
3. Log Sinks & Export Destinations
Log Sinks stream filtered logs matching an inclusion filter to external destinations in real time:
Export Destination Latency Profile Primary Architectural Purpose
Cloud Storage Batch (Files written in chunks every few minutes) Cost-effective, long-term regulatory compliance archiving (e.g., 7-year HIPAA/PCI-DSS storage with Object Lifecycle Management and Bucket Lock).
BigQuery Real-time streaming insert Complex SQL analysis, ad-hoc security investigations, cross-dataset correlations with business data, Looker dashboards.
Cloud Pub/Sub Real-time streaming (sub-second) Exporting security telemetry to external SIEM tools (Splunk, Chronicle, Datadog) or triggering automated event-driven Cloud Functions.
Other GCP Project Real-time Centralized security and audit project aggregation across multi-project enterprise organizations.
Organization-Level Aggregated Sinks
In an enterprise with hundreds of Google Cloud projects under an Organization or Folder hierarchy, configuring log sinks per project creates massive administrative overhead. Architects create an Aggregated Log Sink at the Organization or Folder level with includeChildren = true . This automatically captures and centralizes all audit logs across current and future projects into a single security operations data lake.
Log Analytics Powered by BigQuery
Traditionally, running SQL queries on log data required exporting logs via a sink to BigQuery, paying for BigQuery streaming ingestion and duplicate storage. Log Analytics eliminates this redundancy by embedding a BigQuery SQL engine directly on top of Cloud Logging log buckets. -- Direct SQL query executed inside Cloud Logging Log Analytics
- SELECT
- timestamp,
- resource.type,
- json_payload.user_email,
- http_request.status,
- http_request.latency
- FROM
- `my-project.global._Default._AllLogs`
- WHERE
- http_request.status >= 500
- AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
- ORDER BY
- timestamp DESC
- LIMIT 100;
- Architectural Advantages of Log Analytics
Zero Ingestion Duplication: Query log data using ANSI BigQuery SQL directly where it resides inside the log bucket.
BigQuery Studio & Looker Integration: Linked BigQuery datasets allow security analysts to query log buckets directly from the BigQuery console and visualize log trends in Looker Studio without ETL pipelines.
Field-Level JSON Parsing: Queries nested JSON payloads ( json_payload.request.headers.user_agent ) using standard JSON SQL operators.
- Concrete Architectural Scenario: Multi-Project Enterprise Observability
- Scenario Profile
Organization: Global retail enterprise operating 120 GCP projects partitioned across Development, Staging, and Production folders.
Requirements: Centralized security audit repository; 7-year compliance archiving for financial transactions; real-time PagerDuty alerting for API errors; automated exclusion of high-volume non-critical health check logs to reduce monthly cloud spend. [ Enterprise GCP Organization ]
- ├── Project: eCommerce-Prod-01 ──┐
- ├── Project: eCommerce-Prod-02 ──┼──> [ Aggregated Org Sink (includeChildren=true) ]
- └── Project: Payments-Prod-01 ──┘ │
- ├──> [ Pub/Sub ] ──> Corporate SIEM
- ├──> [ GCS Bucket ] (7-Yr Lock Archive)
- └──> [ Central Security Log Bucket ]
- └── (Log Analytics Enabled)
- Architecture Blueprint
Metrics Collection: Compute Engine VMs running the Google Cloud Ops Agent to stream memory, disk swap, and application logs into Cloud Monitoring.
Cost Optimization via Log Exclusions: Ingestion exclusion filter deployed on _Default bucket: jsonPayload.request_path="/healthz" OR severity=DEBUG , eliminating 45% of unnecessary log ingestion costs.
Centralized Log Aggregation: Organization-level Aggregated Log Sink ( includeChildren=true ) routing all cloudaudit.googleapis.com logs to a centralized Security-Audit-Project .
Compliance Archiving & SIEM Integration: Sinks configured to route financial audit logs to a Cloud Storage bucket with Bucket Lock (Object Retention) for 7 years, and streaming to Pub/Sub for ingestion by Chronicle SIEM.
Intelligent Alerting: Cloud Monitoring alerting policies with Metric Forecast conditions predicting disk saturation 48 hours in advance, and Multi-Window Multi-Burn-Rate policies paging on-call engineers via PagerDuty.
Important
Exam Watch: On the PCA exam, if a question asks how to collect guest OS memory utilization or disk space from Compute Engine VMs, the answer is always the Google Cloud Ops Agent . If a question asks how to aggregate audit logs across an entire enterprise organization into a central project at minimal operational overhead, choose an Organization-level Aggregated Log Sink with includeChildren = true .