Study guide
16.2 Custom Spans, Trace Analysis, and Sampling
Google Cloud Professional Cloud Developer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam and Compute Platform Decisions
1.1 Current Exam, Sources, and Scenario Method 1.2 Cloud Run Execution and Platform Selection 1.3 GKE Modes and Compute Engine Trade-offs
2 Chapter 2: Container Engineering and API Contracts
2.1 Stateless Microservices and Multi-stage Builds 2.2 Container Hardening, Signals, and Layer Caching 2.3 REST, gRPC, and Protobuf Contracts
3 Chapter 3: gRPC Compatibility and Geographic Routing
3.1 Schema Evolution and gRPC Routing 3.2 Geography, Network Tiers, and Load Balancers 3.3 Serverless NEGs, Sessions, and Edge Controls
4 Chapter 4: API Management and Caching
4.1 API Management & Rate Limiting with Apigee and API Gateway 4.2 Memorystore and Cache Access Patterns 4.3 Cache Invalidation, Capacity, and Failover
5 Chapter 5: Messaging, Task Dispatch, and Orchestration
5.1 Event-Driven Messaging with Cloud Pub/Sub 5.2 Task Queues & Rate Throttling with Cloud Tasks 5.3 Event Routing and Service Orchestration: Eventarc & Cloud Workflows
6 Chapter 6: Resilience and Relational Availability
6.1 Designing Resilient Systems: Circuit Breaking, Retries & Failover 6.2 Cloud SQL Availability and Recovery 6.3 Connection Budgets and Spanner Consistency
7 Chapter 7: Distributed Schemas and NoSQL Data Models
7.1 Spanner Keys, Indexes, and Transactions 7.2 Firestore Data, Indexes, and Transactions 7.3 Bigtable Row Keys and Replication
8 Chapter 8: Object Storage and Application Identity
8.1 Cloud Storage Lifecycle and Signed Access 8.2 Workload Identity and Service Authentication 8.3 Secret Delivery, Rotation, and IAP
9 Chapter 9: Security, Data Integration, and Developer Credentials
9.1 Private Communication, Key Rotation, and Vulnerability Response 9.2 AlloyDB Schemas and BigQuery Ingestion 9.3 CLI Configurations, Local ADC, and Cloud Shell
10 Chapter 10: Managed Development and Automated Tests
10.1 Cloud Workstations and the Container Inner Loop 10.2 Emulating Google Cloud Services & Database Auth Proxies 10.3 Automated Testing Strategies: Unit, Integration, and Mocking
11 Chapter 11: AI Development Tools and Build Pipelines
11.1 AI Coding Assistants, Context, MCP, and Test Review 11.2 Cloud Build Workers and Step Dependencies 11.3 Builders, Substitutions, Triggers, and IAM
12 Chapter 12: Artifact Security and Admission Policies
12.1 Artifact Management and Vulnerability Scanning with Artifact Registry 12.2 Build Provenance and Admission Concepts 12.3 Attestors, Policy Enforcement, and Break-glass
13 Chapter 13: Cloud Run and GKE Deployment Configuration
13.1 Cloud Run Deployment and Scaling Configuration 13.2 Cloud Run CPU, Private Egress, and Secrets 13.3 GKE Workload Controllers and Gateway Routing
14 Chapter 14: Kubernetes Resources, Health, and Progressive Releases
14.1 GKE Resources, OOM Diagnosis, and Autoscaling 14.2 Container Health Probes, Lifecycle Hooks & Graceful Shutdown 14.3 Progressive Delivery: Traffic Splitting, Canary, and Blue/Green Rollouts
15 Chapter 15: Source/Event Deployment and Cloud API Integration
15.1 Source Deployment, Event Receivers, and API Versions 15.2 Client Libraries, Service Enablement, and ADC 15.3 API Responses, Pagination, Caches, and Retries
16 Chapter 16: Tracing, Sampling, and Monitoring
16.1 Trace Context and OpenTelemetry Export 16.2 Custom Spans, Trace Analysis, and Sampling 16.3 Application Metrics, Health Checks & Alerting with Cloud Monitoring
17 Chapter 17: Structured Logs, Routing, and Error Management
17.1 Structured Logs, Trace Correlation, and Queries 17.2 Log Routing, Export Destinations, and Exclusions 17.3 Error Reporting Formats and Regression Handling
18 Chapter 18: Profiling, Generative AI, and Assisted Troubleshooting
18.1 Cloud Profiler Types and Flame Graph Analysis 18.2 Generative AI Integration with the Google Gen AI SDK 18.3 Evidence-Based Troubleshooting with AI Assistance
16.2 Custom Spans, Trace Analysis, and Sampling
Key Takeaways Explain This Topic AI
End spans and record status/error events explicitly.
Parallel independent calls can shorten a serial critical path, subject to resource and dependency constraints.
Ratio sampling reduces expected volume but is not a hard budget; tail sampling cannot recover telemetry dropped upstream.
Last updated: October 2026
Automatic vs. Custom Manual Instrumentation
Automatic Instrumentation: Injects tracing middleware transparently into common web frameworks (Express, Spring Boot, Flask, Gin), HTTP client libraries, gRPC stubs, and database drivers. It automatically captures inbound routes, outbound HTTP requests, status codes, and SQL query strings without requiring code changes.
Custom Manual Instrumentation: Necessary when developers need to measure internal algorithmic processing, business-critical transactions, or background worker threads not captured by framework wrappers. // Creating Custom Spans with OpenTelemetry in Go
- import (
- "context"
- "go.opentelemetry.io/otel"
- "go.opentelemetry.io/otel/attribute"
- "go.opentelemetry.io/otel/codes"
- )
- func ProcessTransaction(ctx context.Context, orderID string, amount float64) error {
- tracer := otel.GetTracerProvider().Tracer("order-processor")
- // Start custom child span linked to incoming context
- ctx, span := tracer.Start(ctx, "ProcessTransaction")
- defer span.End() // ALWAYS defer End() to guarantee duration capture
- span.SetAttributes(
- attribute.String("order.id", orderID),
- attribute.Float64("order.amount", amount),
- )
- if err := executeRiskAnalysis(ctx, orderID); err != nil {
- // Record an exception event and mark span status as ERROR
- span.RecordError(err)
- span.SetStatus(codes.Error, err.Error())
- return err
- }
- span.SetStatus(codes.Ok, "Transaction processed successfully")
- return nil
- }
Exam Trap: Forgetting to call span.End() (or omitting defer span.End() ) causes the span to remain permanently open in memory. The span will never be exported to Cloud Trace, resulting in missing spans and an incomplete trace view.
Cloud Trace Console Analysis: Latency Distributions & Waterfall Timelines
The Cloud Trace console provides analytical views to diagnose production performance regressions:
Latency Distribution Curves (Scatter Plots)
The Cloud Trace dashboard renders a scatter plot where the X-axis represents time of day and the Y-axis represents request latency in milliseconds or seconds. Each individual dot represents a sampled trace.
Diagnosing Bimodal Distributions: A healthy service displays a tight, single horizontal band of dots (e.g., 20ms–40ms). If the scatter plot splits into two distinct horizontal bands (e.g., one band at 25ms and a second band at 450ms), the system is exhibiting a bimodal distribution. In microservices, this typically indicates cache hit vs. cache miss behavior, or execution paths branching between fast indexed database reads and slow unindexed full-table scans.
Outliers: Isolated dots appearing high above the baseline represent request timeouts, garbage collection pauses, or database lock contention.
Waterfall Timelines
Clicking any point on the scatter plot opens its Waterfall Timeline :
Visualizes hierarchical parent-child execution order chronologically.
Serial Execution Anti-Pattern: Displays a stair-step cascade where Microservice A calls Service B (100ms), waits for completion, calls Service C (150ms), waits, and calls Service D (120ms). The total latency is 370ms. Refactoring the code to invoke independent services concurrently using asynchronous parallel calls (e.g., Go goroutines with sync.WaitGroup or JavaScript Promise.all ) collapses the overall latency to the duration of the slowest single call (~150ms).
Database Bottleneck Identification: Pinpoints slow individual SQL query spans, revealing lock contention or missing foreign key indexes.
Trace Comparison (Diff Analysis)
Cloud Trace allows developers to select two different time ranges or compare two deployment revisions (e.g., Cloud Run Canary Revision v2 vs. Baseline Revision v1) to visualize latency regressions before promoting new code to 100% of user traffic.
Trace Sampling Strategies: Head-Based vs. Tail-Based & Cost Management
Trace billing depends on the current ingestion path and pricing schedule; check the official observability pricing rather than assuming an old per-span price applies to every ingestion API. In an enterprise system serving 50,000 requests per second across 10 microservices, sampling 100% of requests would generate about 1.3 trillion spans in a 30-day month (assuming ten spans per request), resulting in massive operational costs and unnecessary storage overhead.
Sampling Strategy Decision Point Mechanism Trade-offs & Exam Context
Head-Based Sampling (Standard OTel SDK) Made at the ingress / root span before request execution Probabilistic ratio (e.g., TraceIdRatioBasedSampler(0.01) = 1%) or rate limiter (e.g., 5 traces/sec). Decision is recorded in traceparent flags ( 01 vs 00 ) and propagated to all child services. Lowest overhead and predictable cost. Configure downstream ParentBased sampling to honor the propagated decision. Disadvantage: Rare 1-in-a-million errors or sudden latency spikes that occur on unsampled requests are permanently lost.
Tail-Based Sampling (Collector Layer) Made at the end of transaction after full execution completes An intermediate OpenTelemetry Collector cluster buffers all spans in memory until the entire trace completes. The collector evaluates trace duration and status: sample 100% of traces with errors ( status == ERROR ) or latency > 1,500ms , while sampling healthy traces at 0.1%. Maximum observability fidelity. Can preferentially retain errors/slow traces that reach the collector. Disadvantage: Requires provisioning and managing an OpenTelemetry Collector cluster with substantial RAM to buffer high-throughput in-flight spans.