Study guide
10.3 Distributed Tracing with AWS X-Ray & Service Maps
AWS Certified DevOps Engineer – Professional Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CI/CD Pipeline Architecture & Source Integration
1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models
2 Chapter 2: Automated Testing, Test Gates & Quality Verification
2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates
3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies
3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda
4 Chapter 4: CloudFormation, CDK & Reusable IaC Components
4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance
5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower
5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation
6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration
6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig
7 Chapter 7: High Availability & Multi-Region Resiliency
7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR
8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation
8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery
9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation
9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions
10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray
10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps
11 Chapter 11: Automated Observability & Event-Driven Monitoring
11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries
12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines
12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling
13 Chapter 13: Automated Remediation & Deployment Troubleshooting
13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures
14 Chapter 14: Identity, Access Management & Secrets at Scale
14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager
15 Chapter 15: Security Automation, Data Protection & Compliance Auditing
15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config
10.3 Distributed Tracing with AWS X-Ray & Service Maps
Key Takeaways Explain This Topic AI
AWS X-Ray provides end-to-end distributed tracing across microservices, structuring telemetry into Traces, Segments (compute hosts), Subsegments (downstream calls/custom logic), Annotations, and Metadata.
Annotations are indexed key-value pairs that support search and filtering in the X-Ray console and filter expressions, whereas Metadata consists of non-indexed arbitrary JSON objects that provide contextual diagnostic data without search capabilities.
X-Ray Sampling Rules dynamically regulate trace ingestion volume and costs through a two-tiered model: a reservoir size guaranteeing a fixed number of sampled traces per second, and a fixed rate percentage applied to requests exceeding the reservoir.
The X-Ray Daemon listens locally on UDP port 2000 for non-blocking trace segments emitted by application SDKs and batches trace uploads to the X-Ray API over HTTPS port 443 using the AWSXRayDaemonWriteAccess IAM policy.
CloudWatch ServiceLens unifies metrics, logs, and distributed traces into a correlated service topology, allowing engineers to drill down from high-level Service Map latency nodes directly into underlying CloudWatch Logs and segment waterfall timelines.
Last updated: September 2026
Distributed Tracing & The Microservices Observability Challenge
In modern cloud-native architectures, a single incoming user request frequently cascades through an Application Load Balancer, an Amazon API Gateway, multiple decoupled microservices hosted on AWS Lambda or Amazon ECS/EKS, asynchronous Amazon SQS queues, and distributed databases like Amazon DynamoDB or Aurora. When an end-to-end transaction degrades or fails with an intermittent timeout, traditional log analysis is insufficient because logs are fragmented across dozens of independent log streams with disparate clocks and formats.
AWS X-Ray solves this challenge by providing distributed request tracing, request context propagation, latency profiling, and dynamic service dependency mapping across distributed cloud workloads.
AWS X-Ray Architecture: Traces, Segments, Subsegments, Annotations & Metadata
To effectively instrument applications and analyze traces on the AWS Certified DevOps Engineer Professional exam, engineers must master the X-Ray data model hierarchy. Trace (Unique ID: 1-5759e988-bd862e3fe1be46a994272767)
- │
- ├── Segment: API Gateway (Resource, HTTP Status, Timestamps)
- │
- └── Segment: OrderService (Host: ECS Task Container)
- │
- ├── Subsegment: DynamoDB PutItem (Downstream AWS SDK Call)
- │ ├── Annotations: { "CustomerId": "C-8921", "OrderTier": "Platinum" }
- │ └── Metadata: { "ItemPayload": { ... }, "RetryAttempts": 0 }
- │
- └── Subsegment: PaymentServiceClient (Downstream HTTP Call)
- └── Subsegment: Custom Business Validation (Code Block)
- 1. Trace
A Trace represents the complete, end-to-end execution path of an individual request through the entire application ecosystem. It is uniquely identified by a globally unique Trace ID formatted as: 1-[epoch-timestamp-8-hex-digits]-[random-number-24-hex-digits]
- Example: 1-5759e988-bd862e3fe1be46a994272767
- 2. Segment
A Segment records operational telemetry for the compute resource hosting the application component handling the request (e.g., an EC2 instance, ECS container, or Lambda execution environment). The segment captures host identity (hostname, IP address, AMI ID), HTTP request details (method, client IP, user agent), response attributes (status code, duration), and exception/fault indicators.
3. Subsegment
A Subsegment provides granular, nested timing and diagnostic breakdown within a segment. Subsegments track:
Outbound HTTP/HTTPS requests to internal microservices or third-party APIs.
Downstream AWS SDK API calls (such as writing to Amazon S3, reading from Amazon DynamoDB, or publishing to Amazon SNS).
SQL query executions against relational databases (recording sanitized query strings, connection IDs, and transaction wait times).
Custom code blocks, algorithmic calculations, or internal method execution times.
4. Annotations vs. Metadata (Critical Exam Distinction)
Both annotations and metadata allow developers to inject custom attributes into X-Ray trace subsegments, but their indexing and query behaviors are fundamentally different:
- Feature Annotations Metadata
- Indexing Indexed by the X-Ray service Non-indexed
Searchability Fully searchable in the X-Ray console and via Filter Expressions Cannot be searched or queried; viewable only when inspecting the individual trace payload
Data Structure Simple key-value pairs (String, Number, Boolean) Arbitrary objects, complex nested JSON, arrays, full payloads
Primary Use Case Business dimensions for slicing and filtering traces (e.g., CustomerId , TenantId , OrderType , Environment ) Detailed diagnostic payloads, stack traces, request parameters, debugging context
Filter Expression Syntax annotation.CustomerId = "C-8921" Not supported in filter expressions
Important
DOP-C02 Exam Trap : If a scenario requires engineers to search, group, or filter traces in the AWS X-Ray console based on user tier, transaction ID, or shopping cart value, you must use Annotations . If the scenario asks to store large diagnostic data blobs or complex JSON payloads without search capability, use Metadata .
Centralized Sampling Rules
Tracing every single request in a high-throughput production environment (processing tens of thousands of requests per second) generates unsustainable telemetry storage costs and unnecessary CPU overhead. AWS X-Ray uses Sampling Rules to dynamically balance statistical confidence against cost. {
- "SamplingRule": {
- "RuleName": "HighVolumeOrderCheckout",
- "Priority": 10,
- "ReservoirSize": 50,
- "FixedRate": 0.05,
- "ServiceName": "CheckoutService",
- "ServiceType": "*",
- "Host": "api.enterprise.com",
- "HTTPMethod": "POST",
- "URLPath": "/v2/orders/*",
- "Version": 1
- }
- }
- Sampling Mechanics
Reservoir Size : The target number of matching traces to record per second before applying the fixed rate. For centralized sampling rules, the reservoir is allocated cumulatively across services through sampling targets; it is not a guaranteed per-host minimum. Before a client receives a quota, it can borrow one trace per second from a nonzero reservoir and apply the fixed rate to additional requests.
Fixed Rate : The percentage of matching requests sampled after the reservoir capacity is exhausted (e.g., 0.05 = 5% of subsequent requests).
Centralized Management : Sampling rules can be defined globally in the AWS X-Ray console or via the CreateSamplingRule API. Applications and X-Ray daemons poll the X-Ray API periodically (every few seconds) to fetch updated rules without requiring service restarts or code redeployments.
Rule Evaluation Order : Rules are evaluated strictly in ascending order of their numerical Priority (e.g., Priority 1 is evaluated before Priority 100 ). The default fallback rule has the lowest priority (Priority 10000 ).
X-Ray Daemon Architecture & Networking
The AWS X-Ray Daemon is an open-source software application that receives raw segment documents emitted by application SDKs and forwards them in optimized batches to the AWS X-Ray API. [ Application Process ]
- (X-Ray SDK Client)
- │
- │ UDP 2000 (Non-blocking, raw JSON segments)
- ▼
- [ AWS X-Ray Daemon ]
- │
- │ HTTPS 443 (Batched PutTraceSegments / PutTelemetryRecords)
- ▼
- [ AWS X-Ray Service Backend ]
- Key Architectural Characteristics
Transport Protocol (UDP 2000) : The X-Ray SDK communicates with the local daemon over UDP port 2000 . UDP ensures that application request handling is non-blocking. If the daemon becomes unavailable or crashes, the application continues serving client requests without latency penalties or thread pool exhaustion (dropped segments are recorded in SDK client metrics).
API Uploads (HTTPS 443) : The daemon buffers segments locally and transmits them in compressed batches to the regional X-Ray backend via the xray:PutTraceSegments and xray:PutTelemetryRecords APIs.
Daemon Configuration Parameters :
--bind-address 0.0.0.0:2000 : Binds the UDP listener to all network interfaces (essential when running as a container sidecar or DaemonSet).
--local-mode : Disables EC2 instance metadata lookups when running outside of AWS EC2 (e.g., on-premises or local test environments).
--log-level : Configures daemon diagnostic verbosity ( debug , info , warn , error ).
IAM Permissions : The IAM role assigned to the compute host, ECS task execution role, or EKS node must grant the AWS-managed policy AWSXRayDaemonWriteAccess (or least-privilege actions xray:PutTraceSegments , xray:PutTelemetryRecords , and xray:GetSamplingRules ).
Workload Instrumentation & Context Propagation
Distributed tracing relies on propagating a shared trace context header across network boundaries. AWS uses the X-Amzn-Trace-Id HTTP header: X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272767;Parent=53995cbe3140f4ec;Sampled=1
Root : The globally unique Trace ID generated by the ingress service.
Parent : The Segment ID of the upstream calling service.
Sampled : Tracing decision flag ( 1 = sampled, 0 = not sampled, ? = decision pending).
Platform-Specific Deployment Patterns ┌───────────────────────────────┬──────────────────────────────────────────────────────────────┐
- │ AWS Compute Platform │ X-Ray Deployment & Instrumentation Architecture │
- ├───────────────────────────────┼──────────────────────────────────────────────────────────────┤
- │ AWS Lambda │ - Enable Active Tracing: TracingConfig: Mode: Active │
- │ │ - Runtime automatically provisions and manages internal daemon │
- │ │ - Wrap AWS SDK clients in code: AWSXRay.captureAWS(require) │
- ├───────────────────────────────┼──────────────────────────────────────────────────────────────┤
- │ Amazon API Gateway │ - Enable Active Tracing on stage: tracingEnabled: true │
- │ │ - Automatically generates X-Amzn-Trace-Id and passes it │
- ├───────────────────────────────┼──────────────────────────────────────────────────────────────┤
- │ Amazon ECS (EC2 & Fargate) │ - Deploy X-Ray daemon as a Sidecar Container in the Task │
- │ │ Definition (image: public.ecr.aws/xray/aws-xray-daemon) │
- │ │ - Expose port 2000/udp on localhost container networking │
- ├───────────────────────────────┼──────────────────────────────────────────────────────────────┤
- │ Amazon EKS │ - Deploy X-Ray daemon as a Kubernetes DaemonSet (1 per node) │
- │ │ or adopt AWS Distro for OpenTelemetry (ADOT) Collector │
- │ │ - Application pods send UDP traffic to host node IP │
- └───────────────────────────────┴──────────────────────────────────────────────────────────────┘
- Service Map Visualization & CloudWatch ServiceLens
- Service Map Analysis
The AWS X-Ray Service Map visualizes the dynamic dependency graph of an application architecture, automatically discovered from trace segment telemetry:
Nodes : Represent services, compute platforms, databases, load balancers, or downstream APIs.
Edges : Represent network requests and invocations connecting the nodes.
Node Colors & Health Rings :
Green : Successful requests (2xx OK).
Yellow : Client errors (4xx response codes, excluding 429).
Red : Server faults (5xx response codes).
Purple : Rate-limiting throttles (HTTP 429 Too Many Requests).
Filter Expressions for Rapid Triage
Engineers isolate production regressions using the X-Ray Filter Expression language:
- Identify slow DynamoDB queries: service("DynamoDB") AND responsetime > 1.5
- Find downstream 5xx faults on order checkout: service("OrderService") { fault }
- Search by business annotation: annotation.TenantTier = "Enterprise" AND error = true
- CloudWatch ServiceLens: Unified Observability
Historically, engineers had to pivot between CloudWatch Metrics (graphs), CloudWatch Logs (log groups), and AWS X-Ray (traces). CloudWatch ServiceLens unifies these three pillars into an integrated observability topology:
Highlights high-latency or failing nodes directly on an interactive map.
Selecting any service node immediately reveals correlated CloudWatch metrics (requests, latency, 4xx/5xx errors) and corresponding CloudWatch Logs log streams.
Provides seamless one-click transition from an anomalous latency spike on a metric graph directly to the exact X-Ray trace waterfalls representing the regression.
DOP-C02 Exam Watchouts & Troubleshooting
Issue / Symptom Root Cause Remediation Protocol
ECS application logs report connect: connection refused or UDP socket send error to port 2000 The X-Ray daemon sidecar container is missing from the ECS task definition, or container port 2000/udp is not exposed Add the aws-xray-daemon container to the ECS task definition; configure port mapping for 2000/udp ; ensure containers share localhost networking
Custom fields injected via putMetadata() cannot be searched in the X-Ray console Metadata is non-indexed and does not support filter expressions Update the application code to use putAnnotation(key, value) with string, number, or boolean values
Lambda function downstream DynamoDB calls do not appear as subsegments Active tracing is enabled in Lambda, but the AWS SDK client was not wrapped with the X-Ray SDK Wrap the AWS SDK client in the Lambda handler initialization code (e.g., AWSXRay.captureAWSClient(dynamoClient) )
X-Ray daemon logs report AccessDeniedException: User is not authorized to perform: xray:PutTraceSegments The IAM execution role attached to EC2/ECS/EKS lacks X-Ray write permissions Attach the AWS-managed IAM policy AWSXRayDaemonWriteAccess to the instance profile or ECS Task Role
Traces from API Gateway to downstream Lambda show broken or disconnected segments Trace header X-Amzn-Trace-Id was not propagated or active tracing was disabled on the API Gateway stage Enable tracingEnabled: true on the API Gateway stage deployment; ensure custom upstream proxies do not strip the X-Amzn-Trace-Id header
Current Instrumentation Direction: OpenTelemetry
The AWS X-Ray service, trace storage, service maps, sampling concepts, and X-Ray APIs remain supported . The lifecycle change applies to the original X-Ray language SDKs and standalone daemon: they entered maintenance mode on February 25, 2026, with releases limited to security fixes, and AWS recommends OpenTelemetry for new instrumentation.
For new workloads, use native OpenTelemetry or AWS Distro for OpenTelemetry (ADOT), send telemetry through an OpenTelemetry Collector or the CloudWatch agent, and export traces to X-Ray/CloudWatch. Existing X-Ray SDK applications can continue sending traces, but a modernization plan should migrate the instrumentation and daemon rather than incorrectly treating the X-Ray backend itself as retired. This distinction is a likely exam trap whenever one option removes distributed tracing entirely and another updates only the collection layer.