ExamFinch.com My Study

Study guide

8.2 Tail-Based Sampling vs Head-Based Sampling

Practice this exam

OpenTelemetry Certified Associate OTCA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & Observability Foundations

1.1 OTCA Exam Structure & Observability Paradigms 1.2 Core Telemetry Signals: Traces, Metrics, Logs & Baggage 1.3 Telemetry Analysis, SLOs & High Cardinality

2 Chapter 2: Semantic Conventions & Instrumentation Strategy

2.1 OpenTelemetry Semantic Conventions & Namespaces 2.2 Resource Attributes & Entity Modeling 2.3 Zero-Code vs Code-Based Instrumentation

3 Chapter 3: OpenTelemetry Architecture & The Data Model

3.1 Architectural Separation of API and SDK 3.2 The Core Trace Model: Spans, SpanContext & Links 3.3 Span Events, Attributes & Status Codes 3.4 Composability & Extension: Plugin Interfaces, Distributions & Contrib

4 Chapter 4: Distributed Tracing & Context Propagation

4.1 In-Process & Cross-Process Context Propagation 4.2 W3C Trace Context Specification 4.3 W3C Baggage & Alternative Propagators

5 Chapter 5: OpenTelemetry Metrics API & Instruments

5.1 Synchronous Metric Instruments 5.2 Asynchronous (Observable) Metric Instruments 5.3 Metric Aggregations & Temporality

6 Chapter 6: Logs Data Model & Logging Bridge API

6.1 OpenTelemetry Logs Data Model & Architecture 6.2 The Logging Bridge API & Log Appenders

7 Chapter 7: SDK Pipelines, Processors & Exporters

7.1 Span Processors & Exporters 7.2 Metric Readers & Log Processors 7.3 Standard Exporters & OTLP Wire Protocols

8 Chapter 8: Sampling Strategies, SDK Configuration & Agents

8.1 Head-Based Sampling & Built-in Samplers 8.2 Tail-Based Sampling vs Head-Based Sampling 8.3 OpenTelemetry SDK Configuration & Environment Variables 8.4 Zero-Code Agents: Configuration, Kubernetes Injection & Troubleshooting

9 Chapter 9: OpenTelemetry Collector Architecture & Pipelines

9.1 Collector Core Architecture & Component Lifecycle 9.2 Collector Configuration Structure & Pipelines 9.3 Essential Collector Extensions

10 Chapter 10: Collector Components: Receivers, Processors & Exporters

10.1 Standard Receivers & Ingestion Protocols 10.2 Core Processors & Execution Order 10.3 Standard Exporters & Delivery Guarantees

11 Chapter 11: Data Transformation & The OpenTelemetry Transformation Language (OTTL)

11.1 OTTL Concepts & Syntax Grammar 11.2 Transform Processor In Action 11.3 Advanced Connectors

12 Chapter 12: Collector Deployment, Scaling & Resilience

12.1 Agent vs Gateway Deployment Topologies 12.2 Scaling, Load Balancing & High Availability 12.3 Resilience: Backpressure, Buffering & Persistent Queues

13 Chapter 13: Maintaining, Debugging & Schema Evolution

13.1 Debugging Collector Pipelines & Telemetry Health 13.2 Troubleshooting Broken Traces & Context Loss 13.3 Schema Management & Telemetry Evolution

8.2 Tail-Based Sampling vs Head-Based Sampling

Key Takeaways Explain This Topic AI

Head-based sampling inherently discards the vast majority of operational anomalies: a 1% head-sampling rate discards 99% of downstream errors, unexpected timeouts, and latency spikes.

Tail-based sampling defers sampling decisions until all spans belonging to a trace have finished execution, allowing policies to retain traces based on errors, HTTP status codes, latency thresholds, or VIP tenant attributes.

Tail-based sampling cannot run inside application SDKs because child spans execute across distributed nodes and buffering full traces in application memory poses severe garbage collection and crash risks.

The OpenTelemetry Collector implements tail-based sampling via the tail_sampling processor, which buffers spans across a configurable decision window before evaluating multi-rule policies.

Tail-based sampling strictly requires trace co-location: all spans sharing a TraceId must route to the identical Collector instance using a two-tier architecture with a trace-id-aware loadbalancing exporter.

Last updated: September 2026

8.2 Tail-Based Sampling vs Head-Based Sampling

Quick Answer: Tail-based sampling defers the decision to sample or drop a trace until the very end ("tail") of the request lifecycle, after all child spans have finished executing and all telemetry attributes, status codes, and durations are known. This solves the fundamental flaw of head-based sampling, where probabilistic sampling drops 99% of rare errors and slow requests. Because distributed traces span multiple microservice runtimes, tail-based sampling cannot be implemented inside a single application SDK; it requires an out-of-process intermediate pipeline—specifically the OpenTelemetry Collector's tail_sampling processor—coupled with a trace-id-aware loadbalancing exporter to route all spans of a trace to the identical Collector instance.

While head-based sampling delivers exceptional performance and minimal CPU overhead at the application runtime, it imposes a severe observability compromise. In cloud environments where 99.9% of user requests succeed normally and only 0.1% experience catastrophic failure or unacceptable tail latency, probabilistic head sampling operates blindly, dropping the vast majority of the very incidents on-call engineers need to diagnose. Resolving this conflict requires shifting the decision boundary from the application ingress to an intelligent, out-of-process sampling tier.

The Mathematical Dilemma of Head-Based Sampling

To understand why tail-based sampling is an indispensable requirement for enterprise systems, consider the mathematical reality of probabilistic head-based sampling in high-scale architectures.

The Anomaly Blind Spot

Suppose an e-commerce platform processes 10,000,000 requests per day . The system maintains a healthy 99.9% success rate, meaning that 0.1% of transactions (10,000 requests) experience an unhandled exception, database lock timeout, or HTTP 500 internal server error.

To control tracing ingestion and storage expenses, the engineering team configures head-based sampling at a 1% ratio ( TraceIdRatioBased(0.01) wrapped in ParentBased ):

Total Traces Sampled : 10,000,000 × 0.01 = 100,000 traces recorded and exported.

Error Traces Captured : 10,000 × 0.01 = 100 error traces captured.

Error Traces Discarded : 10,000 × 0.99 = 9,900 error traces permanently lost! Head-Based Sampling (1% Probabilistic Ratio):

  • Total Errors Occurred: [ 10,000 Failure Events ]
  • ├── Captured in Backend: 100 Traces ( 1% ) <-- Tiny statistical sample

└── Dropped at Ingress: 9,900 Traces (99% ) <-- Complete blind spot!

Because the sampling decision was locked in at the root span when the request first touched the API Gateway, 99 out of every 100 failure events were discarded before the error ever occurred. If an SRE investigates an incident that impacted 50 users in a specific geographical region, there is a high probability that zero traces were captured for that incident.

To guarantee capturing all 10,000 error traces using head-based sampling, the platform would have to increase the sampling ratio to 100% , forcing the organization to ingest, process, and pay for 9,990,000 uninteresting "200 OK" traces.

Mechanics of Tail-Based Sampling: Retrospective Decision Making

Tail-based sampling flips the sampling paradigm by deferring the decision until the transaction has completed across all participating microservices. Microservices A, B, C emit 100% of spans via OTLP

  • │
  • ▼
  • ┌────────────────────────────────┐
  • │ OpenTelemetry Collector Buffer │
  • │ Holds spans for decision_wait │
  • │ (e.g., 30-second trace window) │
  • └───────────────┬────────────────┘
  • │
  • ▼

Evaluate All Spans:

  • ├── Any span status == ERROR? ──► KEEP (100% Export)
  • ├── Any span duration > 2000ms? ──► KEEP (100% Export)
  • ├── Any http.response.status_code == 500? ──► KEEP (100% Export)
  • ├── customer.tier == "enterprise"? ──► KEEP (100% Export)
  • └── Normal 200 OK fast traces ──► SAMPLE at 0.1% Ratio
  • The Core Workflow

Full Generation : Application microservices do not drop spans at ingress. Every service creates spans for 100% of requests and exports them immediately over the local network via OTLP.

In-Memory Buffering : An intermediate collector cluster ingests the spans and holds them in an in-memory buffer for a configurable evaluation window (the decision_wait duration, typically 10 to 30 seconds).

DAG Assembly : Spans sharing the same TraceId are assembled into their respective trace tree.

Policy Inspection : Once the decision window expires or the trace root completes, a comprehensive policy engine inspects every span in the trace.

Selective Retention : If any span in the trace matches an error rule, a latency threshold, or a high-value customer attribute, the entire distributed trace (root span to leaf span) is forwarded to storage. If the trace represents a routine, fast success, it is sampled down to a tiny fraction (e.g., 0.1%) or discarded.

Architectural Requirements: Why Tail Sampling Belongs in the Collector

A natural question is: Why can't tail-based sampling be implemented directly inside the application SDK?

There are two insurmountable architectural reasons:

Microservice Memory Isolation : In a distributed system, a single trace traverses multiple distinct operating system processes, container pods, and serverless runtimes. Service A (the API Gateway) completes its root span in 15 milliseconds and sends the response to the user. Service C (the background billing worker) might encounter a database lock 4 seconds later on a completely different Kubernetes node. Service A cannot look into Service C's memory space to determine whether Service C will encounter an error.

Application Memory Protection : Buffering thousands of spans in application heap memory while waiting for asynchronous distributed transactions to finish introduces massive garbage collection overhead, memory bloat, and the risk of Out-Of-Memory (OOM) application crashes. Telemetry processing must never destabilize core business applications.

Therefore, tail-based sampling is an out-of-process concern . It is implemented in the intermediate telemetry tier using the OpenTelemetry Collector —specifically using the tail_sampling processor available in the OpenTelemetry Collector Contrib distribution.

Configuration of the tail_sampling Processor

The tail_sampling processor in the OpenTelemetry Collector Contrib repository exposes rich policy primitives: processors:

tail_sampling:

  • decision_wait: 30s
  • num_traces: 50000
  • expected_new_traces_per_sec: 2000

policies:

  • # Policy 1: Retain 100% of traces with an explicit Error status
  • name: drop-errors-never
  • type: status_code
  • status_code: { status_codes: [ ERROR ] }
  • # Policy 2: Retain 100% of slow requests (tail latency > 2500ms)
  • name: slow-traces-filter
  • type: latency
  • latency: { threshold_ms: 2500 }
  • # Policy 3: Retain 100% of server errors via HTTP status code
  • name: http-5xx-rules
  • type: numeric_attribute

numeric_attribute:

  • key: http.response.status_code
  • min_value: 500
  • max_value: 599
  • # Policy 4: Retain all enterprise VIP customer requests
  • name: vip-tenants
  • type: string_attribute

string_attribute:

  • key: customer.tier
  • values: [ enterprise, platinum ]
  • # Policy 5: Sample remaining routine 200 OK traces at 1%
  • name: routine-traffic-probabilistic
  • type: probabilistic
  • probabilistic: { sampling_percentage: 1.0 }

decision_wait (default 30s): How long the Collector buffers a trace, counted from its first span, before evaluating the policies. Spans that arrive later follow the stored decision while the trace is still in memory; decision_cache settings keep decisions (not spans) for longer so that late spans still follow them.

num_traces (default 50000): The maximum number of distinct traces held in memory; when it is exceeded, the oldest traces are evicted, which bounds memory but can drop data under heavy load.

Policy logic : A trace is kept if any policy decides to sample it; and policies require several conditions together, and drop policies explicitly discard matching traces.

expected_new_traces_per_sec : Pre-allocates internal hash table memory to eliminate dynamic allocation pauses under heavy ingress.

The Trace Co-Location Problem & The Two-Tier Architecture

The central engineering challenge of tail-based sampling in a scaled production environment is trace co-location .

The Fatal Flaw of Standard Round-Robin Load Balancing

Suppose you deploy three instances of the OpenTelemetry Collector ( Collector 1 , Collector 2 , Collector 3 ) behind a standard Kubernetes ClusterIP Service or AWS Application Load Balancer:

Service A generates the root span for TraceId: abc-123 and sends it to the Kubernetes service VIP. The load balancer routes it to Collector 1 .

Service B generates a child span for TraceId: abc-123 and sends it to the VIP. The load balancer routes it to Collector 2 .

Service C generates a child span that throws an HTTP 500 error for TraceId: abc-123 . The load balancer routes it to Collector 3 . Kubernetes Service (Round-Robin)

  • │
  • ┌──────────────────────────┼──────────────────────────┐
  • ▼ ▼ ▼
  • Collector 1 Collector 2 Collector 3
  • ┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐
  • │ Root Span │ │ Child Span │ │ Error Span │
  • │ (Trace abc-123) │ │ (Trace abc-123) │ │ (Trace abc-123) │
  • │ │ │ │ │ │
  • │ Decision: DROP! │ │ Decision: DROP! │ │ Decision: KEEP! │
  • │ (No error seen) │ │ (No error seen) │ │ (Error seen) │
  • └────────────────────┘ └────────────────────┘ └────────────────────┘

The Disaster : Because the spans for TraceId: abc-123 were scattered across three different collector memories:

Collector 1 evaluates only the root span. It sees no error and drops the root span!

Collector 2 evaluates only the middle span and drops it!

Collector 3 sees the error and decides to keep its span—but when it exports to the backend, the trace has no root span, no HTTP request URL, and no client metadata ! The trace is completely severed.

The Mandatory Solution: Two-Tier Collector Topology

To make tail-based sampling function in a horizontally scalable architecture, all spans sharing the same TraceId must arrive at the exact same Collector instance . This requires deploying a two-tier collector topology using the loadbalancing exporter :

Tier 1: Agent / Ingress Tier (Stateless Routing)

Deployed as a Kubernetes DaemonSet on every node or as a local container sidecar.

Applications send OTLP to the node-local agent (through the node's host IP) or, with sidecars, to localhost:4317 / localhost:4318 .

Tier 1 collectors do not run the tail_sampling processor. Instead, they configure the loadbalancing exporter .

The loadbalancing exporter reads the TraceId of every span, computes a consistent hash, and routes all spans with that TraceId to a deterministic member of Tier 2.

Tier 2: Tail-Sampling Gateway Tier (Stateful Evaluation)

Deployed as an independent, scalable pool of OpenTelemetry Collector gateway instances.

Every span belonging to TraceId: abc-123 lands exclusively on Gateway Instance 2 .

Gateway Instance 2 buffers the spans, reconstructs the full distributed trace DAG, executes the tail-sampling policies, and exports complete traces to the storage backend.

Loading diagram... Two-Tier Collector Deployment with Trace-Aware Load Balancing

Head-Based vs. Tail-Based Sampling: Comprehensive Comparison

The following comparison table summarizes the technical differences, resource profiles, and architectural trade-offs:

Evaluation Dimension Head-Based Sampling Tail-Based Sampling

Decision Point At the ingress ("head") when root span is created At the conclusion ("tail") after all child spans complete

Implementation Location Inside the application process via OpenTelemetry SDK Out-of-process in intermediate OpenTelemetry Collectors

Anomaly & Error Capture Very poor; drops errors proportionally to sampling ratio (e.g., 1% ratio drops 99% of errors) Very strong; policies can keep every error or slow trace, within the Collector's memory limits ( num_traces )

Application CPU Overhead Extremely low (single hash or flag check at root) Low (spans emitted immediately without local evaluation)

Application Memory Usage Negligible (dropped spans discard allocations immediately) Negligible (spans flushed asynchronously to local agent)

Collector Memory Usage Minimal (spans stream through pipeline without buffering) High (requires memory to buffer millions of spans across decision_wait window)

Network Egress (App → Collector) Reduced (dropped traces never leave application memory) High (100% of all generated spans must be transmitted to Tier 1 collectors)

Network Egress (Collector → Storage) Directly proportional to sampling ratio Highly optimized (only valuable, anomalous, or selected traces exported)

Deployment Complexity Simple (configured via SDK environment variables) Complex (requires two-tier collector topology with trace-id-aware load balancing)

Latency Impact on Export Zero delay (spans exported immediately upon completion) Delayed by decision_wait duration (typically 10 to 30 seconds)

Best Suited For Homogeneous low-scale systems, dev environments, microservices with tight egress budgets Large-scale enterprise architectures, mission-critical e-commerce, strict SLA/SLO monitoring