Study guide
12.2 Scaling, Load Balancing & High Availability
OpenTelemetry Certified Associate OTCA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & Observability Foundations
1.1 OTCA Exam Structure & Observability Paradigms 1.2 Core Telemetry Signals: Traces, Metrics, Logs & Baggage 1.3 Telemetry Analysis, SLOs & High Cardinality
2 Chapter 2: Semantic Conventions & Instrumentation Strategy
2.1 OpenTelemetry Semantic Conventions & Namespaces 2.2 Resource Attributes & Entity Modeling 2.3 Zero-Code vs Code-Based Instrumentation
3 Chapter 3: OpenTelemetry Architecture & The Data Model
3.1 Architectural Separation of API and SDK 3.2 The Core Trace Model: Spans, SpanContext & Links 3.3 Span Events, Attributes & Status Codes 3.4 Composability & Extension: Plugin Interfaces, Distributions & Contrib
4 Chapter 4: Distributed Tracing & Context Propagation
4.1 In-Process & Cross-Process Context Propagation 4.2 W3C Trace Context Specification 4.3 W3C Baggage & Alternative Propagators
5 Chapter 5: OpenTelemetry Metrics API & Instruments
5.1 Synchronous Metric Instruments 5.2 Asynchronous (Observable) Metric Instruments 5.3 Metric Aggregations & Temporality
6 Chapter 6: Logs Data Model & Logging Bridge API
6.1 OpenTelemetry Logs Data Model & Architecture 6.2 The Logging Bridge API & Log Appenders
7 Chapter 7: SDK Pipelines, Processors & Exporters
7.1 Span Processors & Exporters 7.2 Metric Readers & Log Processors 7.3 Standard Exporters & OTLP Wire Protocols
8 Chapter 8: Sampling Strategies, SDK Configuration & Agents
8.1 Head-Based Sampling & Built-in Samplers 8.2 Tail-Based Sampling vs Head-Based Sampling 8.3 OpenTelemetry SDK Configuration & Environment Variables 8.4 Zero-Code Agents: Configuration, Kubernetes Injection & Troubleshooting
9 Chapter 9: OpenTelemetry Collector Architecture & Pipelines
9.1 Collector Core Architecture & Component Lifecycle 9.2 Collector Configuration Structure & Pipelines 9.3 Essential Collector Extensions
10 Chapter 10: Collector Components: Receivers, Processors & Exporters
10.1 Standard Receivers & Ingestion Protocols 10.2 Core Processors & Execution Order 10.3 Standard Exporters & Delivery Guarantees
11 Chapter 11: Data Transformation & The OpenTelemetry Transformation Language (OTTL)
11.1 OTTL Concepts & Syntax Grammar 11.2 Transform Processor In Action 11.3 Advanced Connectors
12 Chapter 12: Collector Deployment, Scaling & Resilience
12.1 Agent vs Gateway Deployment Topologies 12.2 Scaling, Load Balancing & High Availability 12.3 Resilience: Backpressure, Buffering & Persistent Queues
13 Chapter 13: Maintaining, Debugging & Schema Evolution
13.1 Debugging Collector Pipelines & Telemetry Health 13.2 Troubleshooting Broken Traces & Context Loss 13.3 Schema Management & Telemetry Evolution
12.2 Scaling, Load Balancing & High Availability
Key Takeaways Explain This Topic AI
OpenTelemetry Collector workloads fall into two architectural categories: stateless pipelines (which scale linearly behind standard load balancers) and stateful pipelines (such as tail-based sampling, which require coordinated routing).
Traditional Layer 4 (TCP) and Layer 7 (HTTP) load balancers fail for tail-based sampling because spans belonging to the same distributed trace are scattered across different collector replicas, preventing any single collector from evaluating complete traces.
The loadbalancing exporter solves trace scatter by computing a consistent hash on the 128-bit TraceId of every span, deterministically routing all spans of a given trace to the identical downstream Gateway collector instance.
The loadbalancing exporter discovers downstream Collectors with the k8s resolver (EndpointSlices), the dns resolver (a headless Service's A records), the aws_cloud_map resolver, or a static list.
High availability in production requires deploying multiple Gateway replicas across availability zones with pod anti-affinity, configuring graceful rolling updates with readiness probes, and tuning Horizontal Pod Autoscalers (HPA) using CPU, memory, and custom collector metrics like otelcol_receiver_refused_spans.
Last updated: September 2026
12.2 Scaling, Load Balancing & High Availability
Quick Answer: Scaling an OpenTelemetry Collector fleet depends on whether the telemetry pipeline is stateless (such as batching, basic filtering, and OTLP export, which scale linearly behind standard Layer 4 or Layer 7 load balancers) or stateful (such as tail-based sampling and trace-to-metric aggregation, which require all spans of a trace to reach the identical collector instance). Because standard round-robin load balancers scatter spans across different pods, production architectures deploy the loadbalancing exporter on Tier 1 Agents to route spans using consistent hashing on TraceId . High availability requires multi-replica deployments with pod anti-affinity, graceful rolling updates, and Horizontal Pod Autoscalers (HPA) driven by refused-span metrics ( otelcol_receiver_refused_spans ).
As cloud-native architectures expand to thousands of microservices emitting millions of spans, metric data points, and log records per second, observability infrastructure must scale elastically without dropping critical telemetry. However, scaling an OpenTelemetry Collector fleet presents unique challenges that do not exist in conventional web application scaling. In particular, distributed traces consist of loosely coupled spans emitted across asynchronous microservices over time. Scaling collectors without understanding trace affinity can silently destroy data correlation and break sampling pipelines.
Stateless vs. Stateful Collector Workloads
To scale an OpenTelemetry Collector deployment effectively, platform engineers must categorize pipelines based on statefulness: +-------------------------------------------------------------------------+
- | Collector Pipeline Categorization |
- +-------------------------------------------------------------------------+
- STATELESS PIPELINES STATEFUL PIPELINES
- Ingestion (otlpreceiver) - Tail-based sampling (tail_sampling)
- Batching (batchprocessor) - Trace-to-metrics (spanmetrics)
- Memory limiting (memory_limiter) - Metric deduplication & rollup
- Stateless OTTL (transform) - Sliding window rate limiting
- Egress (otlpexporter) - Trace assembly & error detection
- ---------------------------------- ---------------------------------
- Scaling: Linear & trivial Scaling: Requires consistent routing
- Load Balancer: Standard L4/L7 Load Balancer: loadbalancing exporter
- Stateless Workloads
In a stateless pipeline, every telemetry record (an individual span, metric point, or log line) is processed independently. The components in the pipeline do not need to correlate records with previous or future records:
The batch processor groups whatever records arrive within a timeout window or byte size.
The memory_limiter processor monitors heap memory and applies backpressure.
The transform processor mutates attributes on individual records using OTTL.
The otlp exporter serializes and transmits batches.
Scaling Characteristics : Stateless collector pipelines scale linearly and horizontally. You can place N N N collector replicas behind a standard Kubernetes ClusterIP Service or cloud load balancer. If traffic doubles, simply scale the replica count from 5 to 10. Any collector pod can receive any telemetry record from any microservice.
Stateful Workloads
In a stateful pipeline, components must accumulate records over a time window to perform cross-record analysis or aggregation:
tail_sampling Processor : Holds spans in memory for a configurable duration (e.g., 10–30 seconds) to determine whether the entire distributed trace contained an error (HTTP 5xx), an exception event, or an abnormal duration (latency > 2s). If an error occurred anywhere in the trace, all spans of that trace are sampled; otherwise, the trace is dropped.
spanmetrics Connector : Computes request rates, error rates, and duration histograms (RED metrics) from span streams, requiring continuous aggregation across spans.
Metric Deduplication : Combines overlapping metric streams from multiple agents.
The Scaling Invariant : Stateful workloads cannot be scaled using standard round-robin or least-connection load balancing! If spans belonging to Trace A are scattered across Pod 1, Pod 2, and Pod 3, no single pod has full visibility into the trace. Pod 1 might evaluate its sampling decision before Pod 3 processes the database error span, resulting in premature trace deletion.
The Load Balancing Challenge with Traces (The Scatter Problem)
In a distributed microservice architecture, a single user transaction traverses multiple independent services. Each service generates one or more spans that share the exact same 128-bit TraceId : User Request
- │
- ▼
- [Frontend Service] --> generates Span 1 (TraceId: 4bf92f35...)
- │
- ▼
- [Payment Service] --> generates Span 2 (TraceId: 4bf92f35...)
- │
- ▼
- [Database Service] --> generates Span 3 (TraceId: 4bf92f35... - THROWS ERROR!)
- Why Traditional Layer 4 (TCP) Load Balancers Fail
Layer 4 load balancers (such as Kubernetes ClusterIP or AWS Network Load Balancer) operate at the transport layer, routing TCP/UDP connections:
OpenTelemetry SDKs and agents use gRPC (HTTP/2) for high-performance OTLP transport.
gRPC maintains long-lived, persistent TCP connections over which thousands of multiplexed RPC streams are transmitted.
A Layer 4 load balancer balances only at connection establishment time. Once Frontend Service connects to Gateway Pod 1, all of Frontend Service's spans stream to Pod 1.
When Payment Service connects, the L4 balancer assigns it to Gateway Pod 2. All Payment Service spans stream to Pod 2.
Database Service connects to Gateway Pod 3. All Database Service spans stream to Pod 3.
The Failure Mode : Gateway Pod 1 evaluates Trace 4bf92f35... using only Frontend Service's span. Gateway Pod 1 sees a successful HTTP 200 response from the frontend and drops the trace, completely blind to the fact that Database Service threw a fatal SQL deadlock on Pod 3!
Why Traditional Layer 7 (HTTP) Load Balancers Fail
Layer 7 load balancers (such as Envoy, NGINX, or AWS ALB) understand HTTP/2 frames and distribute individual RPC requests across backend targets:
The load balancer round-robins individual ExportTraceServiceRequest RPC calls across Gateway Pods A, B, and C.
Span 1 lands on Pod A; Span 2 lands on Pod B; Span 3 lands on Pod C.
Every Gateway pod receives a broken, fragmented slice of every trace.
Every pod's tail-sampling cache times out waiting for missing spans, leading to corrupted sampling decisions, disjointed trace graphs, and severe memory bloat.
The loadbalancing Exporter (OpenTelemetry Collector Contrib)
The OpenTelemetry community developed the loadbalancing exporter specifically to solve the trace scatter problem in horizontally scaled environments. It sits on Tier 1 Node Agents or on an ingress routing Gateway layer, acting as a smart, telemetry-aware traffic dispatcher. +-------------------------------------------------------------------------+
- | loadbalancing Exporter: Consistent Hashing |
- +-------------------------------------------------------------------------+
- Incoming Spans from Microservices
- Span A1 (TraceId: 4bf9...) ──────┐
- Span B1 (TraceId: 4bf9...) ──────┼──> [loadbalancing exporter]
- Span C1 (TraceId: 4bf9...) ──────┘ │
│ Evaluates TraceId:
- │ hash("4bf9...") % Ring
- ▼
- Routes 100% of "4bf9..." spans
to the EXACT SAME Gateway Pod!
- │
- ┌─────────────────────────────────────┼─────────────────────────┐
- ▼ ▼ ▼
- +─────────────────+ +─────────────────+ +─────────────────+
- | Gateway Pod 1 | | Gateway Pod 2 | | Gateway Pod 3 |
- | [Trace 4bf9...] | | [Trace 9ca2...] | | [Trace 7ef1...] |
- | All spans for | | | | |
- | trace co-located| | | | |
- | Tail sampling | | | | |
- | evaluates FULL | | | | |
- | trace tree! | | | | |
- +─────────────────+ +─────────────────+ +─────────────────+
- How It Works: Consistent Hashing on TraceId
Batch Inspection : When a batch of spans arrives at the Tier 1 Agent, the loadbalancing exporter iterates through every span in the batch.
Key Extraction : For each span, it extracts the 128-bit TraceId .
Consistent Hashing : It hashes the routing key (the TraceId by default for traces) onto a consistent-hash ring built from the downstream Gateway endpoints. Every Tier 1 instance with the same configuration and endpoint list computes the same mapping.
Batch Re-assembly and Dispatch : The exporter regroups spans destined for the same downstream Gateway instance into discrete OTLP gRPC batches and sends them directly to that specific pod's IP.
Deterministic Invariant : Every span sharing the exact same TraceId —no matter which microservice emitted it, which worker node it ran on, or what millisecond it was generated—is deterministically routed to the identical downstream Gateway Collector instance .
The Value of Consistent Hashing During Cluster Rescaling
When downstream Gateway pods scale up (e.g., from 4 pods to 5 pods) or scale down due to node termination, a naive modulo hash ( H m o d N H \bmod N H mod N ) would remap nearly 100% of all trace IDs, corrupting in-flight tail-sampling buffers across the entire fleet.
Because the loadbalancing exporter implements consistent hashing , adding or removing a Gateway replica remaps only about 1 / N 1/N 1/ N of the hash space, so roughly ( N − 1 ) / N (N-1)/N ( N − 1 ) / N of active traces stay pinned to their existing Gateway pods. Traces that do move during a scale event can still be split, so expect a small amount of disruption.
Downstream Routing Resolvers
The loadbalancing exporter discovers downstream Gateway instances with one of four resolvers:
k8s (Kubernetes Resolver) : Watches the EndpointSlice objects of a Service (given as name.namespace ) and updates the ring as Gateway pods become ready or terminate. The Collector's service account needs get , list , and watch permissions on EndpointSlices.
dns (DNS Resolver) : Resolves a hostname, typically a headless Kubernetes Service such as otel-gateway-headless.monitoring.svc.cluster.local , to all of its IP addresses, re-checking every interval (default 5s ).
static (Static Resolver) : Uses a fixed list of hostnames or IP addresses, suited for local development or fixed virtual machine clusters (an unavailable endpoint is not removed automatically).
aws_cloud_map : Discovers instances registered in AWS Cloud Map.
The routing_key chooses what is hashed: traceID (the default for traces), service , resource , metric , streamID , attributes , or randomness .
Production Configuration Example: loadbalancing Exporter receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
- check_interval: 1s
- limit_percentage: 80
- spike_limit_percentage: 20
batch:
- send_batch_size: 8192
- timeout: 1s
exporters:
loadbalancing:
routing_key: "traceID"
protocol:
otlp:
timeout: 5s
tls:
insecure: true
resolver:
k8s:
service: otel-gateway-headless.observability
ports:
- 4317
service:
pipelines:
traces:
- receivers: [otlp]
- processors: [memory_limiter, batch]
- exporters: [loadbalancing]
- High Availability (HA) Considerations in Kubernetes
Deploying a scalable OpenTelemetry Collector Gateway tier requires designing for node failures, zone outages, and rolling software updates:
1. Multi-Replica Redundancy & Pod Anti-Affinity
Never run a single Collector replica in production. Maintain a minimum of three replicas across separate availability zones. Configure Kubernetes podAntiAffinity so that the Kubernetes scheduler spreads Collector pods across distinct physical nodes and availability zones: affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
- operator: In
- values: ["opentelemetry-collector-gateway"]
- topologyKey: topology.kubernetes.io/zone
- weight: 90
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
- operator: In
- values: ["opentelemetry-collector-gateway"]
- topologyKey: kubernetes.io/hostname
- 2. Pod Disruption Budgets (PDB)
To protect the Collector tier during planned Kubernetes node drains, cluster upgrades, or autoscaling events, define a PodDisruptionBudget : apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
- name: otel-gateway-pdb
- namespace: observability
spec:
minAvailable: 2
selector:
matchLabels:
- app.kubernetes.io/name: opentelemetry-collector-gateway
- 3. Graceful Rolling Updates & Termination Handshakes
During a deployment update, Kubernetes sends a SIGTERM signal to terminating pods. If a Collector terminates abruptly, in-flight batches and network sockets are dropped:
Set terminationGracePeriodSeconds: 60 or higher in the pod specification.
When SIGTERM is received, the Collector executes its reverse lifecycle: receivers stop listening first, processors flush pending batches downstream, and exporters drain sending queues before process exit.
Configure Kubernetes Readiness Probes against the health_check extension on port 13133. Its default endpoint binds only to localhost, so set endpoint: 0.0.0.0:13133 (or the pod IP) for the kubelet to reach it. A pod only receives traffic once all internal pipelines are ready, and traffic is removed from the pod immediately upon termination.
4. Horizontal Pod Autoscaling (HPA) Triggers
Autoscaling Collector Gateways requires monitoring both infrastructure utilization and the Collector's internal metrics, served by default at http://127.0.0.1:8888/metrics (configure a readers entry with host 0.0.0.0 so a cluster Prometheus can scrape it):
Metric Name Source Operational Meaning & Autoscaling Action
CPU Utilization Container Runtime Scale out when average CPU exceeds 70–75% to prevent thread scheduling delays and ingestion latency.
Memory Utilization Container Runtime Scale out when memory exceeds 75% to prevent memory_limiter drops.
otelcol_receiver_refused_spans Collector Internal Critical Custom Metric : Non-zero values indicate the Collector is rejecting spans due to backpressure or memory exhaustion. Must trigger rapid scale-out!
otelcol_receiver_refused_metric_points Collector Internal Non-zero values indicate metrics are actively being dropped at ingress.
otelcol_exporter_queue_size Collector Internal High queue depth relative to queue_capacity indicates downstream backend saturation or network bottlenecks.
HPA Autoscaling Policy Best Practice : Configure HPA with a rapid scale-up policy (e.g., doubling replicas within 15 seconds upon a traffic spike) and a conservative, stabilized scale-down window (e.g., 300 seconds) to prevent pod flapping and cache thrashing during transient dips in telemetry volume.