Study guide

12.3 Resilience: Backpressure, Buffering & Persistent Queues

OpenTelemetry Certified Associate OTCA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & Observability Foundations

1.1 OTCA Exam Structure & Observability Paradigms 1.2 Core Telemetry Signals: Traces, Metrics, Logs & Baggage 1.3 Telemetry Analysis, SLOs & High Cardinality

2 Chapter 2: Semantic Conventions & Instrumentation Strategy

2.1 OpenTelemetry Semantic Conventions & Namespaces 2.2 Resource Attributes & Entity Modeling 2.3 Zero-Code vs Code-Based Instrumentation

3 Chapter 3: OpenTelemetry Architecture & The Data Model

3.1 Architectural Separation of API and SDK 3.2 The Core Trace Model: Spans, SpanContext & Links 3.3 Span Events, Attributes & Status Codes 3.4 Composability & Extension: Plugin Interfaces, Distributions & Contrib

4 Chapter 4: Distributed Tracing & Context Propagation

4.1 In-Process & Cross-Process Context Propagation 4.2 W3C Trace Context Specification 4.3 W3C Baggage & Alternative Propagators

5 Chapter 5: OpenTelemetry Metrics API & Instruments

5.1 Synchronous Metric Instruments 5.2 Asynchronous (Observable) Metric Instruments 5.3 Metric Aggregations & Temporality

6 Chapter 6: Logs Data Model & Logging Bridge API

6.1 OpenTelemetry Logs Data Model & Architecture 6.2 The Logging Bridge API & Log Appenders

7 Chapter 7: SDK Pipelines, Processors & Exporters

7.1 Span Processors & Exporters 7.2 Metric Readers & Log Processors 7.3 Standard Exporters & OTLP Wire Protocols

8 Chapter 8: Sampling Strategies, SDK Configuration & Agents

8.1 Head-Based Sampling & Built-in Samplers 8.2 Tail-Based Sampling vs Head-Based Sampling 8.3 OpenTelemetry SDK Configuration & Environment Variables 8.4 Zero-Code Agents: Configuration, Kubernetes Injection & Troubleshooting

9 Chapter 9: OpenTelemetry Collector Architecture & Pipelines

9.1 Collector Core Architecture & Component Lifecycle 9.2 Collector Configuration Structure & Pipelines 9.3 Essential Collector Extensions

10 Chapter 10: Collector Components: Receivers, Processors & Exporters

10.1 Standard Receivers & Ingestion Protocols 10.2 Core Processors & Execution Order 10.3 Standard Exporters & Delivery Guarantees

11 Chapter 11: Data Transformation & The OpenTelemetry Transformation Language (OTTL)

11.1 OTTL Concepts & Syntax Grammar 11.2 Transform Processor In Action 11.3 Advanced Connectors

12 Chapter 12: Collector Deployment, Scaling & Resilience

12.1 Agent vs Gateway Deployment Topologies 12.2 Scaling, Load Balancing & High Availability 12.3 Resilience: Backpressure, Buffering & Persistent Queues

13 Chapter 13: Maintaining, Debugging & Schema Evolution

13.1 Debugging Collector Pipelines & Telemetry Health 13.2 Troubleshooting Broken Traces & Context Loss 13.3 Schema Management & Telemetry Evolution

12.3 Resilience: Backpressure, Buffering & Persistent Queues

Key Takeaways Explain This Topic AI

Telemetry backpressure propagates upstream: when backends throttle or fail, exporter queues fill, memory rises, and receivers refuse data with retryable errors (gRPC UNAVAILABLE or HTTP 503 when the memory limiter refuses it).

Default in-memory sending queues (sending_queue) provide zero-I/O buffering for short transient spikes, but introduce catastrophic data loss if the Collector process crashes or restarts during an extended backend outage.

The file_storage extension keeps a persistent sending queue in an on-disk key-value database (bbolt), so queued batches survive a Collector restart and drain when the backend returns.

Out-of-memory (OOM) fatal crashes are prevented by properly sizing the memory_limiter processor—setting limit_mib to ~80% and spike_limit_mib to ~20% of container memory limits—and configuring the Go runtime GOMEMLIMIT environment variable.

Combining durable file storage, conservative memory limits, and upstream client retry policies establishes an end-to-end resilient telemetry architecture capable of surviving network partitions and backend brownouts.

Last updated: September 2026

12.3 Resilience: Backpressure, Buffering & Persistent Queues

Quick Answer: When downstream observability backends experience outages or rate limits, the OpenTelemetry Collector protects itself and upstream systems through backpressure propagation : exporter queues fill, memory rises, and receivers return retryable errors (gRPC UNAVAILABLE or HTTP 503 ), prompting application SDKs to retry and, if their own queues fill, shed telemetry locally. While default in-memory sending queues offer zero I/O overhead, they lose their contents if the pod crashes or restarts; the file_storage extension backs a persistent queue on disk that survives container restarts. To prevent fatal out-of-memory (OOM) crashes, the memory_limiter processor must be placed first in every pipeline with limit_mib sized to 80% and spike_limit_mib to 20% of container limits, complemented by the Go runtime GOMEMLIMIT setting.

Observability pipelines operate under asynchronous, unpredictable conditions. Backend storage clusters (such as Prometheus TSDBs, OpenSearch clusters, or cloud APM platforms) undergo scheduled maintenance, hit API rate limits, or suffer transient network partitions. During downstream downtime, the OpenTelemetry Collector must act as a reliable shock absorber. If the Collector cannot manage backpressure, buffer incoming surges, and protect its memory boundaries, it risks crashing with Linux Out-Of-Memory (OOM) errors, corrupting data buffers, or cascading failures back into customer-facing applications.

Backpressure Propagation in Telemetry Pipelines

In an OpenTelemetry pipeline, telemetry data flows downstream from applications to storage backends. Conversely, backpressure travels upstream , flowing in reverse from the destination storage system back to the telemetry source: =============================== TELEMETRY DATA FLOW (DOWNSTREAM) ===============================>

[App Microservice SDK] ──> [Receiver] ──> [Processor] ──> [Exporter Queue] ──> [Remote Backend]

  • <============================= BACKPRESSURE PROPAGATION (UPSTREAM) =============================
  • [SDK drops locally] <── [HTTP 429] <── [Halts Work] <── [Queue Full] <── [Backend Outage]
  • [Protects App RAM] [gRPC 14/8] [Limiter Drops] [Rejects Batches] [HTTP 429 / 503]
  • The Step-by-Step Anatomy of a Downstream Failure

When a downstream observability backend slows down (e.g., returning HTTP 429 Too Many Requests due to rate limits) or suffers a hard outage (network timeout, HTTP 503 Service Unavailable):

Exporter Sending Queue Fills : The exporter's retry mechanism kicks in with exponential backoff. In-flight batches cannot be drained, and new batches arriving from upstream processors accumulate in the exporter's sending_queue up to its configured capacity ( queue_size ).

Exporter Rejects New Batches : Once the sending_queue is full, new data cannot be enqueued; it is counted in otelcol_exporter_enqueue_failed_* (unless block_on_overflow: true makes the caller wait for space).

The Error Travels Back, Unless Something Absorbs It : Without an asynchronous stage in between, the enqueue error returns through the processors to the receiver. The batch processor, however, has already accepted the data and answered its caller, so a queue-full error after it becomes a drop rather than backpressure. This is one reason newer Collector releases can batch inside the exporter ( sending_queue::batch ) instead.

Memory Pressure Triggers memory_limiter : As queued and buffered data pushes memory past the soft limit, the memory_limiter processor refuses new data with a retryable error (and above the hard limit it also forces garbage collection).

Receivers Reject Ingress Requests : The OTLP receiver turns a refused request into a retryable status for the client:

OTLP/gRPC Receiver : a plain retryable error becomes gRPC 14 (UNAVAILABLE) ; an error that already carries 8 (RESOURCE_EXHAUSTED) keeps it.

OTLP/HTTP Receiver : UNAVAILABLE maps to HTTP 503 (Service Unavailable) and RESOURCE_EXHAUSTED to HTTP 429 (Too Many Requests) .

Application SDKs React : Upstream OpenTelemetry SDKs receive the rejection status codes:

The SDK's OTLP exporter retries retryable responses with exponential backoff while the BatchSpanProcessor keeps queuing new spans.

If the outage persists and the SDK's local in-memory buffer ( max_queue_size , default 2048 spans) fills to capacity, the SDK drops telemetry locally at the application (SDKs that implement the self-observability conventions record this in otel.sdk.processor.span.processed with error.type=queue_full ).

The Golden Rule of Observability Backpressure

Telemetry is an operational monitoring tool; it must never compromise the primary business function of the application. Upstream backpressure propagation guarantees that when catastrophic storage failures occur, telemetry is shed gracefully at the edges rather than allowing unchecked buffer growth to consume memory, trigger application container crashes, or lock up user request threads.

In-Memory Buffering vs. File-Backed Persistent Storage

The OpenTelemetry Collector provides two primary strategies for managing queued telemetry inside exporters: in-memory sending queues and file-backed persistent queues . +-------------------------------------------------------------------------+

  • | In-Memory Queue vs. File-Backed Persistent Queue |
  • +-------------------------------------------------------------------------+
  • IN-MEMORY SENDING QUEUE FILE-BACKED PERSISTENT QUEUE
  • (sending_queue: storage: none) (sending_queue: storage: file_storage)
  • [RAM: Go Process Heap] [RAM Buffer] ──> [On-Disk Queue DB]
  • Blazing fast (zero disk I/O) - Survives container crashes & OOMs
  • Microsecond latency - Survives Kubernetes node evictions
  • Zero storage volume config - Spools gigabytes during outages
  • ---------------------------------- -----------------------------------
  • FATAL FLAW: Process crash or RECOVERY: On boot, Collector reads
  • pod restart permanently wipes the queue from disk and drains all

all buffered telemetry in RAM! queued batches to restored backend!

1. In-Memory Sending Queue ( sending_queue with default memory storage)

How It Operates : The exporter keeps a bounded queue of batches in the Go heap, sized by queue_size (default 1,000 requests).

Strengths : Ultra-high throughput, zero disk I/O latency, no requirement for Kubernetes PersistentVolumeClaims (PVCs) or stateful storage drivers.

The Critical Vulnerability : RAM is volatile. If the Collector container runs out of memory, is terminated by a Kubernetes rolling update, or experiences a worker node hardware reboot during an outage, every single telemetry batch buffered in memory is lost forever .

2. File-Backed Persistent Storage Queue ( file_storage extension)

How It Operates : The file_storage extension (Contrib) stores data in an on-disk key-value database (bbolt) in its directory (default /var/lib/otelcol/file_storage ), which can live on local SSD, a cloud block volume, or a Kubernetes PVC. With sending_queue::storage set, the queue lives only on disk; there is no in-memory queue.

The Persistence Lifecycle :

When a batch enters the queue, it is serialized and written to the storage database before the enqueue call returns.

Consumers read batches from storage and export them; a batch is deleted from storage after success or a permanent error.

During an outage, batches accumulate on disk until queue_size batches are stored or the disk (or the extension's optional max_size ) is full; after that, new data is rejected.

Crash Recovery : If the Collector is killed or restarted, the new process opens the same directory and resumes exporting the stored batches. In Kubernetes this works only if the same volume is attached to the replacement pod.

Memory Safety & Out-Of-Memory (OOM) Prevention

Because the OpenTelemetry Collector is implemented in Go, managing memory boundaries requires understanding both the Go runtime Garbage Collector (GC) and the Collector's internal pipeline memory safeguards. +-------------------------------------------------------------------------+

  • | Collector Container Memory Layout (Example: 2 GiB) |
  • +-------------------------------------------------------------------------+
  • [ 0 MiB ] [ 2048 MiB ]
  • ├────────────────────────────┼──────────────┼─────────────────────┤
  • │ Normal Pipeline Operation │ Soft Limit │ Hard Container Limit│
  • │ Heap Allocations & Batches │ Limiter Sheds│ (Kernel OOM Killer) │
  • │ │ (Spike Zone) │ │
  • ├────────────────────────────┼──────────────┼─────────────────────┤
  • 0 MiB 1228 MiB 1638 MiB 2048 MiB
  • (Limit - Spike) (limit_mib) (Container Max)

1. Normal Ingestion: Heap stays well below 1228 MiB.

2. Sudden Traffic Burst: Memory crosses 1228 MiB -> memory_limiter drops batches.

3. Go GC Intervention: GOMEMLIMIT (about 1638 MiB) makes the garbage collector work harder near the limit.

4. OOM Avoidance: Process stays below 2048 MiB container limit. SIGKILL avoided!

The Go Runtime Memory Dilemma

By default, Go's garbage collector uses a target percentage ( GOGC=100 ), meaning GC triggers only when the heap doubles relative to reachable memory. During sudden bursts of telemetry (such as a cascading microservice outage generating massive trace spikes), memory allocations occur faster than the Go runtime schedules GC sweeps. The container's Resident Set Size (RSS) exceeds the Kubernetes memory limit, and the Linux kernel OOM Killer immediately terminates the container with SIGKILL (Exit Code 137) .

The memory_limiter Processor

The memory_limiter processor is the primary line of defense against OOM terminations.

The Sizing Formula :

Let M M M be the container's hard memory limit (e.g., 2048 MiB 2048\text{ MiB} 2048 MiB in Kubernetes resources.limits.memory ).

limit_mib : The hard target limit for the Collector process. Must be set to approximately 80% of M M M : limit_mib = 2048 MiB × 0.80 ≈ 1638 MiB \text{limit\_mib} = 2048\text{ MiB} \times 0.80 \approx 1638\text{ MiB} limit_mib = 2048 MiB × 0.80 ≈ 1638 MiB

spike_limit_mib : The safety buffer to absorb surges between measurement cycles. Must be set to approximately 20% of M M M : spike_limit_mib = 2048 MiB × 0.20 ≈ 410 MiB \text{spike\_limit\_mib} = 2048\text{ MiB} \times 0.20 \approx 410\text{ MiB} spike_limit_mib = 2048 MiB × 0.20 ≈ 410 MiB

Soft Threshold Activation : When memory usage exceeds limit_mib - spike_limit_mib ( 1638 − 410 = 1228 MiB 1638 - 410 = 1228\text{ MiB} 1638 − 410 = 1228 MiB ), the processor refuses incoming data with retryable errors; if usage goes above limit_mib it also forces garbage collection.

check_interval : How often memory usage is checked (recommended: 1s for general workloads, 500ms for bursty workloads).

CRITICAL INVARIANT : The memory_limiter processor must always be placed as the FIRST processor in every pipeline sequence ( processors: [memory_limiter, batch, ...] ). If placed after the batch or transform processors, memory is allocated and consumed before the limiter can evaluate whether sufficient headroom exists to process the payload!

Tuning Go Runtime Environment Variables ( GOMEMLIMIT & GOMAXPROCS )

In modern containerized deployments (Go 1.19+), platform engineers must configure two critical runtime environment variables:

GOMEMLIMIT : A soft memory limit for the Go runtime. The memory_limiter README recommends setting it to about 80% of the Collector's hard memory limit (e.g., GOMEMLIMIT=1638MiB for a 2 GiB container); the garbage collector then runs more often as memory approaches the limit, reclaiming heap before the kernel's OOM killer acts.

GOMAXPROCS : Since Go 1.25, the runtime derives the default GOMAXPROCS from the container's CPU limit on Linux. Binaries built with older Go versions used the host's core count, which could cause CPU-quota throttling in Kubernetes; they needed go.uber.org/automaxprocs or an explicit GOMAXPROCS matching the CPU limit.

Production Configuration: Resilient Collector with file_storage

The following complete configuration demonstrates production-grade resilience, combining the file_storage extension, properly sized memory_limiter , and a persistent OTLP exporter queue: extensions:

# Health probe for Kubernetes liveness/readiness

health_check:

endpoint: 0.0.0.0:13133

# Durable on-disk storage (bbolt) for the persistent queue

file_storage:

  • directory: /var/lib/otelcol/file_storage
  • create_directory: true
  • timeout: 10s

compaction:

  • on_rebound: true
  • directory: /var/lib/otelcol/file_storage/compaction

receivers:

otlp:

protocols:

grpc:

endpoint: 0.0.0.0:4317

http:

endpoint: 0.0.0.0:4318

processors:

# CRITICAL: memory_limiter MUST be the first processor in every pipeline

memory_limiter:

  • check_interval: 1s
  • limit_mib: 1638 # 80% of 2048 MiB container limit
  • spike_limit_mib: 410 # 20% of 2048 MiB container limit

batch:

  • send_batch_size: 8192
  • timeout: 1s

exporters:

otlp:

endpoint: remote-backend.observability.net:4317

tls:

  • insecure: false
  • ca_file: /etc/ssl/certs/backend-ca.crt
  • # Persistent sending queue stored through file_storage

sending_queue:

  • enabled: true
  • storage: file_storage
  • num_consumers: 16
  • queue_size: 10000

retry_on_failure:

  • enabled: true
  • initial_interval: 5s
  • max_interval: 60s
  • max_elapsed_time: 15m

service:

extensions: [health_check, file_storage]

pipelines:

traces:

  • receivers: [otlp]
  • processors: [memory_limiter, batch]
  • exporters: [otlp]

metrics:

  • receivers: [otlp]
  • processors: [memory_limiter, batch]
  • exporters: [otlp]

logs:

  • receivers: [otlp]
  • processors: [memory_limiter, batch]
  • exporters: [otlp]

telemetry:

logs:

level: info

metrics:

readers:

- pull:

exporter:

prometheus:

  • host: 0.0.0.0
  • port: 8888
  • Resilience Comparison Reference
  • Operational Dimension Default In-Memory Buffer File-Backed Storage Buffer ( file_storage )
  • Underlying Medium Volatile Go Process RAM Durable disk (bbolt database on a local disk or Persistent Volume)
  • I/O Latency Sub-microsecond (RAM speed) Low millisecond (buffered disk writes)
  • Storage Capacity Constrained by container RAM (MiBs) Constrained only by attached disk volume (GiBs)

Behavior on Pod Crash Total data loss for all queued items Queued batches kept on disk and resumed after restart

Behavior on Node Eviction Total data loss when node terminates Preserved only if the same volume is reattached to the new pod

  • Backend Outage Tolerance Minutes (until memory limit is hit) Hours to days (until disk volume fills)
  • Production Recommendation Ephemeral dev/staging environments Mission-critical enterprise production tiers

Loading diagram... Backpressure Propagation and File Storage WAL Pipeline