Study guide

9.2 Collector Configuration Structure & Pipelines

OpenTelemetry Certified Associate OTCA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & Observability Foundations

1.1 OTCA Exam Structure & Observability Paradigms 1.2 Core Telemetry Signals: Traces, Metrics, Logs & Baggage 1.3 Telemetry Analysis, SLOs & High Cardinality

2 Chapter 2: Semantic Conventions & Instrumentation Strategy

2.1 OpenTelemetry Semantic Conventions & Namespaces 2.2 Resource Attributes & Entity Modeling 2.3 Zero-Code vs Code-Based Instrumentation

3 Chapter 3: OpenTelemetry Architecture & The Data Model

3.1 Architectural Separation of API and SDK 3.2 The Core Trace Model: Spans, SpanContext & Links 3.3 Span Events, Attributes & Status Codes 3.4 Composability & Extension: Plugin Interfaces, Distributions & Contrib

4 Chapter 4: Distributed Tracing & Context Propagation

4.1 In-Process & Cross-Process Context Propagation 4.2 W3C Trace Context Specification 4.3 W3C Baggage & Alternative Propagators

5 Chapter 5: OpenTelemetry Metrics API & Instruments

5.1 Synchronous Metric Instruments 5.2 Asynchronous (Observable) Metric Instruments 5.3 Metric Aggregations & Temporality

6 Chapter 6: Logs Data Model & Logging Bridge API

6.1 OpenTelemetry Logs Data Model & Architecture 6.2 The Logging Bridge API & Log Appenders

7 Chapter 7: SDK Pipelines, Processors & Exporters

7.1 Span Processors & Exporters 7.2 Metric Readers & Log Processors 7.3 Standard Exporters & OTLP Wire Protocols

8 Chapter 8: Sampling Strategies, SDK Configuration & Agents

8.1 Head-Based Sampling & Built-in Samplers 8.2 Tail-Based Sampling vs Head-Based Sampling 8.3 OpenTelemetry SDK Configuration & Environment Variables 8.4 Zero-Code Agents: Configuration, Kubernetes Injection & Troubleshooting

9 Chapter 9: OpenTelemetry Collector Architecture & Pipelines

9.1 Collector Core Architecture & Component Lifecycle 9.2 Collector Configuration Structure & Pipelines 9.3 Essential Collector Extensions

10 Chapter 10: Collector Components: Receivers, Processors & Exporters

10.1 Standard Receivers & Ingestion Protocols 10.2 Core Processors & Execution Order 10.3 Standard Exporters & Delivery Guarantees

11 Chapter 11: Data Transformation & The OpenTelemetry Transformation Language (OTTL)

11.1 OTTL Concepts & Syntax Grammar 11.2 Transform Processor In Action 11.3 Advanced Connectors

12 Chapter 12: Collector Deployment, Scaling & Resilience

12.1 Agent vs Gateway Deployment Topologies 12.2 Scaling, Load Balancing & High Availability 12.3 Resilience: Backpressure, Buffering & Persistent Queues

13 Chapter 13: Maintaining, Debugging & Schema Evolution

13.1 Debugging Collector Pipelines & Telemetry Health 13.2 Troubleshooting Broken Traces & Context Loss 13.3 Schema Management & Telemetry Evolution

9.2 Collector Configuration Structure & Pipelines

Key Takeaways Explain This Topic AI

Collector YAML declares components under receivers, processors, exporters, connectors, and extensions, and activates them in the service block; each pipeline needs at least one receiver and one exporter.

Declaring a component under receivers, processors, or exporters merely configures an instance; it remains completely inactive until explicitly referenced in a pipeline under service.pipelines.

Pipelines are signal-specific (traces, metrics, logs) and support multiple named variants using forward-slash syntax (such as traces/sampling and traces/compliance).

Processor execution order within a pipeline is strictly linear and sequential: memory_limiter must be placed first to prevent OOM kills, followed by filters and transforms, with batch placed immediately before exporters.

Multiple pipelines operate independently and concurrently, allowing fan-in from multiple receivers and fan-out to multiple backend exporters without cross-signal data interference.

Last updated: September 2026

9.2 Collector Configuration Structure & Pipelines

Quick Answer: OpenTelemetry Collector configurations are defined in YAML using the top-level keys receivers , processors , exporters , connectors (when used), extensions , and service . The service block glues the configuration together by defining active extensions and signal-specific pipelines ( traces , metrics , logs ). Processors in a pipeline execute in the exact linear order listed. Critical rule: memory_limiter should execute first to guard against out-of-memory crashes, while batch should execute after any sampling or filtering, right before the exporters.

The OpenTelemetry Collector relies on a purely declarative YAML configuration file. Rather than writing custom integration code or managing disparate agents, operators declare components, tune their parameters, and wire them together into independent data-processing pipelines. Understanding the hierarchical structure of this YAML document and mastering pipeline mechanics is central to operating a Collector.

The Top-Level YAML Configuration Blocks

A Collector configuration file is organized around these root-level keys (a sixth, connectors , appears when pipelines are joined by a connector; see Section 11.3): receivers: # 1. Declares input protocols and network bindings

...

processors: # 2. Declares in-memory transformation, filtering, and batching

...

exporters: # 3. Declares destination endpoints, formats, and egress queues

...

extensions: # 4. Declares auxiliary runtime capabilities (health, profiling)

...

service: # 5. Activates extensions, telemetry, and wires components into pipelines

...

1. receivers

The receivers section configures how telemetry enters the Collector. Each receiver instance is defined using the component name or a named instance format: component_type[/custom_name] . receivers:

otlp: # Default instance of the otlpreceiver

protocols:

grpc:

endpoint: 0.0.0.0:4317

http:

  • endpoint: 0.0.0.0:4318
  • otlp/internal: # Named instance of the otlpreceiver for internal traffic

protocols:

grpc:

  • endpoint: 127.0.0.1:5555
  • prometheus: # Scrape receiver for Prometheus metrics

config:

scrape_configs:

  • job_name: 'collector-self'
  • scrape_interval: 15s

static_configs:

  • targets: ['localhost:8888']
  • 2. processors

The processors section defines in-memory operations applied to telemetry as it moves through the pipeline. Like receivers, multiple instances of the same processor type can be declared using named suffixes: processors:

  • memory_limiter: # Must be configured to prevent OOM
  • check_interval: 1s
  • limit_percentage: 75
  • spike_limit_percentage: 20
  • batch: # Buffers data before sending to exporters
  • send_batch_size: 8192
  • timeout: 1s
  • send_batch_max_size: 10240
  • filter/drop_health: # Named filter processor to drop health probes
  • error_mode: ignore

traces:

span:

  • 'attributes["url.path"] == "/healthz"'
  • 3. exporters

The exporters section defines where processed telemetry is delivered, along with connection timeouts, TLS certificates, compression algorithms, and queuing mechanisms: exporters:

  • otlp/primary: # Primary OTLP gRPC backend
  • endpoint: tempo.monitoring.svc.cluster.local:4317

tls:

insecure: true

sending_queue:

  • enabled: true
  • queue_size: 5000

retry_on_failure:

  • enabled: true
  • initial_interval: 5s
  • max_interval: 30s
  • debug: # Replaces legacy 'logging' exporter for troubleshooting
  • verbosity: detailed
  • 4. extensions

The extensions section configures auxiliary components that provide server management, security, and diagnostics without processing telemetry batches: extensions:

health_check:

endpoint: 0.0.0.0:13133

pprof:

endpoint: 0.0.0.0:1777

zpages:

  • endpoint: 0.0.0.0:55679
  • 5. service (The Master Orchestrator)

The service section binds all declared components together into a working Collector. If a component is declared under receivers , processors , exporters , or extensions but is never referenced in the service block, it is completely ignored by the Collector runtime : service:

extensions: [health_check, pprof, zpages] # Activates declared extensions

pipelines:

traces:

  • receivers: [otlp, otlp/internal]
  • processors: [memory_limiter, filter/drop_health, batch]
  • exporters: [otlp/primary, debug]

metrics:

  • receivers: [otlp, prometheus]
  • processors: [memory_limiter, batch]
  • exporters: [otlp/primary]
  • telemetry: # Configures internal monitoring of the Collector itself

logs:

level: info

metrics:

readers:

- pull:

exporter:

prometheus:

  • host: 0.0.0.0
  • port: 8888
  • Multi-Signal Pipelines

A pipeline defines a unidirectional data processing pathway for a specific telemetry signal. Each pipeline connects one or more receivers, through a sequence of zero or more processors, to one or more exporters. +-------------------------------------------------------------+

  • | PIPELINE |
  • | |
  • | [Receivers] ===> [Processor 1] ===> [Processor 2] ===> [Exporters]
  • | (Fan-In) (Sequential execution order) (Fan-Out)
  • +-------------------------------------------------------------+
  • Pipeline Signal Types and Naming Syntax

Pipelines are strictly typed by signal:

traces : Ingests, processes, and exports distributed trace spans.

metrics : Ingests, processes, and exports gauge, counter, and histogram data points.

logs : Ingests, processes, and exports structured log records.

When multiple pipelines are needed for the same signal (e.g., routing production traces differently from development traces), operators declare named pipelines using a forward slash:

  • traces/production
  • traces/compliance_audit
  • metrics/internal
  • Fan-In and Fan-Out Mechanics

The Collector architecture natively supports complex routing topologies through fan-in and fan-out:

Receiver Fan-In : A single pipeline can reference multiple receivers ( receivers: [otlp, zipkin, jaeger] ). Telemetry arriving on any of these receivers is converted into internal pdata and fed into the shared processor chain.

Receiver Fan-Out : A single receiver can be referenced across multiple pipelines ( traces , metrics , logs ). Each OTLP export request carries one signal, so the otlp receiver hands trace requests to every traces pipeline that lists it and metric requests to the metrics pipelines.

Exporter Fan-Out : A single pipeline can reference multiple exporters ( exporters: [otlp/primary, otlp/backup, debug] ). The pipeline hands every processed batch to each listed exporter, cloning the data when a consumer might modify it.

Pipeline Independence and Isolation

Pipelines do not share data: a transform in the traces pipeline never touches metrics . They are not perfectly isolated, though. A receiver shared by several pipelines waits for its downstream consumers, so backpressure in one pipeline can slow the shared receiver; separate receivers keep failure domains apart.

Processor Execution Order

Processors in a pipeline execute in the EXACT sequential order in which they are listed under service.pipelines.<signal>.processors . They do not execute concurrently, and they do not execute based on internal priority heuristics. CORRECT PRODUCTION ORDER:

  • [Receivers]
  • |
  • v
  • 1. memory_limiter <-- Drops/throttles before memory allocation; prevents OOM
  • |
  • v
  • 2. filter <-- Discards unwanted data immediately (saves CPU/memory)
  • |
  • v
  • 3. transform <-- Scrubs PII, normalizes attributes on retained data
  • |
  • v
  • 4. k8sattributes <-- Enriches retained spans with pod/namespace metadata
  • |
  • v
  • 5. batch <-- Assembles optimal network payloads right before sending
  • |
  • v
  • [Exporters]
  • The Operational Rules of Processor Sequencing

memory_limiter MUST ALWAYS BE FIRST : The memory limiter processor monitors the Go runtime heap. If available memory drops below configured safety limits, it drops data or signals backpressure to the receiver. If memory_limiter is placed after batch or transform , the Collector will allocate large amounts of heap space parsing, mutating, and buffering payloads, triggering an operating system Out-Of-Memory (OOM) kill before the limiter ever evaluates the payload!

filter and transform BEFORE batch : Filtering unwanted spans (such as high-frequency /healthz checks) must happen before batching. If batch is placed before filter , the Collector wastes CPU cycles and memory assembling large batches of data, only for the subsequent filter processor to tear those batches apart and discard half the items. This leads to inefficient, half-empty network payloads.

batch Near the End (After Sampling and Filtering) : The batch processor's README says to place it after memory_limiter and after any sampling processors, because batching should happen after data is dropped. Placing it just before the exporters lets outgoing requests carry full batches. (Newer Collector releases can also batch inside the exporter through sending_queue::batch , see Section 10.3.)

Annotated Complete Production YAML Configuration

The following complete, production-ready configuration demonstrates the integration of all five configuration blocks across traces, metrics, and logs: # ===================================================================

  • # 1. RECEIVERS: Ingest telemetry across multiple protocols
  • # ===================================================================

receivers:

otlp:

protocols:

grpc:

endpoint: 0.0.0.0:4317

http:

endpoint: 0.0.0.0:4318

prometheus:

config:

scrape_configs:

  • job_name: 'otel-collector'
  • scrape_interval: 15s

static_configs:

- targets: ['localhost:8888']

  • # ===================================================================
  • # 2. PROCESSORS: In-memory processing, ordering, and batching
  • # ===================================================================

processors:

# Rule: memory_limiter must always be the first processor!

memory_limiter:

  • check_interval: 500ms
  • limit_percentage: 80
  • spike_limit_percentage: 20

# Rule: filter early to reduce resource consumption

filter/drop_health:

error_mode: ignore

traces:

span:

  • 'attributes["url.path"] == "/healthz"'
  • 'attributes["url.path"] == "/ready"'

# Rule: enrich retained telemetry

resourcedetection:

  • detectors: [env, system]
  • timeout: 2s

# Rule: batch immediately before exporters

batch:

  • send_batch_size: 4096
  • timeout: 500ms
  • send_batch_max_size: 8192
  • # ===================================================================
  • # 3. EXPORTERS: Deliver processed pdata to destinations
  • # ===================================================================

exporters:

otlp/tempo:

endpoint: tempo.monitoring.svc.cluster.local:4317

tls:

insecure: true

sending_queue:

  • enabled: true
  • queue_size: 5000

retry_on_failure:

enabled: true

otlp/mimir:

endpoint: mimir.monitoring.svc.cluster.local:4317

tls:

insecure: true

debug:

verbosity: basic

  • # ===================================================================
  • # 4. EXTENSIONS: Auxiliary management and diagnostics
  • # ===================================================================

extensions:

health_check:

endpoint: 0.0.0.0:13133

pprof:

endpoint: 0.0.0.0:1777

zpages:

endpoint: 0.0.0.0:55679

  • # ===================================================================
  • # 5. SERVICE: Wires components into pipelines and activates runtime
  • # ===================================================================

service:

extensions: [health_check, pprof, zpages]

pipelines:

traces:

  • receivers: [otlp]
  • processors: [memory_limiter, filter/drop_health, resourcedetection, batch]
  • exporters: [otlp/tempo, debug]

metrics:

  • receivers: [otlp, prometheus]
  • processors: [memory_limiter, resourcedetection, batch]
  • exporters: [otlp/mimir]

telemetry:

logs:

level: info

metrics:

readers:

- pull:

exporter:

prometheus:

  • host: 0.0.0.0
  • port: 8888

Loading diagram... Collector Multi-Signal Pipelines and Sequential Processor Ordering