Study guide
5.2 Asynchronous (Observable) Metric Instruments
OpenTelemetry Certified Associate OTCA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & Observability Foundations
1.1 OTCA Exam Structure & Observability Paradigms 1.2 Core Telemetry Signals: Traces, Metrics, Logs & Baggage 1.3 Telemetry Analysis, SLOs & High Cardinality
2 Chapter 2: Semantic Conventions & Instrumentation Strategy
2.1 OpenTelemetry Semantic Conventions & Namespaces 2.2 Resource Attributes & Entity Modeling 2.3 Zero-Code vs Code-Based Instrumentation
3 Chapter 3: OpenTelemetry Architecture & The Data Model
3.1 Architectural Separation of API and SDK 3.2 The Core Trace Model: Spans, SpanContext & Links 3.3 Span Events, Attributes & Status Codes 3.4 Composability & Extension: Plugin Interfaces, Distributions & Contrib
4 Chapter 4: Distributed Tracing & Context Propagation
4.1 In-Process & Cross-Process Context Propagation 4.2 W3C Trace Context Specification 4.3 W3C Baggage & Alternative Propagators
5 Chapter 5: OpenTelemetry Metrics API & Instruments
5.1 Synchronous Metric Instruments 5.2 Asynchronous (Observable) Metric Instruments 5.3 Metric Aggregations & Temporality
6 Chapter 6: Logs Data Model & Logging Bridge API
6.1 OpenTelemetry Logs Data Model & Architecture 6.2 The Logging Bridge API & Log Appenders
7 Chapter 7: SDK Pipelines, Processors & Exporters
7.1 Span Processors & Exporters 7.2 Metric Readers & Log Processors 7.3 Standard Exporters & OTLP Wire Protocols
8 Chapter 8: Sampling Strategies, SDK Configuration & Agents
8.1 Head-Based Sampling & Built-in Samplers 8.2 Tail-Based Sampling vs Head-Based Sampling 8.3 OpenTelemetry SDK Configuration & Environment Variables 8.4 Zero-Code Agents: Configuration, Kubernetes Injection & Troubleshooting
9 Chapter 9: OpenTelemetry Collector Architecture & Pipelines
9.1 Collector Core Architecture & Component Lifecycle 9.2 Collector Configuration Structure & Pipelines 9.3 Essential Collector Extensions
10 Chapter 10: Collector Components: Receivers, Processors & Exporters
10.1 Standard Receivers & Ingestion Protocols 10.2 Core Processors & Execution Order 10.3 Standard Exporters & Delivery Guarantees
11 Chapter 11: Data Transformation & The OpenTelemetry Transformation Language (OTTL)
11.1 OTTL Concepts & Syntax Grammar 11.2 Transform Processor In Action 11.3 Advanced Connectors
12 Chapter 12: Collector Deployment, Scaling & Resilience
12.1 Agent vs Gateway Deployment Topologies 12.2 Scaling, Load Balancing & High Availability 12.3 Resilience: Backpressure, Buffering & Persistent Queues
13 Chapter 13: Maintaining, Debugging & Schema Evolution
13.1 Debugging Collector Pipelines & Telemetry Health 13.2 Troubleshooting Broken Traces & Context Loss 13.3 Schema Management & Telemetry Evolution
5.2 Asynchronous (Observable) Metric Instruments
Key Takeaways Explain This Topic AI
Asynchronous (Observable) instruments operate on a pull/callback model, where registered callbacks are executed periodically by the SDK during metric collection cycles rather than inline per request.
Because asynchronous callbacks execute on an SDK background collection thread, they are NOT associated with an active distributed execution Context and cannot capture inline TraceId/SpanId exemplars.
The three asynchronous instruments are ObservableCounter (monotonic cumulative totals like CPU execution time), ObservableUpDownCounter (non-monotonic counts like active threads or open file descriptors), and ObservableGauge (instantaneous non-additive states like temperature or memory usage percentage).
Batch Callbacks enable registering a single callback function across multiple observable instruments, allowing expensive underlying subsystem statistics (such as /proc/diskstats) to be scraped once and recorded atomically.
Choosing between synchronous and asynchronous instruments depends primarily on the event origin: synchronous for high-frequency internal application events, and asynchronous for periodic polling of external subsystem states.
Last updated: September 2026
5.2 Asynchronous (Observable) Metric Instruments
Quick Answer: Asynchronous (Observable) metric instruments—also known as callback or gauge-like instruments—are not called inline during application transactions. Instead, the OpenTelemetry SDK invokes user-registered callback functions periodically at every collection/scrape interval to report the current state of a system. Asynchronous instruments execute outside the request lifecycle, meaning they are NOT associated with an active distributed trace Context and cannot automatically record trace exemplars. The three asynchronous types are ObservableCounter , ObservableUpDownCounter , and ObservableGauge .
While synchronous instruments excel at capturing measurements triggered directly by application business logic (such as an incoming HTTP call or a completed database transaction), many critical operational metrics originate outside the immediate request flow. Operating system metrics, hardware telemetry, kernel counters, thread pool internals, and external cache statistics exist as standing states rather than discrete request-level events.
Attempting to measure these states synchronously—such as querying the Linux kernel's /proc/stat on every incoming HTTP request—would introduce unacceptable CPU and I/O overhead to production services. To solve this problem, OpenTelemetry provides Asynchronous (Observable) Instruments .
The Asynchronous Invocation Model
Asynchronous instruments operate on an inverted control flow often described as a pull model or callback model : Application Startup:
└── meter.create_observable_gauge("system.memory.usage", callbacks=[observe_memory])
Runtime (No Overhead on Request Threads):
- ├── Request 1 Processed ──> (No metric code executed)
- ├── Request 2 Processed ──> (No metric code executed)
- └── Request N Processed ──> (No metric code executed)
Periodic Collection Cycle (default every 60 s):
- └── OpenTelemetry SDK MetricReader Scraper
- ├── Invokes observe_memory(observer)
- │ ├── Reads OS memory statistics
- │ └── observer.observe(used_bytes, attributes)
- └── Exports gathered measurements via OTLP
- The Role of Registered Callbacks
When defining an observable instrument, the developer does not receive an object with an add() or record() method. Instead, the developer supplies one or more callback functions during instrument creation (or registers callbacks after instantiation).
These callbacks remain dormant while the application executes requests. When the configured OpenTelemetry SDK MetricReader (such as a periodic exporting metric reader or a Prometheus scrape endpoint) initiates a collection cycle, the SDK invokes the registered callbacks. The callback inspects the target resource, reads its current state, and submits observations to an Observer object.
Context Isolation: No Ambient TraceContext
A foundational concept is the execution context of asynchronous callbacks:
Asynchronous instrument callbacks are NEVER associated with an active distributed execution Context!
When a synchronous instrument executes, it runs on the application thread handling a specific request, allowing the SDK to access Context.current() and capture the active TraceId and SpanId as an exemplar. In contrast, asynchronous callbacks execute on an independent SDK timer or exporter thread. At the instant the callback runs, there is no active HTTP request, no active RPC span, and no distributed trace context in scope.
Consequently, asynchronous instruments cannot record exemplars or link measurements directly to individual distributed traces . They provide macro-level system state observations, not request-correlated traces.
- The Three Asynchronous Instrument Types
- The OpenTelemetry Metrics API provides three asynchronous instrument primitives: Asynchronous Instruments
- │
- ┌──────────────────────────┼──────────────────────────┐
- ▼ ▼ ▼
- ObservableCounter ObservableUpDownCounter ObservableGauge
- (Monotonic) (Non-Monotonic) (Instantaneous)
- Cumulative Total Fluctuating State Non-Additive
- 1. ObservableCounter (Asynchronous Counter)
An ObservableCounter is an asynchronous instrument that reports a monotonically increasing cumulative value via a callback. Like its synchronous counterpart, it represents a sum that only ever increases (or resets to zero upon system reboot).
Operational Characteristics
Callback Reporting: The callback observes the cumulative total reached at the instant of the collection scrape.
Monotonicity: Each observed value must be greater than or equal to the previously observed value under the same attribute set (unless a system reboot or counter reset occurred).
Standard Use Cases:
- Operating system CPU execution time in seconds obtained from /proc/stat ( process.cpu.time )
- Cumulative network octets or packets received by a network interface since OS boot
- Cumulative page faults handled by the virtual memory subsystem
- Cumulative garbage collection collection time reported by a runtime JMX/V8 engine
- 2. ObservableUpDownCounter (Asynchronous UpDownCounter)
An ObservableUpDownCounter is an asynchronous instrument that reports non-monotonic values via a callback, where the reported value can rise and fall over time, and where values are additive across entities .
Operational Characteristics
Additive Property: The reported values can be meaningfully summed across multiple instances, nodes, or processes. For example, summing the active thread counts of 10 microservice pods yields the total active threads across the service fleet.
Standard Use Cases:
- Current number of JVM threads ( jvm.thread.count )
- Process open file descriptor count ( process.unix.file_descriptor.count )
- Total allocated heap memory bytes currently retained by an application
- Total number of worker jobs currently queued in an external Celery or Redis queue
- 3. ObservableGauge (Asynchronous Gauge)
An ObservableGauge is an asynchronous instrument that records non-additive instantaneous values via a callback.
The Critical Concept: Non-Additive Values
The fundamental distinction between an ObservableUpDownCounter and an ObservableGauge lies in additivity :
If you have two application servers where Server A has 50 active threads and Server B has 70 active threads, summing them equals 120 total active threads. Thread count is additive, making it an ObservableUpDownCounter .
If Server A has a CPU temperature of 75°C and Server B has a CPU temperature of 65°C, adding them together ( 75 + 65 = 140 ∘ C 75 + 65 = 140^\circ\text{C} 75 + 65 = 14 0 ∘ C ) is mathematically meaningless. You can calculate the average (70°C), minimum, or maximum, but you cannot sum them. Therefore, temperature is non-additive, making it an ObservableGauge .
- Standard Use Cases
- Current CPU core temperature in Celsius
- Memory utilization percentage (0.0% to 100.0%)
- Instantaneous filesystem usage fraction ( system.filesystem.utilization )
- Current room humidity or atmospheric pressure in an edge IoT facility
- Remaining battery percentage
- Batch Callbacks (Multi-Instrument Callbacks)
In real-world infrastructure monitoring, querying system status often produces multiple related metrics from a single operational source. For example, reading Linux /proc/diskstats provides read operations, write operations, read bytes, write bytes, and I/O time simultaneously.
If an engineer registers five separate Observable instruments with five independent callbacks, the SDK will invoke five separate file system operations on /proc/diskstats . This creates two major problems:
Redundant CPU and I/O Overhead: Parsing the same file five times per scrape cycle.
Temporal State Skew: Measurements are captured at slightly different millisecond offsets, resulting in inconsistencies between related metrics.
The Batch Callback Solution
OpenTelemetry solves this via Batch Callbacks . A batch callback allows an application to register a single callback associated with a list of multiple observable instruments. When the collection cycle executes, the callback runs exactly once, retrieves all metrics from the underlying subsystem, and observes values across all target instruments simultaneously: // Go exposes multi-instrument callbacks through Meter.RegisterCallback.
- // (Java offers meter.batchCallback; Python's API has only per-instrument callbacks.)
- diskIO, _ := meter.Int64ObservableCounter("system.disk.io",
- metric.WithUnit("By"), metric.WithDescription("Disk bytes transferred"))
- diskOps, _ := meter.Int64ObservableCounter("system.disk.operations",
- metric.WithUnit("{operation}"), metric.WithDescription("Disk operations"))
// One callback reads /proc/diskstats once and observes both instruments.
- _, err := meter.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
- for _, dev := range readDiskStats() { // parses /proc/diskstats once per collection
- read := metric.WithAttributes(
- attribute.String("system.device", dev.Name),
- attribute.String("disk.io.direction", "read"))
- write := metric.WithAttributes(
- attribute.String("system.device", dev.Name),
- attribute.String("disk.io.direction", "write"))
- o.ObserveInt64(diskIO, dev.ReadBytes, read)
- o.ObserveInt64(diskIO, dev.WriteBytes, write)
- o.ObserveInt64(diskOps, dev.Reads, read)
- o.ObserveInt64(diskOps, dev.Writes, write)
- }
- return nil
- }, diskIO, diskOps)
- if err != nil {
- log.Fatal(err)
- }
- Comparison: Synchronous vs. Asynchronous Instruments
Architectural Dimension Synchronous Instruments Asynchronous (Observable) Instruments
Invocation Method Inline push ( add() , record() ) on calling thread Periodic pull callback ( observe() ) on SDK thread
Trigger Point Exact moment an application event occurs Each collection: the reader's interval (60 s by default for the periodic reader) or a scrape
Trace Context Binding YES: Automatically captures active TraceId and SpanId NO: Executes outside request context; no ambient span
Exemplar Support Fully supported (links metric outliers to trace spans) Not supported (no trace context available)
Primary Overhead Location Execution path of live user requests (must be non-blocking) Scrape interval execution (background timer thread)
Primary Target Domain Request counts, latency distributions, inline queue deltas OS stats, CPU times, thread pools, memory levels, temperatures
Complete Master Table: All Seven OpenTelemetry Instruments
The Metrics API defines seven instruments:
- Instrument Name Synchronicity Monotonic? Additive? Core API Method Trace Context? Canonical Production Example
- Counter Synchronous YES ( ≥ 0 \ge 0 ≥ 0 ) YES add(value, attrs) YES shop.orders.completed (custom)
- UpDownCounter Synchronous NO ( + / − +/- + / − ) YES add(value, attrs) YES http.server.active_requests
- Histogram Synchronous N/A (Distribution) YES record(value, attrs) YES http.server.request.duration
- Gauge Synchronous NO NO record(value, attrs) YES a fan-speed change listener
- ObservableCounter Asynchronous YES ( ≥ 0 \ge 0 ≥ 0 ) YES Callback observe() NO process.cpu.time
ObservableUpDownCounter Asynchronous NO ( + / − +/- + / − ) YES Callback observe() NO process.unix.file_descriptor.count