Study guide
10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
Key Takeaways Explain This Topic AI
With useACK = true, a forwarder keeps each sent block in a wait queue until the receiver acknowledges it, and resends unacknowledged data to another indexer.
Indexer acknowledgment gives at-least-once delivery, so a crash after writing but before the acknowledgment can create duplicate events.
With useACK = true, maxQueueSize = auto becomes 7 MB (500 KB without ACK), and the wait queue is three times maxQueueSize.
Listing several groups in defaultGroup (for example defaultGroup = prod, dr) clones all forwarded data to every group, and each destination meters license for what it indexes.
_TCP_ROUTING in inputs.conf routes a whole input to specific tcpout groups (works on universal forwarders); DEST_KEY = _TCP_ROUTING in transforms.conf routes individual events on a heavy forwarder.
Last updated: September 2026
Indexer Acknowledgment, Data Cloning & Forwarder Routing
Quick Summary: Enterprise forwarder topologies require more than simple stream distribution; they must protect data in flight, support multi-environment replication, and selectively direct high-value records. This section explores Indexer Acknowledgment ( useACK = true ) to protect against loss of in-flight data, parallel Data Cloning for synchronized environments, and the critical distinction between input-level and event-level selective data routing.
The Resilience Challenge in High-Volume Ingestion
When a forwarder transmits data over a standard TCP connection, it relies on the operating system's network stack for transport confirmation. A standard TCP ACK confirms only that the packets were received by the operating system kernel buffer on the destination host.
The Critical Failure Scenario: If the destination indexer experiences a kernel panic, sudden power outage, or splunkd crash after the OS receives the network packet but before the Splunk indexing pipeline writes the raw event to disk buckets, the forwarder believes the data was delivered successfully. The forwarder advances its internal read pointers (updating its fishbucket record) and drops the events from memory. When the indexer reboots, the data residing in volatile RAM is gone forever, resulting in silent data loss .
To solve this vulnerability, Splunk Enterprise provides Indexer Acknowledgment .
Indexer Acknowledgment ( useACK = true )
Indexer acknowledgment adds an application-level confirmation on top of TCP: the receiver confirms each block after it has handled it. +-----------------------+ +-----------------------+
- | FORWARDER | | INDEXER |
- | (Memory waitACK | | (Splunk Pipeline & |
- | Queue) | | Hot Buckets) |
- +-----------------------+ +-----------------------+
- | |
- | ---- 1. Data Block + ACK Token (ID: 5001) ------> |
- | | (Passes through input,
- | [Block retained in forwarder memory buffer] | parsing, & index queues)
- | |
- | | ---- 2. Writes rawdata
- | | to hot bucket
- | | journal on disk
- | v
- | <--- 3. Returns ACK Response (ID: 5001) ---------+ [Disk Commit Confirmed]
- |
- v
- 4. Forwarder releases block
- from memory buffer;
- advances fishbucket pointer
- How Indexer Acknowledgment Works
Forwarder keeps a copy: With useACK = true , the forwarder keeps a copy of each block it sends in a wait queue until the receiver acknowledges it.
Receiver acknowledges: The receiver sends an acknowledgment when it has fully handled the data, typically when it has written it to disk in indexing.
Forwarder releases the copy: On acknowledgment, the forwarder drops the copy from its wait queue.
No acknowledgment: If none arrives, for example because the indexer crashed or the connection timed out ( readTimeout , default 300 seconds, is used when reading acknowledgments), the forwarder resends the data to another indexer in the group.
useACK can be set in [tcpout] or in a [tcpout:<group>] stanza, but not for an individual [tcpout-server:...] .
Crash Recovery & At-Least-Once Delivery
If the indexer crashes, the connection breaks, or the acknowledgment does not arrive before the read times out ( readTimeout , default 300 seconds):
The forwarder closes the broken socket.
The forwarder takes the unacknowledged blocks from its wait queue.
The forwarder immediately initiates a connection to an alternate, healthy indexer in its load-balancing target group.
The unacknowledged blocks are retransmitted in order to the new indexer.
Important
Indexer Acknowledgment enforces an at-least-once delivery model . If an indexer successfully commits a block to disk but crashes a millisecond before the ACK packet reaches the network wire, the forwarder will retransmit the block to another indexer upon reconnection. This can introduce transient duplicate events in the index. However, in enterprise administration, search-time deduplication ( dedup ) or transactional grouping easily resolves duplicates, whereas silent data loss is irreversible.
Architectural Trade-Offs & Queue Tuning for useACK
Indexer Acknowledgment is recommended for data you cannot afford to lose, but plan for its memory and network overhead.
1. Forwarder Memory Footprint ( maxQueueSize )
Every block sent with acknowledgment stays in memory until it is acknowledged, so the forwarder needs more queue space:
With maxQueueSize = auto (the default), the output queue is 500 KB without useACK and 7 MB with it.
With useACK = true , the forwarder also creates a wait queue three times the size of maxQueueSize (for example, maxQueueSize = 1024KB gives a 3072 KB wait queue).
If the queues fill, for example on a slow or high-latency link, the forwarder applies backpressure to its inputs. File monitors simply stop reading until the queues drain, and the data stays in the files.
Increase maxQueueSize deliberately (for example maxQueueSize = 20MB ) on forwarders with high throughput or high-latency links.
2. Network Latency & the Bandwidth-Delay Product (BDP)
When forwarding data across Wide Area Networks (WANs) or inter-continental links with high Round-Trip Time (RTT)—such as 100 ms between London and New York—the forwarder must wait 100 ms for every ACK token.
According to the Bandwidth-Delay Product formula: Buffer Required = Throughput × Round-Trip Latency \text{Buffer Required} = \text{Throughput} \times \text{Round-Trip Latency} Buffer Required = Throughput × Round-Trip Latency
If the forwarder's queues are smaller than the data that can be in flight during one round trip, the forwarder pauses while waiting for acknowledgments, capping throughput below line speed. Size maxQueueSize for the link's throughput and latency.
3. No Setting Is Needed on the Indexer
Indexer acknowledgment is enabled on the forwarder with useACK in outputs.conf . The receiving [splunktcp://9997] input does not need a matching setting.
Data Cloning: Parallel Stream Distribution
Data Cloning is a forwarder capability that duplicates incoming raw data streams and transmits identical, full-fidelity copies to two or more independent indexer clusters or environments simultaneously. +-----------------------+
- | FORWARDER MEMORY |
- | (Raw Ingestion Block) |
- +-----------------------+
- |
- +------------------------------+------------------------------+
- | |
- v (Stream Copy A) v (Stream Copy B)
- +-------------------------------------------------+ +-------------------------------------------------+
- | TARGET GROUP: [tcpout:prod_cluster] | | TARGET GROUP: [tcpout:dev_cluster] |
- | - server = idx-prod01:9997, idx-prod02:9997 | | - server = idx-dev01:9997, idx-dev02:9997 |
- | - useACK = true | | - useACK = false |
- +-------------------------------------------------+ +-------------------------------------------------+
- Data Cloning vs. Indexer Cluster Replication
Administrators frequently confuse Data Cloning with Indexer Cluster Replication. They serve fundamentally different architectural objectives:
- Characteristic Data Cloning ( outputs.conf ) Indexer Cluster Replication ( server.conf )
- Initiating Component Forwarder (UF or HF) Indexer Peer Nodes
- Mechanism Duplicates raw streams across output groups Replicates storage buckets peer-to-peer over port 9887
- Network Target Separate, independent Splunk deployments Peer nodes within the same Indexer Cluster
- Primary Purpose Dual-environment sync (Dev/Prod, DR) High availability (RF/SF) within a single cluster
- License Consumption Ingestion metered on both destinations Ingestion metered once at primary ingestion
- Configuring Data Cloning in outputs.conf
Data cloning is configured by listing multiple target groups separated by commas in the defaultGroup attribute within the global [tcpout] stanza: [tcpout]
defaultGroup = prod_cluster, test_cluster
- # Production Cluster Target Group
- [tcpout:prod_cluster]
- server = idx-p01.corp.internal:9997, idx-p02.corp.internal:9997
- autoLBFrequency = 30
- useACK = true
- maxQueueSize = 20MB
- # Staging/Test Environment Target Group
- [tcpout:test_cluster]
- server = idx-t01.corp.internal:9997, idx-t02.corp.internal:9997
- autoLBFrequency = 30
- useACK = false
- Operational Considerations of Data Cloning
Network Egress Multiplication: Data cloning doubles (or triples, depending on group count) the outbound network bandwidth utilized by the forwarder. On saturated WAN links, cloning can cause network interface contention.
Independent Queue Execution: Each target group maintains its own independent egress queue and load-balancing engine. If test_cluster experiences an outage, its queue will block independently; however, if forwarder memory resources are constrained, backpressure from the failed group can eventually impact overall forwarder ingestion unless queues are properly isolated.
License Metering: Because each destination cluster operates an independent indexing tier, each cluster indexes the full volume of cloned data against its own local license pool.
Selective Data Routing: Input-Level vs. Event-Level
In complex enterprise architectures, forwarding all data to all indexers is neither efficient nor cost-effective. Organizations must selectively route specific data streams—such as sending payment logs to a high-security PCI-compliant cluster, directing firewall traffic to an operational cluster, or filtering debug logs to nullQueue .
Splunk provides two distinct mechanisms for selective routing:
Input-Level Routing (via _TCP_ROUTING in inputs.conf ).
Event-Level Routing (via props.conf and transforms.conf ).
Method 1: Input-Level Routing via inputs.conf
Input-level routing assigns an entire data input stanza (such as a directory monitor, script, or UDP listener) to one or more specific outputs.conf target groups using the _TCP_ROUTING attribute.
Supported On: Both Universal Forwarders and Heavy Forwarders . # inputs.conf on Universal Forwarder
- # Route Linux audit and authentication logs exclusively to security indexers
- [monitor:///var/log/audit/audit.log]
- sourcetype = linux:audit
- _TCP_ROUTING = security_indexers
- # Route standard web server logs exclusively to general indexers
- [monitor:///var/log/nginx/access.log]
- sourcetype = nginx:access
- _TCP_ROUTING = general_indexers
- # Selectively clone sensitive database logs to both security and audit clusters
- [monitor:///var/log/mysql/mysql-audit.log]
- sourcetype = mysql:audit
- _TCP_ROUTING = security_indexers, audit_archive
When _TCP_ROUTING is specified on an input, Splunk ignores the defaultGroup setting in outputs.conf for that specific input and directs the data stream solely to the declared groups.
Multiple target groups can be comma-separated to clone only that specific input without globally cloning all forwarder data.
Method 2: Event-Level Selective Routing via props.conf & transforms.conf
Event-level routing inspects the raw text of individual events using regular expressions, dynamically directing each record to a target group or discarding it entirely.
Caution
The Architectural Pipeline Boundary: Event-level routing requires the typing pipeline , where TRANSFORMS run. Universal forwarders do not run it for normal data; they stream unparsed chunks. Therefore, event-level selective routing via props.conf and transforms.conf cannot be executed on a Universal Forwarder . It must run on a Heavy Forwarder or an Indexer .
Configuration on a Heavy Forwarder:
- Step 1: Define the transform trigger in props.conf : # props.conf on Heavy Forwarder
- [cisco:asa]
- TRANSFORMS-routing = route_firewall_critical, route_firewall_info
- TRANSFORMS-filter = drop_debug_events
- Step 2: Configure regex rules in transforms.conf : # transforms.conf on Heavy Forwarder
- # Route critical severity events (%ASA-1 through %ASA-4) to PCI cluster
- [route_firewall_critical]
- REGEX = %ASA-[1-4]-
- DEST_KEY = _TCP_ROUTING
- FORMAT = pci_cluster
- # Route informational severity events (%ASA-5 through %ASA-6) to general cluster
- [route_firewall_info]
- REGEX = %ASA-[5-6]-
- DEST_KEY = _TCP_ROUTING
- FORMAT = general_cluster
- # Drop low-value debug noise (%ASA-7-) entirely to nullQueue before forwarding
- [drop_debug_events]
- REGEX = %ASA-7-
- DEST_KEY = queue
- FORMAT = nullQueue
DEST_KEY = _TCP_ROUTING : Reassigns the target group metadata for matching events to the group specified in FORMAT (which must match a [tcpout:<group>] stanza in outputs.conf ).
DEST_KEY = queue and FORMAT = nullQueue : Directs matching events to the bit-bucket ( nullQueue ), dropping them from memory immediately and preventing them from consuming network bandwidth or indexer storage.
Routing Methods Comparison & Decision Matrix
Technical Dimension Input-Level Routing ( inputs.conf ) Event-Level Routing ( transforms.conf ) Global Data Cloning ( outputs.conf )
Granularity Entire file, port, or scripted input Individual event payload via regex Entire forwarder egress stream
Supported Forwarders Universal Forwarders & Heavy Forwarders Heavy Forwarders & Indexers only Universal Forwarders & Heavy Forwarders
CPU & RAM Impact Virtually zero (metadata tag at input) Moderate to high (regex evaluation per event) Moderate (network egress doubled)
Primary Use Case Partitioning log types by file source Compliance filtering, severity segregation Multi-site sync, Staging/Prod parity
Configuration Files inputs.conf ( _TCP_ROUTING ) props.conf + transforms.conf outputs.conf ( defaultGroup = A, B )
- Administrative Pitfalls & Troubleshooting
- Pitfall 1: Attempting Regex Routing on Universal Forwarders
Symptom: An administrator deploys an app containing props.conf and transforms.conf with DEST_KEY = _TCP_ROUTING to a fleet of Universal Forwarders, but all events continue flowing to the defaultGroup .
Root Cause: Universal Forwarders lack the parsing pipeline. They ignore DEST_KEY transformations in transforms.conf completely.
Remedy: Either migrate the ingestion point to a Heavy Forwarder tier or partition data at the file/directory level using _TCP_ROUTING in inputs.conf .
Pitfall 2: High WAN Latency with Default Queue Limits
Symptom: Ingestion stalls across inter-site forwarders when useACK = true is enabled, even though bandwidth utilization is well below link capacity.
Root Cause: The maxQueueSize in outputs.conf is too small to handle the bandwidth-delay product of the network link, forcing the forwarder to wait for ACK tokens before enqueuing new data.
Remedy: Increase maxQueueSize in outputs.conf on the forwarder to cover the link's bandwidth-delay product (remember the wait queue is three times this value).
Pitfall 3: Indexer Bottlenecks Propagating Upstream
Symptom: Universal Forwarders across the enterprise stop reading logs; monitoring scripts report file tailing delays.
Root Cause: Downstream indexers are slow to write to disk. With useACK = true the forwarders wait for acknowledgments, so their queues fill and backpressure pauses their file inputs.
Remedy: Monitor indexer disk write latencies and pipeline queues using the Monitoring Console. Resolve indexer storage contention to restore forwarder throughput.