Study guide
10.1 Forwarder Setup & outputs.conf Load Balancing
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
10.1 Forwarder Setup & outputs.conf Load Balancing
Key Takeaways Explain This Topic AI
outputs.conf on a forwarder sets global defaults in [tcpout] (defaultGroup, indexAndForward) and indexer pools in [tcpout:<group>] with server = host:port lists.
Load balancing is automatic: the forwarder switches to a randomly chosen indexer every autoLBFrequency (default 30 seconds) or after autoLBVolume bytes (default 0, off).
A universal forwarder switches only at safe points unless EVENT_BREAKER_ENABLE/EVENT_BREAKER are set for the source type, so large files can pin a UF to one indexer.
Indexer discovery replaces static server lists: forwarders set indexerDiscovery and an [indexer_discovery:<name>] stanza with manager_uri and pass4SymmKey; the cluster manager's polling_rate (1-10) paces polling.
Forwarder TLS uses clientCert, sslPassword, and sslVerifyServerCert in outputs.conf, with the CA bundle in server.conf [sslConfig] sslRootCAPath; indexers receive on [splunktcp-ssl:<port>].
Last updated: September 2026
Forwarder Setup & outputs.conf Load Balancing
Quick Summary: In distributed Splunk Enterprise deployments, forwarders collect data at the edge and stream it to the indexing tier. The configuration file outputs.conf dictates how forwarders establish network sessions, distribute data volumes across peer indexers, adapt dynamically to auto-scaling clusters via Indexer Discovery, and encrypt in-flight wire streams using TLS/SSL.
Architecture of Forwarder Data Egress
Forwarders—both Universal Forwarders (UFs) and Heavy Forwarders (HFs)—act as the primary data collection agents in an enterprise Splunk deployment. While inputs are defined in inputs.conf , the egress transmission of captured data is governed exclusively by outputs.conf .
The forwarding tier uses the proprietary Splunk-to-Splunk (s2s) protocol, which operates by default over TCP port 9997 . The primary objective of the forwarding architecture is to deliver balanced ingestion across the indexer tier without overwhelming any single node, causing pipeline starvation, or truncating individual events during network handoffs. +-----------------------------------------------------------------------------------+
- | FORWARDER (UF or HF) |
- | +--------------------+ +--------------------+ +-------------------------+ |
- | | inputs.conf | -> | Ingestion Pipeline | -> | outputs.conf | |
- | | (Files, TCP, Win) | | (Memory Queue) | | (Routing & Load Balance)| |
- | +--------------------+ +--------------------+ +-------------------------+ |
- +-----------------------------------------------------------------------------------+
- |
- Splunk-to-Splunk Protocol (s2s: TCP 9997)
- Automatic Load Balancing (30s / Volume Switches)
- |
- +--------------------------------+--------------------------------+
- | | |
- v v v
- +------------------+ +------------------+ +------------------+
- | INDEXER PEER 1 | | INDEXER PEER 2 | | INDEXER PEER 3 |
- | inputs.conf: | | inputs.conf: | | inputs.conf: |
- | [splunktcp:// | | [splunktcp:// | | [splunktcp:// |
- | 9997] | | 9997] | | 9997] |
- +------------------+ +------------------+ +------------------+
- Complete Anatomy of outputs.conf
The outputs.conf configuration file is structured hierarchically. Global default parameters are declared in the [tcpout] stanza, while specific destinations, load-balancing groups, and connection behaviors are organized into named target groups using [tcpout:<target_group>] stanzas.
1. The Global [tcpout] Stanza
The [tcpout] stanza specifies default forwarder behavior and defines which target group or groups receive incoming data streams when an input does not specify an explicit destination: [tcpout]
- defaultGroup = primary_indexers
- indexAndForward = false
- disabled = false
defaultGroup = <target_group1>, <target_group2>, ... :
Identifies one or more comma-separated target group stanzas that receive forwarded data by default. If multiple groups are declared here, Splunk clones the data, transmitting identical streams to each group in parallel.
indexAndForward = false|true :
Applies almost exclusively to Heavy Forwarders and full Splunk Enterprise instances. When set to false (the default), the instance passes data directly to its egress queues without writing events to local index storage buckets. When set to true , the instance indexes the data locally into its own hot buckets and forwards a copy to the specified output target group.
disabled = false|true :
When set to true , completely deactivates the TCP forwarding engine, preventing any egress connections on port 9997.
2. The Named Target Group Stanza [tcpout:<target_group>]
A target group represents a logical pool of indexers. The forwarder load-balances across all reachable servers listed in the group: [tcpout:primary_indexers]
- server = idx01.corp.internal:9997, idx02.corp.internal:9997, idx03.corp.internal:9997
- autoLBFrequency = 30
- autoLBVolume = 0
- compressed = false
- dnsResolutionInterval = 300
- maxQueueSize = 20MB
- useACK = true
server = <host1>:<port1>, <host2>:<port2>, ... :
A comma-separated list of receiving indexer endpoints. Each host entry must define the fully qualified domain name (FQDN) or IP address and the receiving s2s TCP port (typically 9997 ).
compressed = false|true :
Specifies whether data chunks are compressed (using zlib) on the forwarder prior to transmission. While setting this to true reduces WAN bandwidth consumption, it increases CPU overhead on both the forwarder and the indexer. Modern high-bandwidth enterprise LANs generally keep compression disabled.
dnsResolutionInterval = <integer> (default 300 ):
The base interval, in seconds, for re-resolving the host names in server . The runtime interval is extended by 30 seconds for each additional indexer in the list.
3. Receiving Configuration on the Indexing Tier ( inputs.conf )
For an indexer to accept s2s data streams sent from forwarders, the indexer must define a network receiver stanza in its local inputs.conf : [splunktcp://9997]
connection_host = dns
When [splunktcp://9997] is enabled, the indexer opens TCP port 9997 and binds it to its inputQueue , passing raw byte streams into the parsing and typing pipelines.
Core outputs.conf Attribute Reference
Attribute Name Stanza Scope Default Value Administrative Function
defaultGroup [tcpout] None Comma-separated list of target groups that receive data streams by default.
server [tcpout:<group>] None Comma-separated list of indexer endpoints ( host:port ) forming the load-balancing pool.
autoLBFrequency [tcpout] or [tcpout:<group>] 30 (seconds) Time interval before switching to another, randomly chosen indexer.
autoLBVolume [tcpout] or [tcpout:<group>] 0 (disabled) Byte volume threshold that forces a connection switch before the time interval expires.
dnsResolutionInterval [tcpout] or [tcpout:<group>] 300 (seconds) Base interval for re-resolving indexer host names.
indexerDiscovery [tcpout:<group>] None Binds the target group to an Indexer Discovery stanza pointing to the Cluster Manager.
useACK [tcpout] or [tcpout:<group>] false Enables indexer acknowledgment so unacknowledged data is resent.
maxQueueSize [tcpout] or [tcpout:<group>] auto (500 KB, or 7 MB with useACK ) Size of the output queue.
Automatic Load Balancing Mechanics
Unlike traditional network load balancers (F5, AWS ALB, NGINX) that terminate HTTP requests or distribute individual packets, Splunk utilizes client-side, connection-based load balancing built directly into the forwarder binary.
How Splunk Distributes Streams Across Indexers
The forwarder resolves the indexers listed in the target group's server setting, re-resolving DNS every dnsResolutionInterval (300 seconds by default, plus 30 seconds per additional indexer).
It sends data to one indexer at a time over a single connection.
When the time or volume threshold is reached, it switches to another indexer, chosen at random from the list, at a point where it is safe to switch.
Load balancing is always automatic. The current outputs.conf.spec has no separate autoLB switch to turn on. Forwarder egress timeline (autoLBFrequency = 30s):
- [00:00-00:30] ==> Indexer 2 (switch)
- [00:30-01:00] ==> Indexer 3 (switch: next indexer chosen randomly)
- [01:00-01:30] ==> Indexer 1 (switch)
- [01:30-02:00] ==> Indexer 3
- Time-Based Load Balancing ( autoLBFrequency )
By default, Splunk load balancing is controlled by time via autoLBFrequency :
Default value: 30 seconds ( autoLBFrequency = 30 ).
Every 30 seconds, the forwarder switches to another indexer in the group, chosen at random, once it reaches a safe switching point.
Tuning Consideration: Setting autoLBFrequency too low (e.g., 5 seconds) causes excessive TCP socket open/close overhead, TLS handshake churn, and pipeline serialization pauses. Setting it too high (e.g., 600 seconds) causes data to pool unevenly across indexers, creating disk utilization imbalances and hot bucket skew across the indexer cluster.
Volume-Based Load Balancing ( autoLBVolume )
In environments characterized by bursty, high-volume ingestion—such as database extraction dumps, batch application exports, or sudden security incident floods—a forwarder can transmit tens of gigabytes within a 30-second window. Under standard time-based load balancing, the entire multi-gigabyte burst would be sent to a single indexer, causing that indexer's inputQueue to fill while neighboring indexers sit idle.
To prevent burst-induced indexer starvation, administrators configure volume-based load balancing: [tcpout:primary_indexers]
- server = idx01:9997, idx02:9997, idx03:9997
- autoLBFrequency = 30
- autoLBVolume = 10485760
autoLBVolume = <bytes> : The volume of data to send to one indexer before a new indexer is randomly selected. 0 (the default) turns volume-based switching off.
Example: autoLBVolume = 10485760 (10 MB). The forwarder switches indexers as soon as it transmits 10 MB, even if only 2 seconds have elapsed since the connection opened.
Rule of Operation: The forwarder checks autoLBVolume first. If that volume has not been reached but autoLBFrequency has, it switches on time.
Event-Boundary Switching & Stream Integrity
A forwarder must not switch indexers in the middle of an event, or half of a multi-line event would land on each indexer.
Heavy Forwarders
A heavy forwarder has already parsed the data into events, so it can switch between events whenever the timer or volume threshold fires.
Universal Forwarders and EVENT_BREAKER
A universal forwarder sends unparsed data and does not know where events end. Without help, it waits for a point where switching is safe, such as the end of a file or a pause in the data. When it reads a large file continuously, it can therefore stay "stuck" on one indexer long after autoLBFrequency has passed, which leaves indexers unevenly loaded.
The fix is to tell the UF where events end for that source type, in props.conf on the universal forwarder : [my_app:log]
- EVENT_BREAKER_ENABLE = true
- EVENT_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}
With EVENT_BREAKER the UF's lightweight chunked line breaker splits the stream at event boundaries so it can switch indexers on schedule. forceTimebasedAutoLB = true in outputs.conf forces switches on the timer instead. Splunk recommends EVENT_BREAKER_ENABLE / EVENT_BREAKER for universal forwarders.
Dynamic Indexer Discovery
In modern cloud architectures, indexer clusters scale horizontally to match variable analytical workloads. Adding or decommissioning peer indexers requires updating the server list in outputs.conf across thousands of forwarders. Manually managing static IP addresses or deploying configuration updates via the Deployment Server introduces configuration latency, temporary load imbalances, and administrative overhead.
To solve this challenge, Splunk provides Dynamic Indexer Discovery . +----------------------------------------------------------------+
- | CLUSTER MANAGER (CM) |
- | - server.conf: [indexer_discovery] |
- | - Maintains real-time registry of all active peer indexers |
- +----------------------------------------------------------------+
- ^ (TCP 8089 HTTPS REST)
- | Forwarder polls CM (interval set by CM)
- | Returns JSON list of active peers
- v
- +----------------------------------------------------------------+
- | UNIVERSAL FORWARDER (UF) |
- | - outputs.conf: indexerDiscovery = cluster_manager_pool |
- | - Dynamically updates internal load-balancing pool |
- +----------------------------------------------------------------+
- | |
- v (Streams over TCP 9997) v (Streams over TCP 9997)
- +--------------------------------+ +--------------------------------+
- | ACTIVE INDEXER PEER 1 | | ACTIVE INDEXER PEER 2 |
- +--------------------------------+ +--------------------------------+
- Architectural Workflow of Indexer Discovery
Peer Registration: Each peer reports its receiving port and status to the cluster manager over the management port (TCP 8089).
Dynamic Polling: Forwarders are configured with the cluster manager's URI rather than a static list of indexers. They poll the manager over TCP 8089 at an interval the manager sets from its polling_rate and the number of connected forwarders.
Peer List Response: The Cluster Manager validates the forwarder's shared secret key ( pass4SymmKey ) and returns the current list of healthy, active indexer peers.
Local Pool Update: The forwarder dynamically updates its internal load-balancing table. If an indexer enters maintenance mode or goes offline, the Cluster Manager removes it from the list; the forwarder drops that indexer from its rotation during the next poll, routing traffic only to surviving peers.
- Configuration on the Cluster Manager ( server.conf )
- Indexer discovery is enabled on the cluster manager: [indexer_discovery]
- pass4SymmKey = EnterpriseDiscoverySecretKey2026
- polling_rate = 10
- indexerWeightByDiskCapacity = false
pass4SymmKey : Shared secret that forwarders must present. Use the same value in each forwarder's [indexer_discovery:<name>] stanza.
polling_rate = <1-10> (default 10 ): Not a number of seconds. It is a rate factor that, together with the number of connected forwarders, sets how often forwarders poll: (number_of_forwarders / polling_rate + 30 seconds) .
indexerWeightByDiskCapacity = true : Tells forwarders to use weighted load balancing, sending more data to peers with larger total disk capacity .
Configuration on the Forwarder ( outputs.conf )
On the forwarder, the target group points to an indexer discovery stanza instead of a static server list: [tcpout]
defaultGroup = discovered_cluster
- [tcpout:discovered_cluster]
- indexerDiscovery = cluster_manager_pool
- useACK = true
- [indexer_discovery:cluster_manager_pool]
- manager_uri = https://cm01.corp.internal:8089
- pass4SymmKey = EnterpriseDiscoverySecretKey2026
indexerDiscovery = <name> : Binds the [tcpout:<group>] stanza to the discovery definition.
manager_uri : The cluster manager's management URI. master_uri is the deprecated name for the same setting.
pass4SymmKey : Must match the value in the cluster manager's [indexer_discovery] stanza.
polling_interval (default 5 seconds) is only how long the forwarder waits after startup before its first fetch. After that, the cluster manager's response sets the polling interval.
Forwarder Transport Security: TLS/SSL Encryption
In enterprise environments governed by security standards such as PCI-DSS, HIPAA, or ISO 27001, transmitting log data in cleartext across the network is prohibited. Splunk enables administrators to secure the s2s protocol (TCP 9997) using Transport Layer Security (TLS/SSL). +-----------------------+ +-----------------------+
- | FORWARDER | | INDEXER |
- | - outputs.conf: | | - inputs.conf: |
- | clientCert | TLS Encrypted s2s | [splunktcp-ssl: |
- | sslPassword | ----------------------> | 9997] |
- | sslVerifyServerCert| (TCP 9997) | [SSL] serverCert |
- | server.conf CA bundle | | server.conf CA bundle |
- +-----------------------+ +-----------------------+
- Configuring TLS/SSL in outputs.conf
To enable TLS on the forwarder, configure certificate settings in the target group stanza of outputs.conf : [tcpout:secure_cluster]
- server = idx01.corp.internal:9997, idx02.corp.internal:9997
- useACK = true
- # TLS settings
- clientCert = $SPLUNK_HOME/etc/auth/mycerts/forwarder_combined.pem
- sslPassword = <private key password>
- sslVerifyServerCert = true
- sslCommonNameToCheck = idx01.corp.internal, idx02.corp.internal
- The trusted CA bundle is set once per instance in server.conf : [sslConfig]
- sslRootCAPath = $SPLUNK_HOME/etc/auth/mycerts/enterprise_root_ca.pem
Attribute Breakdown:
clientCert = <path> : The forwarder's certificate in PEM format, usually with its private key. The old name sslCertPath is deprecated. Setting clientCert alone turns on SSL for the connection.
sslPassword : The password for the private key.
sslVerifyServerCert = true|false (default false ): When true , the forwarder checks that the indexer's certificate is valid and signed by a trusted CA. Both the common name and the alternate name are then checked for a match.
sslCommonNameToCheck / sslAltNameToCheck : Optional lists of names the indexer's certificate must contain. They work only when sslVerifyServerCert = true .
sslRootCAPath in outputs.conf is deprecated . Use server.conf [sslConfig] sslRootCAPath for the CA bundle.
- Indexer Configuration for TLS Reception ( inputs.conf )
- On the receiving indexers: [splunktcp-ssl:9997]
- disabled = 0
- [SSL]
- serverCert = $SPLUNK_HOME/etc/auth/mycerts/indexer_combined.pem
- sslPassword = <private key password>
- requireClientCert = true
[splunktcp-ssl:<port>] opens a TLS-only Splunk-to-Splunk receiving port. Note the stanza form: there is no // .
[SSL] holds the server certificate and its password. requireClientCert = true enforces mutual TLS, so each forwarder must present a certificate signed by a CA the indexer trusts. ( rootCA here is deprecated in favor of server.conf sslRootCAPath .)
- Administrative Pitfalls & Troubleshooting
- Pitfall 1: Over-Aggressive Load Balancing Frequency
Symptom: High CPU utilization on forwarders and indexers, frequent TcpOutputFd - Connection reset messages in splunkd.log , and delayed data indexing.
Root Cause: Configuring autoLBFrequency to an excessively low value, such as 5 seconds. Because each connection switch requires socket teardown, TCP three-way handshake, and TLS negotiation, aggressive switching wastes significant network and CPU cycles on session management rather than data transmission.
Remedy: Keep the 30-second default for autoLBFrequency . If large files pin a universal forwarder to one indexer, set EVENT_BREAKER_ENABLE / EVENT_BREAKER for that source type. Consider autoLBVolume for bursty senders.
Pitfall 2: Static server Entries in Clustered Environments
Symptom: Data is directed to decommissioned indexers, generating connection errors; newly provisioned indexers receive zero data.
Root Cause: Relying on static server lists in outputs.conf when operating an elastic or clustered indexer tier.
Remedy: Implement dynamic Indexer Discovery. Forwarders retrieve the real-time cluster membership from the Cluster Manager, automatically adapting to node additions and maintenance states.
Pitfall 3: Certificate Verification Failures ( sslVerifyServerCert )
Symptom: Forwarders display SSL certificate validation failed: certificate has expired or common name does not match in $SPLUNK_HOME/var/log/splunk/splunkd.log , and forwarding stops completely.
Root Cause: Indexer certificates were renewed without updating the SAN extension, or local system clock skew caused the forwarder to evaluate the certificate as not yet valid or expired.
Remedy: Ensure Network Time Protocol (NTP) is synchronized across all forwarders and indexers. Validate that certificates contain wildcard SAN entries covering all indexer FQDNs (e.g., DNS:*.corp.internal ).