Study guide
8.3 Configuring Search Peers & Distributed Search Groups
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
8.3 Configuring Search Peers & Distributed Search Groups
Key Takeaways Explain This Topic AI
Add search peers on the search head in Settings > Distributed search > Search peers, or with splunk add search-server https://<peer>:8089 -auth <sh_user>:<pw> -remoteUsername <peer_admin> -remotePassword <pw>.
Adding a peer through Splunk Web or the CLI copies the search head's trusted.pem to the peer's $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_serverName>/; peers added by editing distsearch.conf need that copy done manually.
The [distributedSearch] stanza of distsearch.conf holds servers, disabled_servers, statusTimeout (default 10), connectionTimeout (10), sendTimeout (30), and receiveTimeout (600).
Distributed search groups are [distributedSearch:<group>] stanzas whose servers must be a subset of the main list; searches target them with splunk_server_group=<group>.
Only one group can be default = true; if none is, searches without splunk_server_group run against the full peer list.
Last updated: September 2026
Configuring Search Peers & Distributed Search Groups
In a distributed Splunk deployment, establishing secure, resilient connections between Search Heads and Indexers is essential for search execution. Administrators must actively register search peers, configure communication timeouts, monitor peer availability, and logically partition indexer fleets using Distributed Search Groups.
All distributed search connections are governed by configuration stanzas within distsearch.conf . Understanding how to manipulate these settings through Splunk Web, the Command Line Interface (CLI), and direct configuration edits ensures operational stability across complex network topologies.
Establishing Search Peer Connections
Before a Search Head can dispatch queries to an indexer, an administrator must register the indexer as a Search Peer . When the relationship is established, the search head's public key is placed on the peer over the management port (TCP 8089), and the peer uses it to authenticate the search head's later requests. +-----------------------+ +-----------------------+
- | SEARCH HEAD | | SEARCH PEER |
- | | | (INDEXER) |
- | 1. splunk add | HTTPS REST (TCP 8089) | |
- | search-server with | ----------------------> | 2. Authenticates the |
- | -remoteUsername / | | remote admin creds |
- | -remotePassword | | |
- | | SH public key (trusted.pem) |
- | 3. Sends its public | ----------------------> | 4. Stores it under |
- | key | | etc/auth/distServerKeys/<SH serverName>/ |
- | 5. Adds peer to | | |
- | distsearch.conf | | |
- +-----------------------+ +-----------------------+
- Method 1: Configuring Search Peers via Splunk Web
Log in to the Search Head's Splunk Web interface with administrative privileges.
Navigate to Settings > Distributed Search > Search Peers .
Click New Search Peer .
Enter the complete URI of the target indexer's management interface, including protocol and port (e.g., https://idx01.corp.internal:8089 ).
Provide credentials for an admin user on the remote indexer ( Remote username and Remote password ). The search head uses them once to set up trust by installing its public key on the peer.
Click Save , and repeat for each peer. The peer appears in the Search peers table.
- Method 2: Configuring Search Peers via the CLI
- Run this on the search head, once per peer: splunk add search-server https://idx01.corp.internal:8089 \
- auth admin:SearchHeadPassword \
- remoteUsername admin -remotePassword IndexerPassword
- splunk list search-server -auth admin:SearchHeadPassword
- splunk remove search-server https://idx01.corp.internal:8089 -auth admin:SearchHeadPassword
-auth : credentials for the search head (where the command runs).
-remoteUsername / -remotePassword : credentials for an admin-level user on the search peer . The peer's admin password must have been changed from the default, or the search head cannot authenticate to it.
The URI must include the scheme ( https:// or http:// ) and the peer's management port .
How the Trust Is Established (Public Key Authentication)
Access to peers is controlled through public key authentication , not stored passwords:
The search head has a key pair under $SPLUNK_HOME/etc/auth/distServerKeys/ , and its public key is trusted.pem .
When you add a peer with Splunk Web or the CLI, Splunk automatically copies the search head's trusted.pem to the peer at $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_serverName>/trusted.pem .
If you add peers by editing distsearch.conf instead, you must copy that key file to each peer yourself and restart the peers.
A peer that serves several search heads keeps one directory per search head under distServerKeys/ .
Important: A search head cannot also act as a search peer. The only exception is the Monitoring Console, which works as a "search head of search heads". Indexer clusters connect their search heads to the peers automatically, and search head clusters have their own rules for connecting to peers.
Core distsearch.conf Settings
$SPLUNK_HOME/etc/system/local/distsearch.conf on the search head holds the peer list and the distributed search options: [distributedSearch]
- servers = https://idx01.corp.internal:8089,https://idx02.corp.internal:8089,https://idx03.corp.internal:8089
- statusTimeout = 10
- connectionTimeout = 10
- sendTimeout = 30
- receiveTimeout = 600
- disabled_servers = https://idx04.corp.internal:8089
- Setting Default What it does
- disabled false Turns distributed search on ( false ) or off ( true )
- servers none Comma-separated peer URIs in the form scheme://host:port
- disabled_servers none Peers that stay configured but are not monitored or searched
statusTimeout 10 Seconds to wait when gathering a peer's basic info from /services/server/info ; read/write timeouts are set to twice this value
- connectionTimeout 10 Seconds allowed to connect to a peer
- sendTimeout / receiveTimeout 30 / 600 Seconds allowed for sending to and receiving from a peer
- bestEffortSearch false If true , a peer that lacks the knowledge bundle still takes part in searches
Exam trap: Older material lists checkTimedOutServersFrequency , removedTimedOutServers , and autoAddServers . The current spec says the first two are no longer supported and are ignored , and autoAddServers is deprecated . Heartbeat multicast settings are deprecated as well.
Monitoring Search Peer Status
Splunk Web shows each peer's state under Settings > Distributed search > Search peers . You can also query it from the search head: | rest splunk_server=local /services/search/distributed/peers
- | table title status replicationStatus version
- What Happens When a Peer Is Unreachable
The search head marks the peer as Down in the peer list and keeps searching the peers that are Up .
The search still completes, but with incomplete results , and Splunk Web warns the user that some peers could not be searched.
Peers listed in disabled_servers , or disabled in Splunk Web, are neither monitored nor searched.
If a peer lacks the current knowledge bundle, it is skipped by default. bestEffortSearch = true changes that behavior.
Administrative Monitoring Tools
Monitoring Console: its distributed search dashboards show peer health and knowledge bundle replication.
_internal : search splunkd.log on the search head for distributed search and bundle replication warnings.
Distributed Search Groups: Logical Fleet Partitioning
In large enterprise deployments with tens or hundreds of indexers, treating all search peers as a single homogenous pool is often inefficient or non-compliant with data governance requirements. Splunk allows administrators to organize search peers into Distributed Search Groups .
Architectural Rationale for Search Groups
Data Sovereignty & Geographic Boundaries: Isolating indexers by region (e.g., EU_Indexers vs. US_Indexers ) to ensure European queries do not dispatch across transatlantic links.
Regulatory Compliance: Restricting access to high-security environments (such as PCI-DSS cardholder environments or HIPAA repositories) so that only compliance auditors query those dedicated indexers.
Workload Tiering: Separating high-performance indexers hosting real-time security data from dense storage indexers hosting archived operational metrics.
Defining Search Groups in distsearch.conf
Groups are defined in distsearch.conf on the search head with [distributedSearch:<group_name>] stanzas. Each group's servers list must be a subset of the peers in the main [distributedSearch] stanza, and it uses host:port peer identifiers: [distributedSearch]
servers = https://10.1.1.11:8089,https://10.1.1.12:8089,https://10.2.1.11:8089,https://10.2.1.12:8089
- [distributedSearch:PCI_Indexers]
- default = false
- servers = 10.1.1.11:8089,10.1.1.12:8089
- [distributedSearch:EU_Region]
- default = false
- servers = 10.2.1.11:8089,10.2.1.12:8089
- The default Attribute
Only one group may have default = true . A search that names no group then runs against that group's peers.
If no group is the default (or the default group has no servers list), a search that names no group runs against the full [distributedSearch] peer list.
Groups can overlap. A peer can belong to several groups.
Indexer clusters: Splunk says search groups are generally not valid for indexer clustering . The cluster places primary bucket copies arbitrarily, so a group-limited search can return incomplete results. Groups are useful with clusters only in special cases, such as a search head that searches several clusters, or a cluster plus standalone indexers.
Restricting Search Execution with splunk_server_group and splunk_server
Users and scheduled searches target specific search groups or individual indexers using internal search fields:
- 1. Filtering by Search Group ( splunk_server_group ) index=card_payments splunk_server_group=PCI_Indexers
- | stats count by payment_status
The search head sends this search only to the peers in [distributedSearch:PCI_Indexers] . The other peers receive nothing, which saves their CPU, disk reads, and network bandwidth.
Multiple groups can be combined using boolean logic: index=network (splunk_server_group=EU_Region OR splunk_server_group=PCI_Indexers)
- | stats sum(bytes) by src_ip
- 2. Filtering by Specific Indexer ( splunk_server )
If an administrator needs to investigate or troubleshoot a specific indexer's local storage: index=_internal splunk_server=idx-us01.corp.internal
| stats count by sourcetype
This directs the query exclusively to the named indexer host.