Study guide
8.2 Roles of the Search Head and Search Peers
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
8.2 Roles of the Search Head and Search Peers
Key Takeaways Explain This Topic AI
The search head hosts Splunk Web, authenticates users and enforces roles, owns the knowledge objects, dispatches searches, and merges results; it stores no indexed event data of its own.
Search peers are indexers that store data in buckets and run the remote part of each search using the search head's knowledge bundle, not their own local knowledge objects.
The knowledge bundle is a subset of files from $SPLUNK_HOME/etc/system, apps, and users; peers store it under $SPLUNK_HOME/var/run/searchpeers.
Bundles replicate in full the first time and as deltas afterward, using the classic, cascading, or mounted replicationPolicy.
If a bundle exceeds maxBundleSize (default 2048 MB) replication stops; exclude large files with [replicationDenylist] ([replicationBlacklist] is deprecated).
Last updated: September 2026
The Roles of the Search Head and Search Peers
Blueprint objective 8.2 asks you to explain what the search head does and what the search peers do. The two roles split the work, and the knowledge bundle is what lets peers search on the search head's behalf.
Architectural Separation: Search Heads vs. Search Peers
Distributed search divides computational workloads between two distinct server tiers: Search Heads and Search Peers . +-------------------------------------------------------------+
- | SEARCH HEAD |
- | - User Interface (TCP 8000) & REST API (TCP 8089) |
- | - SPL Parsing, AST Generation, & Execution Planning |
- | - Knowledge Object Management & Bundle Creation |
- | - Reduce Phase: Merging, Sorting, & Presentation |
- +-------------------------------------------------------------+
- |
- TCP 8089 (REST API / HTTPS Management)
- Knowledge Bundle Distribution (tar.gz)
- Search Dispatch & Peer Status Checks
- |
- +----------------------+----------------------+
- | |
- v v
- +-------------------------------+ +-------------------------------+
- | SEARCH PEER (INDEXER 1) | | SEARCH PEER (INDEXER 2) |
- | - Ingestion (TCP 9997 s2s) | | - Ingestion (TCP 9997 s2s) |
- | - Time-series Index (*.tsidx)| | - Time-series Index (*.tsidx)|
- | - Rawdata Storage & Journal | | - Rawdata Storage & Journal |
- | - Map Phase: Filter & Eval | | - Map Phase: Filter & Eval |
- +-------------------------------+ +-------------------------------+
- 1. The Search Head (Query Coordinator & Reducer)
A Search Head is a Splunk Enterprise instance dedicated to user interaction and analytical orchestration. It does not store user-indexed events locally. Its core administrative responsibilities include:
Client Servicing: Hosting the Splunk Web graphical interface (TCP 8000) and serving REST API client requests.
Authentication & Authorization: Authenticating users (via internal backends, LDAP, or SAML) and enforcing Role-Based Access Controls (RBAC) defined in authorize.conf .
Query Compilation: Parsing user SPL strings, building execution plans, and splitting the search into a remote part (run on peers) and a local part (run on the search head).
Knowledge Object Repository: Housing user-created and application-level knowledge objects—including saved searches, alerts, field extractions, lookups, macros, tags, and data models.
Knowledge Bundle Packaging: Compressing active search-time knowledge objects into tarball archives and distributing them to connected search peers.
The Reduce Phase: Aggregating, deduplicating, sorting, and calculating final statistical calculations on intermediate result streams received from search peers.
2. Search Peers (Data Indexers & Mappers)
A Search Peer is an Indexer that has established a trusted communication channel with one or more Search Heads. In distributed search, Search Peers act as workers executing localized data retrieval. Their responsibilities include:
Data Storage: Writing raw compressed journals and time-series inverted index files ( *.tsidx ) across bucket directories ( hot , warm , cold ).
Knowledge Bundle Execution: Storing the knowledge bundles received from search heads in $SPLUNK_HOME/var/run/searchpeers/ and using them (not the peer's own knowledge objects) for search-time field extractions and lookups.
The Map Phase: Scanning local index buckets within the query's designated time boundary ( earliest and latest ), retrieving candidate events matching index-time keys, evaluating distributable streaming commands (such as eval , where , and rex ), and computing partial summary aggregates.
Streaming Results: Returning compressed intermediate results back to the Search Head over the management REST channel.
Search Head vs. Search Peer Responsibilities
Operational Characteristic Search Head Search Peer (Indexer)
Primary Function User interface, query parsing, result reduction Data ingestion, index storage, running the remote part of each search
Cluster Membership Search Head Cluster (SHC) Indexer Cluster (Peer Nodes)
Storage Focus Knowledge objects, user profiles, search dispatch artifacts Inverted indexes ( .tsidx ), compressed rawdata journals
- Map-Reduce Role Reducer (finalizes results) Mapper (generates intermediate result sets)
- Inbound Ports TCP 8000 (Web), TCP 8089 (REST) TCP 9997 (s2s Forwarding), TCP 8089 (REST Management)
Hardware Scaling Target High CPU cores, high RAM for concurrent searches Fast I/O (high IOPS SSD/NVMe), storage capacity, high CPU
Knowledge Bundle Replication Mechanics
Search peers run the search on the search head's behalf, so they need the search head's search-time knowledge: field extractions, lookups, event types, tags, macros, and so on. When a peer runs a distributed search it ignores its own local knowledge objects and uses only what is in the search head's knowledge bundle . SEARCH HEAD SEARCH PEER (INDEXER)
- $SPLUNK_HOME/etc/system, apps, users $SPLUNK_HOME/var/run/searchpeers/
- │ (subset of files) ▲
- ▼ │
- $SPLUNK_HOME/var/run/<bundle>.bundle ── TCP 8089 (full, then deltas) ──┘
- 1. What the Bundle Contains
Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system , $SPLUNK_HOME/etc/apps , and $SPLUNK_HOME/etc/users on the search head.
By default, peers receive nearly the entire contents of the search head's apps , including lookup files. Large lookups are the usual cause of oversized bundles.
[replicationAllowlist] and [replicationDenylist] in distsearch.conf control which files go into the bundle. The denylist wins when a file matches both.
2. Replication Cycle
The search head builds bundles under $SPLUNK_HOME/var/run .
It replicates the bundle periodically in the background or when it starts a search .
The first replication to a new peer sends the full bundle. Later replications usually send only the delta since the last one.
Each peer stores bundles under $SPLUNK_HOME/var/run/searchpeers , one set per search head. Because the path differs from the search head's, search scripts must not hard-code paths to resources.
3. Replication Policies ( replicationPolicy in [replicationSettings] )
- Policy How it works
- classic The search head replicates the bundle directly to every peer
- cascading The search head sends the bundle to a few peers, which pass it on to the others
- mounted The bundle is placed on shared storage that peers read directly
- rfs Remote file system such as S3 (not supported for on-premises Splunk Enterprise)
- Keeping the Knowledge Bundle Small
If very large or frequently changing files are part of the bundle, replication slows, peers use more disk under searchpeers , and searches can run against stale knowledge. Once the bundle is larger than maxBundleSize (default 2048 MB ), replication stops and Splunk logs an error and shows an admin message.
Excluding Files with [replicationDenylist]
On the search head, add patterns to distsearch.conf . The older stanza name [replicationBlacklist] still works but is deprecated : [replicationSettings]
maxBundleSize = 2048
- [replicationDenylist]
- # Exclude one very large lookup that searches do not need on the peers
- big_threat_feed = apps/TA-threat-intel/lookups/massive_feed.csv
- # Exclude every CSV lookup in one app
- noisy_lookups = apps/legacy_app/lookups/*.csv
- Pattern Rules
Patterns are matched against file paths relative to $SPLUNK_HOME/etc , such as apps/<app>/lookups/<file> or users/<user>/<app>/... .
The syntax is Splunk's pattern style, which is mostly a regular expression: ... matches anything, * matches anything except a directory separator, and . matches a literal dot.
If you exclude a lookup that a search still needs, run that lookup on the search head instead, for example with lookup local=true , because the peers no longer have the file.
After changing the lists, watch the next replication in the Monitoring Console or splunkd.log to confirm the bundle shrank below maxBundleSize .