Study guide
3.2 Configuration Layering, Precedence & Resolution Rules
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
3.2 Configuration Layering, Precedence & Resolution Rules
Key Takeaways Explain This Topic AI
Global-context files (inputs, outputs, indexes, server, and similar) resolve as system local > app local > app default > system default.
In the global context, ties between apps at the same tier go to the app whose directory name sorts first lexicographically; uppercase sorts before lowercase and digits before letters.
App/user-context settings (searches, knowledge objects) resolve as current user > current app (local, then default) > other apps with exported settings (reverse-lexicographic) > system local > system default.
Splunk merges settings attribute by attribute: each key comes from the highest-priority file that sets it, and settings in a [default] stanza fill in anything left unset.
Indexer cluster peers add peer-apps directories: peer-apps local outranks even system local, and peer-apps default sits above app default.
Last updated: September 2026
Configuration Layering, Precedence & Resolution Rules
One of the most powerful architectural features of Splunk Enterprise is its layered configuration model. Multiple applications, add-ons, and system files can define settings for the exact same stanza simultaneously. When the splunkd daemon initializes or executes a task, it dynamically merges these disparate files into a single, cohesive, effective runtime configuration.
To manage this layering reliably, Splunk enforces deterministic precedence hierarchies. The exact sequence of precedence depends entirely on whether a configuration setting is evaluated within the Global context or the App and User context .
The Two Configuration Contexts
Splunk configuration files are partitioned into two distinct evaluation contexts based on the operational subsystem requiring the configuration: Configuration Context Boundary
├── Global Context (System, Indexing, Inputs, Forwarding)
│ └── Evaluated independent of UI sessions, user identities, or active apps.
│ Governs background daemons, network ports, index storage, and ingestion.
- │
- └── App and User Context (Search Time & Knowledge Objects)
└── Evaluated within a user session and active application framework.
Governs dashboards, saved searches, event types, field extractions, and tags.
1. Global Context
Global context applies to background daemon processes, data ingestion pipelines, indexing operations, forwarder egress, and system administration. These activities affect the Splunk instance as an entire operating unit, independent of who is logged into Splunk Web or which application view is active.
Key files operating in the Global context include:
- server.conf (clustering, licensing, daemon identity)
- inputs.conf (file monitors, network listening ports, modular inputs)
- outputs.conf (forwarder destination groups, load balancing)
- indexes.conf (index definitions, bucket paths, retention)
- authentication.conf (LDAP, SAML, user accounts)
- authorize.conf (role definitions and capabilities)
- distsearch.conf (search peer communication)
- web.conf (Splunk Web server configuration)
- 2. App and User Context
App and User context applies to search activities, user interfaces, and knowledge objects. When a user runs a search or loads a dashboard, Splunk evaluates configurations based on who is running the search and where (in which app) the search is executing.
Key files operating in the App/User context include:
- savedsearches.conf (reports, alerts, scheduled searches)
- eventtypes.conf (event classifications)
- tags.conf (field tags)
- macros.conf (search macros)
- collections.conf (KV Store collections)
Search-time aspects of props.conf and transforms.conf (search-time field extractions, lookups, calculated fields)
Global Context Precedence Hierarchy
When splunkd evaluates configurations in the Global context, it follows a strict four-tier precedence hierarchy. Higher-numbered tiers override lower-numbered tiers:
Precedence Rank Directory Tier Path on Disk Architectural Description
1 (Highest) System local $SPLUNK_HOME/etc/system/local/ Host-specific system administrative overrides.
2 App local $SPLUNK_HOME/etc/apps/<app_name>/local/ App-specific customizations and local administrator edits.
3 App default $SPLUNK_HOME/etc/apps/<app_name>/default/ Packaged app defaults and Technology Add-on baselines.
4 (Lowest) System default $SPLUNK_HOME/etc/system/default/ Factory baseline configuration shipped with Splunk. Global Precedence Order:
- System local --> App local --> App default --> System default
- [HIGHEST] [LOWEST]
- Conflict Resolution Between Competing Apps (Lexical Ordering)
In an enterprise environment, many apps and add-ons live in $SPLUNK_HOME/etc/apps/ . What happens when two apps set the same attribute in the same stanza of a global-context file such as inputs.conf or indexes.conf ?
First, all app local directories outrank all app default directories . Within the app local tier (or within the app default tier), Splunk breaks ties by the lexicographical (ASCII/UTF-8) order of the app directory names , and the name that sorts first wins :
Numbers sort before letters and are compared character by character: myapp1 > myapp10 > myapp2 > myapp20 .
Uppercase letters sort before lowercase letters. Z outranks a , so any app named with an uppercase first letter outranks apps named in lowercase.
Important
In the global context, when two apps set the same attribute at the same tier (for example, both in local/ ), the app whose directory name comes first in lexicographical order wins. In the app/user context the order is reversed. There, other apps are evaluated in reverse-lexicographical order, and only for exported settings.
For example, suppose both etc/apps/Splunk_TA_nix/local/inputs.conf and etc/apps/custom_inputs/local/inputs.conf set a different index for [monitor:///var/log/messages] . Uppercase S (0x53) sorts before lowercase c (0x63), so Splunk_TA_nix comes first and its value wins in the global context.
Administrators use this to force an outcome deliberately. A site-wide override app named with a leading digit, such as 000_corp_overrides , sorts ahead of alphabetic app names, so its global-context settings win ties against them. A name like zzz_... would lose.
Expanded Order on Indexer Cluster Peers
Indexer cluster peers add a directory type for configuration that the cluster manager distributes. On a peer, the global-context order is: peer-apps local > system local > app local > peer-apps default > app default > system default . The cluster manager pushes common peer configuration from $SPLUNK_HOME/etc/manager-apps/ (formerly master-apps/ ) into $SPLUNK_HOME/etc/peer-apps/ (formerly slave-apps/ ).
App and User Context Precedence Hierarchy
When a search executes or a knowledge object is referenced, Splunk evaluates configurations through the App and User context. In this context, user customization takes precedence over app defaults, and current app settings take precedence over external apps:
Precedence Rank Directory Tier Path on Disk Scope
1 (Highest) User local $SPLUNK_HOME/etc/users/<username>/<app>/local/ Private knowledge objects for the current user.
2 App local $SPLUNK_HOME/etc/apps/<current_app>/local/ Local customizations within the active app.
3 App default $SPLUNK_HOME/etc/apps/<current_app>/default/ Shipped defaults within the active app.
4 Other apps (local) $SPLUNK_HOME/etc/apps/<other_apps>/local/ Exported (globally shared) settings from other apps, in reverse-lexicographic order of app name.
5 Other apps (default) $SPLUNK_HOME/etc/apps/<other_apps>/default/ Exported shipped settings from other apps, in reverse-lexicographic order.
6 System local $SPLUNK_HOME/etc/system/local/ Global system overrides.
7 (Lowest) System default $SPLUNK_HOME/etc/system/default/ Factory baseline defaults. App/User Precedence Order:
- User local --> Current App local --> Current App default --> Other Apps --> System local --> System default
- [HIGHEST] [LOWEST]
- Knowledge Object Export Scoping ( metadata )
An object defined inside an app is only visible to "Other Apps" (tiers 4 and 5) if its sharing scope has been configured to global. Object sharing is governed by metadata/default.meta and metadata/local.meta :
export = none : Accessible only within the defining app.
export = system : Exported globally across all applications and search contexts.
Stanza-Level and Attribute-Level Merging Mechanics
A critical misconception among new administrators is the belief that higher-precedence files overwrite entire configuration files or entire stanzas. This is false.
Splunk uses a fine-grained, non-destructive merging model:
Files are not replaced: An attribute defined in system/local does not replace the file in apps/*/local .
Stanzas are not discarded: If a stanza exists in both a default file and a local file, Splunk merges the contents of both stanzas into a single composite stanza.
Attribute-level resolution: Splunk inspects each key-value pair independently:
If an attribute exists in a higher-precedence tier, that attribute's value wins.
If an attribute is absent from the higher-precedence tier, the value is inherited from the next highest tier where the key is defined.
Any non-conflicting keys across all tiers are preserved in the final merged stanza.
Concrete Merging Walkthrough: Data Ingestion Scenario
Consider an administrator monitoring /var/log/secure on a Linux host. Four configuration files contribute settings to the input [monitor:///var/log/secure] :
Tier 4: System Default ( $SPLUNK_HOME/etc/system/default/inputs.conf )
The shipped file does not contain this monitor stanza. It does contain a [default] stanza, and settings in [default] apply to every input stanza that does not set them itself: [default]
- index = default
- Tier 3: App Default ( $SPLUNK_HOME/etc/apps/Splunk_TA_nix/default/inputs.conf ) [monitor:///var/log/secure]
- sourcetype = linux_secure
- index = os
- Tier 2: App Local ( $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/inputs.conf ) [monitor:///var/log/secure]
- index = linux_security
- ignoreOlderThan = 7d
- Tier 1: System Local ( $SPLUNK_HOME/etc/system/local/inputs.conf ) [monitor:///var/log/secure]
- disabled = 1
- Step-by-Step Resolution Analysis
When splunkd evaluates this configuration in the Global context, it merges the stanza attribute-by-attribute across the four tiers:
Attribute Defined In Winning Tier Effective Resolved Value Technical Rationale
disabled system/local System local (Tier 1) 1 The only definition, and the highest tier. The input is disabled.
index system/default ( [default] ), apps/TA/default , apps/TA/local App local (Tier 2) linux_security App local beats App default and System default; System local does not set it.
sourcetype apps/TA/default App default (Tier 3) linux_secure No higher tier overrides it.
ignoreOlderThan apps/TA/local App local (Tier 2) 7d A non-conflicting attribute is simply carried into the merged stanza.
- Final Effective Runtime Stanza Calculated by Splunk: [monitor:///var/log/secure]
- disabled = 1
- index = linux_security
- sourcetype = linux_secure
- ignoreOlderThan = 7d
The result combines settings from three layers ( system/local , apps/Splunk_TA_nix/local , and apps/Splunk_TA_nix/default ), plus the [default] fallback from system/default . Splunk merges attributes; it does not discard whole files or stanzas.