Study guide
3.3 Using btool to Inspect & Debug Configurations
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
3.3 Using btool to Inspect & Debug Configurations
Key Takeaways Explain This Topic AI
The splunk btool command-line utility evaluates on-disk configuration files across all directory tiers and calculates the effective, merged runtime configuration.
The --debug flag is the most powerful diagnostic parameter for btool, annotating every stanza and attribute with the absolute file path where the winning setting was defined.
The splunk btool check command validates configuration files against schema specifications (*.spec) in $SPLUNK_HOME/etc/system/README/ to identify typos and syntax errors.
btool functions as an offline parser of on-disk configuration files; it does not read in-memory daemon state or require the splunkd process to be running.
By default, btool evaluates configurations in the Global context; administrators must pass --app=<app_name> or --user=<username> to inspect App/User search-time configurations accurately.
Last updated: September 2026
Using btool to Inspect & Debug Configurations
In an enterprise Splunk deployment with dozens of apps, Technology Add-ons, and system-level overrides, manually determining which configuration file defined a specific setting is nearly impossible. Searching through directories using grep or find fails to account for Splunk's complex precedence rules, lexical sorting, and attribute-level merging.
To solve this challenge, Splunk provides btool —a purpose-built command-line diagnostic utility that executes the exact same parsing and layering logic as the core splunkd engine, revealing the final effective configuration on disk.
The Purpose and Mechanics of btool
The primary purpose of btool is to answer two critical administrative questions:
- What is the effective value of a configuration setting? ( btool ... list )
- Which exact file on disk caused that setting to win? ( btool ... list --debug )
btool traverses the directory hierarchy under $SPLUNK_HOME/etc , parses every relevant .conf file, resolves all precedence conflicts, and outputs the resulting composite configuration to standard output ( stdout ). Diagnostic Flow of btool:
- etc/system/default/ ──┐
- etc/apps/*/default/ ──┼──> [ btool Layering Engine ] ──> Merged Effective Output
- etc/apps/*/local/ ──┤ (Applies Global or (with optional file
- etc/system/local/ ──┘ App/User Precedence) provenance via --debug)
- Core Command Syntax
The btool utility is invoked through the Splunk CLI wrapper located at $SPLUNK_HOME/bin/splunk . The fundamental syntax structure is: splunk btool <conf_file_prefix> list [stanza_name] [options]
Important
When specifying the configuration file prefix, omit the .conf file extension. To inspect inputs.conf , specify inputs ; to inspect server.conf , specify server .
- Essential Command Invocations
- 1. Listing the Entire Effective Configuration
To display the complete merged configuration for a specific configuration file across all stanzas: $SPLUNK_HOME/bin/splunk btool inputs list
- $SPLUNK_HOME/bin/splunk btool props list
- $SPLUNK_HOME/bin/splunk btool indexes list
- 2. Filtering by a Specific Stanza
Because configuration outputs can span thousands of lines, you can isolate an individual stanza by appending the exact stanza name in brackets: $SPLUNK_HOME/bin/splunk btool inputs list [monitor:///var/log/messages]
- $SPLUNK_HOME/bin/splunk btool indexes list [web_traffic]
- $SPLUNK_HOME/bin/splunk btool server list [clustering]
- 3. Combining with Operating System Utilities
Because btool outputs raw text, it is routinely piped to standard administrative command-line tools: # Page through effective settings
$SPLUNK_HOME/bin/splunk btool server list | less
- # Search for specific attribute keys
- $SPLUNK_HOME/bin/splunk btool outputs list | grep -E "(server|autoLB|useACK)"
- Tracing Attribute Provenance with --debug
The most important diagnostic flag available in btool is --debug . When passed to the list command, --debug prefixes every single line of output with the absolute filesystem path of the configuration file where that specific stanza header or attribute key was defined. $SPLUNK_HOME/bin/splunk btool inputs list --debug
Deconstructing --debug Output
Examine this sample output obtained by running btool on an indexer: /opt/splunk/etc/system/local/inputs.conf [monitor:///var/log/secure]
- /opt/splunk/etc/system/local/inputs.conf disabled = 1
- /opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf ignoreOlderThan = 7d
- /opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf index = linux_security
- /opt/splunk/etc/apps/Splunk_TA_nix/default/inputs.conf sourcetype = linux_secure
- /opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
Notice how immediately clear the root cause of any behavior becomes:
The stanza header is listed with the highest-priority file that contains it ( system/local/inputs.conf ).
The attribute disabled = 1 was enforced by system/local/inputs.conf , completely overriding any app settings.
The attribute index = linux_security originated from apps/Splunk_TA_nix/local/inputs.conf .
The attribute sourcetype = linux_secure was supplied by apps/Splunk_TA_nix/default/inputs.conf .
The attributes ignoreOlderThan and index came from apps/Splunk_TA_nix/local/inputs.conf , and host was inherited from the [default] stanza in system/default/inputs.conf .
Without btool --debug , an administrator attempting to troubleshoot why /var/log/secure is not ingesting data might spend hours checking Splunk_TA_nix , completely unaware that an override in system/local/inputs.conf disabled the input.
Validating Syntax and Integrity with btool check
In addition to merging configurations, btool includes a built-in syntax verification engine executed via btool check : $SPLUNK_HOME/bin/splunk btool check
How btool check Works
Inside $SPLUNK_HOME/etc/system/README/ , Splunk maintains configuration specification files ( *.spec ) and example files ( *.example ) for every supported .conf file (e.g., inputs.conf.spec , props.conf.spec ). These specification files define:
- All legitimate stanza regex formats
- All supported attribute keys
- Expected value types (integer, boolean, string)
When btool check runs, it scans the .conf files and compares their keys against these .spec files. A mistyped setting name is reported as an invalid key with its file, line number, and value. btool check does not suggest the setting you meant, so compare against the spec yourself.
Example btool check Output: Checking: /opt/splunk/etc/apps/search/local/inputs.conf
Invalid key in stanza [monitor:///var/log/httpd/access.log] in /opt/splunk/etc/apps/search/local/inputs.conf, line 4: indx (value: web)
Checking: /opt/splunk/etc/apps/custom_ta/local/props.conf
Invalid key in stanza [custom_syslog] in /opt/splunk/etc/apps/custom_ta/local/props.conf, line 2: TIME_FORM (value: %Y-%m-%d)
Appending --debug to btool check prints the status of every file examined, including valid files, which is useful when diagnosing file permission issues: $SPLUNK_HOME/bin/splunk btool check --debug
Contextual Evaluation: The --app and --user Parameters
By default, btool evaluates configurations strictly within the Global context . While this works perfectly for inputs.conf , outputs.conf , server.conf , and indexes.conf , it produces inaccurate results when debugging search-time knowledge objects (such as savedsearches.conf or search-time props.conf ).
To instruct btool to evaluate configurations through the lens of a specific application or user, pass the --app and --user flags:
- Evaluating Within an Application Scope $SPLUNK_HOME/bin/splunk btool props list --app=Splunk_TA_nix
- $SPLUNK_HOME/bin/splunk btool savedsearches list --app=search
Evaluating Within a Specific User Scope $SPLUNK_HOME/bin/splunk btool savedsearches list --app=search --user=analyst_jane --debug
This command evaluates settings starting from $SPLUNK_HOME/etc/users/analyst_jane/search/local/savedsearches.conf down through the App/User precedence chain, allowing administrators to diagnose why a user's private report or dashboard is behaving unexpectedly.
Critical Operational Pitfalls of btool
While btool is an indispensable utility, misunderstanding its operational boundaries leads to significant administrative confusion. Administrators must keep four critical caveats in mind:
1. On-Disk State vs. In-Memory Daemon State
btool reads files directly from the filesystem at the moment the command is executed. It does not query the running splunkd process or inspect in-memory memory structures.
If an administrator modifies an inputs.conf file on disk, running btool immediately displays the new setting.
However, the running splunkd service continues operating under its old configuration until the service is restarted ( splunk restart ) or the relevant subsystem is explicitly reloaded (for example splunk reload deploy-server on a deployment server, or Splunk Web's /debug/refresh endpoint for many knowledge objects). When in doubt, restart.
Pitfall: An administrator sees the correct setting in btool and assumes the live system is using it, when in reality splunkd has not reloaded the file.
2. Offline Operation & Service Independence
Because btool operates strictly as a static file parser, it does not require splunkd to be running. It can be executed while Splunk is completely stopped. This makes it an ideal disaster recovery tool for troubleshooting startup failures caused by corrupt configurations or invalid stanzas.
3. Management Node Boundary
Running btool on a Deployment Server or Cluster Manager evaluates only the configurations active on that management node itself. It cannot evaluate what a downstream forwarder or peer indexer will compute after receiving a deployment bundle.
To troubleshoot forwarder configuration, btool must be run on the forwarder itself.
To troubleshoot indexer cluster configuration, btool must be run on an individual peer indexer.
4. Operating System User Permissions
Always run btool as the operating system user that owns the Splunk installation (typically the splunk service account). If executed as root without preserving the environment, btool may generate temporary cache files or lock files owned by root , preventing splunkd from accessing them later.
Essential btool Diagnostic Reference Table
- Administrative Task Recommended Command Syntax
- List effective inputs with file sources splunk btool inputs list --debug
- Inspect a specific monitor stanza splunk btool inputs list [monitor:///var/log/syslog] --debug
- Validate all configuration files for typos splunk btool check
- Trace winning props for an app splunk btool props list --app=Splunk_TA_nix --debug
- Check private saved searches for a user splunk btool savedsearches list --app=search --user=<username> --debug
- Inspect effective forwarder load balancing splunk btool outputs list [tcpout:production_indexers] --debug
- Verify index paths and retention limits splunk btool indexes list [main] --debug