Study guide
7.1 CLI Administration & Basic Data Onboarding Workflows
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
7.1 CLI Administration & Basic Data Onboarding Workflows
Key Takeaways Explain This Topic AI
Every input sets the same basics: index (default main), sourcetype, host, source, and disabled; Splunk Web's Add Data wizard, the CLI, and inputs.conf all write these same keys.
CLI commands talk to splunkd on the management port (TCP 8089) and authenticate with -auth user:password or a session cached by an interactive login.
splunk add monitor <path> writes a persistent [monitor://] stanza to inputs.conf and tracks read progress in the fishbucket; splunk add oneshot <path> indexes a file once without creating a monitor stanza.
On a universal forwarder, splunk add forward-server <host>:9997 writes a [tcpout] target group to outputs.conf, and splunk list forward-server shows active and configured-but-inactive destinations.
splunk btool <conf> list --debug shows merged settings with the file each one came from, and splunk btool check reports invalid keys.
Last updated: September 2026
CLI Administration & Basic Data Onboarding Workflows
Quick Summary: Proficient Splunk administration relies heavily on the Command Line Interface (CLI). From managing the core splunkd daemon lifecycle to onboarding data sources and verifying configuration precedence with btool , the CLI allows administrators to execute precise, scriptable operations. Understanding CLI syntax, token-based authentication mechanisms, and the underlying configuration files altered by CLI commands is essential for enterprise operations.
The Splunk CLI Environment & Daemon Lifecycle Controls
The primary executable for all command-line operations is $SPLUNK_HOME/bin/splunk .
In Unix/Linux environments, $SPLUNK_HOME defaults to /opt/splunk for Splunk Enterprise or /opt/splunkforwarder for Universal Forwarders.
In Windows environments, the default path is C:\Program Files\Splunk or C:\Program Files\SplunkUniversalForwarder .
Administrators typically add $SPLUNK_HOME/bin to the system $PATH or execute commands directly from the bin directory.
Core Daemon Lifecycle Commands
Managing the splunkd background daemon involves several critical administrative commands: # Start the Splunk daemon
splunk start
- # Start Splunk, automatically accepting the license agreement without prompt
- splunk start --accept-license --answer-yes
- # Stop the Splunk daemon gracefully
- splunk stop
- # Perform a clean restart (stops daemon, flushes queues, reloads configs, restarts)
- splunk restart
- # Check whether splunkd is currently running and view its Process ID (PID)
- splunk status
- # Configure Splunk to launch automatically on host boot under a dedicated service user
- splunk enable boot-start -user splunk
- Operational Mechanics of Lifecycle Commands
splunk start : Validates critical configuration file syntax, verifies port availability (checking that 8089, 8000, etc., are not bound by other processes), checks for conflicts such as a stale lock or PID file, and launches splunkd (which in turn starts helper processes such as the KV store).
splunk stop : Initiates a graceful shutdown sequence. It stops accepting new ingestion connections, signals running search jobs to terminate or finalize, flushes memory queues to disk, closes open bucket file handles, and terminates the daemon cleanly. Abruptly killing the process ( kill -9 ) can corrupt active hot bucket journal files.
splunk enable boot-start : Generates system startup scripts. In modern Linux distributions running systemd, running splunk enable boot-start -user splunk -systemd-managed 1 creates a validated systemd unit file ( splunk.service ) configured with appropriate user privileges, resource limits ( ulimits ), and restart policies.
Authentication, Credentials & Session Token Mechanics
Every CLI command that inspects or alters the state of Splunk communicates with the splunkd process over the management port (default TCP 8089) via REST API endpoints. Consequently, CLI commands require administrative authentication.
Providing Credentials
Administrators can supply credentials using several methods:
Inline Credentials ( -auth flag): splunk list forward-server -auth admin:MyPassword123
Caution: Passing credentials directly via -auth exposes plaintext passwords in the shell process table ( ps aux ) and in user command history ( .bash_history ).
Interactive Prompt: If a command requiring authentication is executed without the -auth flag: splunk list forward-server
The CLI prompts interactively: Your session is invalid. Please login.
Splunk username: admin
Password:
Session Token Storage: Upon successful interactive authentication, the CLI generates an authentication session token and caches it on the host filesystem:
Linux/macOS: $HOME/.splunk/authToken (or $SPLUNK_HOME/var/run/splunk/authToken_8089 )
Windows: %USERPROFILE%\.splunk\authToken Subsequent CLI commands automatically read this cached token, permitting administrative operations without prompting until the token reaches its expiration timeout (configured by sessionTimeout in server.conf , defaulting to 1 hour).
Dedicated CLI Login / Logout: Administrators can explicitly establish or terminate a CLI session: # Log in and cache a session token
splunk login -auth admin:MyPassword123
- # Invalidate and delete the cached session token
- splunk logout
- The Basic Settings Every Input Carries
Whether you create an input in Splunk Web, with the CLI, or by editing inputs.conf , every input sets the same small group of basics. These defaults apply before any parsing happens:
- Setting What it controls Default if you omit it
- index Destination index for the data default , which resolves to main
sourcetype Data format; drives parsing and search-time knowledge Automatic source type assignment (avoid this in production)
host Machine the data came from The instance's own host name ( [default] host in inputs.conf ); network inputs use connection_host
- source Where the data came from File path for monitor inputs, tcp:<port> / udp:<port> for network inputs
- disabled Whether the input runs false (enabled)
In Splunk Web, Settings > Add Data walks through the same choices. You pick a method (Upload, Monitor, or Forward), select the source, set the source type on the Set Source Type page, and then choose app context, host, and index on the Input Settings page. The CLI writes the same keys into an inputs.conf file.
CLI Data Onboarding Commands & Workflows
While graphical data onboarding is accessible via Splunk Web, CLI commands are standard for headless installations, automated server provisioning, and forwarder deployments.
1. Persistent Monitoring: splunk add monitor
The splunk add monitor command instructs Splunk to monitor a file or directory continuously. # Monitor a specific log file with explicit index and sourcetype
splunk add monitor /var/log/secure -index os_security -sourcetype linux_secure
- # Monitor an entire directory (add a whitelist later in inputs.conf if needed)
- splunk add monitor /var/log/httpd/ -index web -sourcetype apache_access
- Under the Hood: What splunk add monitor Modifies
The command writes a new monitor stanza directly into $SPLUNK_HOME/etc/apps/search/local/inputs.conf (or $SPLUNK_HOME/etc/system/local/inputs.conf depending on context): [monitor:///var/log/secure]
- index = os_security
- sourcetype = linux_secure
- disabled = 0
The Fishbucket Registration: splunkd reads the first 256 bytes of the target file to compute a beginning CRC fingerprint. That CRC, the seek address it has read up to, and a CRC of the data at that point are stored in Splunk's tracking database, the fishbucket ( $SPLUNK_DB/fishbucket/splunk_private_db , exposed as the _thefishbucket index). This state tracking guarantees that if Splunk restarts, it resumes reading from the exact byte offset where it left off, avoiding duplicate events.
2. Batch One-Time Loading: splunk add oneshot
When an administrator needs to index an archived log file or historical dump without establishing continuous file tracking, splunk add oneshot provides a synchronous batch mechanism: splunk add oneshot /var/log/historical_dump_2025.log -index legacy_data -sourcetype legacy_app
Key Differences: add monitor vs. add oneshot
- Feature splunk add monitor splunk add oneshot
- Ingestion Behavior Continuous, persistent monitoring Single batch execution; stops at EOF
- Configuration Stanza Created Writes [monitor://...] stanza to inputs.conf Does NOT write to inputs.conf
Fishbucket State Tracking Actively maintains seek pointers in _thefishbucket Bypasses persistent fishbucket tracking
Handling Appended Data Automatically detects and indexes new appended bytes Ignores subsequent file modifications
Primary Use Case Live production application and system logs Historical archives, one-off diagnostics, sandbox testing
3. Forwarder Target Configuration: splunk add forward-server
Configuring forwarder egress to route data to downstream indexers is performed using forward-server CLI commands: # Add an active indexer receiving target
splunk add forward-server indexer01.corp.internal:9997 -auth admin:<password>
- # Add a second indexer target for automatic load balancing
- splunk add forward-server indexer02.corp.internal:9997 -auth admin:<password>
- # List all configured forwarder destinations and their connection status
- splunk list forward-server
- # Remove a decommissioned indexer from the forwarder configuration
- splunk remove forward-server indexer01.corp.internal:9997
- Under the Hood: What add forward-server Modifies
- The command writes target group stanzas into $SPLUNK_HOME/etc/system/local/outputs.conf : [tcpout]
- defaultGroup = default-autolb-group
- [tcpout:default-autolb-group]
- server = indexer01.corp.internal:9997, indexer02.corp.internal:9997
splunk list forward-server queries splunkd over port 8089 to report the status of TCP connections, identifying whether each indexer is Active (currently receiving data) or Configured (available in the failover/load-balancing pool).
Configuration Verification & Inspection via CLI
Because Splunk evaluates multiple layers of .conf files across system , apps , and user directories, diagnosing the effective runtime configuration is impossible by reading a single file. The CLI provides specialized tools to inspect merged configuration state:
1. Inspecting Resolved Settings with splunk btool
btool is the premier diagnostic command for Splunk administrators. It simulates the exact configuration merging logic that splunkd executes at startup: # View the merged settings for a configuration file type (e.g. inputs.conf)
splunk btool inputs list
- # Trace the exact file path and line number where each attribute was set
- splunk btool inputs list --debug
- # Filter for a specific stanza within the resolved configuration
- splunk btool inputs list --debug | grep -A 5 "monitor:///var/log/secure"
- # Inspect server.conf across all apps to verify the license manager setting (manager_uri)
- splunk btool server list --debug | grep -A 4 "\[license\]"
- 2. Validating Configuration Syntax with btool check
To detect typos, illegal attribute names, and syntax errors before restarting the production daemon: splunk btool check
If an administrator misspelled an attribute (for example, typing sourctyp = custom instead of sourcetype = custom in inputs.conf ), splunk btool check reports an "Invalid key in stanza" line naming the file, line number, and key.
3. Hot-Reloading Subsystems Without Daemon Restart
While changes to low-level networking or cluster peering require a full restart ( splunk restart ), several operational subsystems can be reloaded dynamically via the CLI: # Reload Deployment Server classes and bundles
splunk reload deploy-server
- # Reload authentication settings (e.g. after editing LDAP configurations)
- splunk reload auth
- # Force splunkd to reload monitor inputs without dropping active connections
- splunk _internal call /services/data/inputs/monitor/_reload -auth admin:password
- Administrative Pitfalls & Best Practices
Credential Exposure in Command Line Arguments: Avoid typing -auth username:password in scripts or shared terminal sessions. Use an interactive splunk login session (which caches a session token for the OS user) or a dedicated automation account with the minimum capabilities it needs.
Local Stanza Pollution: CLI commands like splunk add monitor default to writing into the search app's local directory ( $SPLUNK_HOME/etc/apps/search/local/inputs.conf ) or $SPLUNK_HOME/etc/system/local/inputs.conf . In structured enterprise environments, administrators should package inputs inside modular, version-controlled Technology Add-ons (e.g., TA-linux-inputs/default/inputs.conf ) distributed centrally via the Deployment Server rather than running manual CLI input additions on production servers.
Fishbucket Collision after File Moves: If a log file is indexed via splunk add oneshot and later moved into a monitored directory, Splunk will index it again, because a oneshot upload does not leave a monitor checkpoint in the fishbucket.
Missing Service User Context: Running splunk start as the root superuser after Splunk was installed to run under an unprivileged user (e.g. splunk ) alters file permissions in $SPLUNK_HOME/var/ . When the unprivileged user subsequently attempts to start Splunk, splunkd fails with permission-denied errors on files it can no longer write. Always execute CLI commands as the dedicated splunk service user.