Study guide
13.3 Scripted Inputs & PowerShell Data Collection
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
13.3 Scripted Inputs & PowerShell Data Collection
Key Takeaways Explain This Topic AI
Scripted inputs are [script://<cmd>] stanzas; interval sets seconds (default 60) or a cron schedule, 0 reruns the script continuously, and -1 runs it once at startup.
The command must live in $SPLUNK_HOME/etc/system/bin, $SPLUNK_HOME/etc/apps/<app>/bin, or $SPLUNK_HOME/bin/scripts; a .path pointer file can reference a program elsewhere.
Standard output becomes event data, while error messages from scripts appear in splunkd.log under the ExecProcessor component.
passAuth = <user> passes a session key for that user to the script on standard input, so the script can call the Splunk REST API without stored passwords.
Universal forwarders do not bundle Python; Windows forwarders can run [powershell://<name>] inputs with script and schedule settings, and since 9.1 they install as the least-privileged NT SERVICE\SplunkForwarder virtual account.
Last updated: September 2026
Scripted Inputs & PowerShell Data Collection
Quick Summary: Scripted inputs run a command on a schedule and index whatever it writes to standard output, which lets Splunk collect data from APIs, command-line tools, and system utilities. Know where scripts must live, how interval works (seconds, cron, 0 for continuous, -1 for once), that standard error goes to splunkd.log , how passAuth hands a session key to the script, and how Windows [powershell://] inputs are scheduled.
Anatomy of a Scripted Input in inputs.conf
A scripted input instructs Splunk's background daemon ( splunkd ) to periodically invoke an external script or executable binary, capture its generated output, and stream that output into the data ingestion pipeline.
Where Scripts Must Live
inputs.conf.spec restricts the command to one of these directories:
- $SPLUNK_HOME/etc/system/bin/
- $SPLUNK_HOME/etc/apps/<app_name>/bin/
- $SPLUNK_HOME/bin/scripts/
The path can be absolute, can use $SPLUNK_HOME , or can start with . to mean "inside the current app" (for example ./bin/collect.sh ). To run a program located elsewhere, place a .path file in one of the allowed directories. It is a one-line pointer to the real command and its arguments, and it may point anywhere on the file system. # App-relative script (recommended)
- [script://./bin/collect_cloud_metrics.sh]
- interval = 300
- sourcetype = cloud:metrics
- index = ops_metrics
- # Pointer file for a program outside the allowed directories
- [script://$SPLUNK_HOME/etc/apps/org_net_audit/bin/network_audit.path]
- interval = 0 2 * * *
- sourcetype = net:audit
- index = security
- Scheduling Mechanisms: Seconds, Cron Expressions, and Continuous Scripts
The interval setting in inputs.conf controls when splunkd runs the script:
- Value Behavior
- Number of seconds (default 60 ) Run every n seconds; fractional values are allowed
- Cron schedule, e.g. 0 */2 * * * Run on that schedule. With a cron schedule the script is not run at startup
- 0 Run continuously : as soon as the script exits, the input restarts it
- 1 Run once , at startup
- Periodic vs. Continuous Scripts
Periodic scripts ( interval > 0 or cron): the script runs, writes events to standard output, and exits. This suits polling an API, collecting system statistics, or checking disk space.
Continuous scripts ( interval = 0 ): the script is expected to keep running, for example streaming from a message queue. If it exits or crashes, Splunk starts it again immediately, so a failing script can restart in a tight loop. Make sure errors are logged to standard error.
Execution Environment & Process Model
Scripted inputs are managed by splunkd 's internal execProcessor subsystem. The runtime environment possesses specific operating system characteristics that administrators must take into account:
1. Operating System User Context & Permissions
A scripted input runs as the same operating system account as splunkd : typically the splunk account on Linux, and on Windows the account the Splunk service runs as (a virtual account, Local System, or a domain account).
The script has no more and no less permission than that OS user:
If a script attempts to read /var/log/messages or query hardware devices that require root permissions, the execution will fail with permission denied errors.
Sudo privileges should be avoided; if required, configure strict /etc/sudoers rules for the specific command.
2. Runtime Environment Variables
Scripts run with Splunk's environment, including SPLUNK_HOME , so they can build paths relative to the installation instead of hard-coding them. Run splunk envvars to see the variables Splunk sets, and use splunk cmd <program> to run a program with that environment when testing.
3. Python Runtime & Environment Isolation
Splunk Enterprise (including heavy forwarders) bundles its own Python 3 runtime ( $SPLUNK_HOME/bin/splunk cmd python3 ). The universal forwarder does not . On a UF, a scripted input must be an executable, shell, batch, or PowerShell script, or must call an interpreter already installed on the host.
For Python scripts on Splunk Enterprise, python.version in the input stanza can select the Python version.
Splunk's environment points library paths at Splunk's own libraries, which can conflict with system programs or third-party Python packages that a script calls.
The Dependency Conflict Hazard: If a script relies on external Python modules (e.g., requests , boto3 , or database drivers) that are not bundled with Splunk, administrators must either:
Vendor the third-party libraries directly into the application's bin/lib/ directory and append that directory to sys.path within the script.
Use a shell wrapper script that invokes the host system's native Python virtual environment ( venv ), sanitizing LD_LIBRARY_PATH and PYTHONPATH prior to execution.
4. REST API Authentication via passAuth
Often, a scripted input needs to query Splunk's own internal REST API (TCP port 8089) to fetch configuration settings, trigger saved searches, or inspect cluster state. Hard-coding an admin password in a script file is a serious security risk.
- Splunk solves this using the passAuth parameter in inputs.conf : [script://./bin/audit_forwarders.py]
- disabled = false
- interval = 3600
- passAuth = splunk-system-user
When passAuth is configured, splunkd generates a temporary, short-lived session token with the privileges of the specified user (e.g., splunk-system-user or admin ).
Prior to executing the script, splunkd writes this session token to the child process's standard input ( stdin ) .
The script reads the token from stdin and passes it in the Authorization header when querying https://localhost:8089 : import sys, urllib.request, ssl
- # Read session token injected via stdin by passAuth
- session_key = sys.stdin.readline().strip()
- req = urllib.request.Request("https://localhost:8089/services/server/info")
- req.add_header("Authorization", f"Splunk {session_key}")
# Execute authenticated REST call...
Standard Streams: stdout vs. stderr and The Buffering Trap
The interaction between splunkd and the script's output streams is rigidly segregated: +----------------------------+
- | Scripted Input Process |
- | (Python / Bash / Binary) |
- +----------------------------+
- | |
- stdout Stream | | stderr Stream
- v v
- +-----------------------------------+ +-----------------------------------+
- | Splunk execProcessor | | Splunk Logging Engine |
- | (Injects into Ingestion Pipeline)| | (Captures diagnostic messages) |
- +-----------------------------------+ +-----------------------------------+
- | |
- v v
- inputQueue -> parsingQueue $SPLUNK_HOME/var/log/splunk/
- (Indexed as Event Data) splunkd.log (WARN / ERROR)
- 1. Standard Output ( stdout ) -> Ingestion Pipeline
Everything the script writes to stdout is captured by execProcessor and injected directly into the Splunk ingestion pipeline:
The byte stream enters inputQueue tagged with the source , sourcetype , index , and host defined in the inputs.conf stanza.
It then passes to parsingQueue , where line breaking ( LINE_BREAKER ), timestamp extraction ( TIME_PREFIX , TIME_FORMAT ), and transformations occur.
Best practice: Format script output as clean, single-line structured strings (e.g., key-value pairs or JSON objects) containing explicit timestamps to streamline indexing.
2. Standard Error ( stderr ) -> splunkd.log
Anything written to stderr bypasses the ingestion pipeline completely:
splunkd intercepts stderr and writes it directly to $SPLUNK_HOME/var/log/splunk/splunkd.log .
Log entries appear under the ExecProcessor component with a log level of ERROR or WARN : 09-23-2026 14:35:10.124 -0400 ERROR ExecProcessor [14205 ExecProcessor] -
message from "/opt/splunk/etc/apps/TA-cloud/bin/collect.py" urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>
Administrators troubleshooting failing scripted inputs should immediately inspect splunkd.log for ExecProcessor messages: index=_internal sourcetype=splunkd component=ExecProcessor
3. The Critical Developer Trap: Output Buffering
In many programming languages (especially Python and C/C++), standard output is block-buffered rather than line-buffered when redirected to an operating system pipe.
When splunkd executes a script, stdout is connected to a pipe, not a terminal (TTY).
If a Python script prints events in a loop, Python may hold the data in an internal 4 KB or 8 KB memory buffer instead of writing it immediately to the pipe.
A periodic script flushes its buffer when it exits, but a continuous script ( interval = 0 ) that never exits delivers events only when the buffer fills, so they arrive late and in bursts.
The Fix: Scripts must explicitly flush stdout after printing events:
In Python 3: print(event_data, flush=True) or sys.stdout.flush() .
Or set PYTHONUNBUFFERED=1 inside a wrapper script before launching the interpreter.
Script Error Handling, Timeouts, and Resource Leaks
Failing to design scripted inputs defensively can destabilize the entire host system:
1. Non-Zero Exit Codes
A script that fails usually leaves evidence in two places: the messages it wrote to standard error, and the ExecProcessor entries in splunkd.log . A scheduled script simply runs again at its next interval; a continuous script ( interval = 0 ) is restarted as soon as it exits.
2. Execution Overlap Prevention
If a script scheduled with interval = 60 sometimes takes 90 seconds, do not rely on Splunk to manage the overlap.
Design for it: Keep each run shorter than the interval, and give network calls explicit timeouts. A script that hangs keeps its process and resources until it exits or Splunk stops.
3. Hanging Processes and Resource Leaks
Splunk does not know when a script is "stuck". A script that waits forever on the network keeps running.
If a script initiates a network connection to a third-party server that hangs indefinitely without a TCP timeout, the script process remains alive in the operating system process table indefinitely.
Over time, hung scripts can hold process slots, file handles, sockets, and memory on the host.
Defensive Coding Requirements:
Always specify explicit timeouts on all network socket connections, HTTP requests, and database queries (e.g., requests.get(url, timeout=15) ).
Implement signal handlers ( SIGTERM , SIGINT ) in scripts to ensure graceful shutdown when splunkd stops or restarts.
Windows PowerShell Inputs
On Microsoft Windows platforms, administrative data collection frequently relies on PowerShell. While administrators can execute PowerShell via generic [script://...] stanzas (e.g., calling powershell.exe -File script.ps1 ), Splunk on Windows provides a native mechanism: the PowerShell input ( [powershell://<name>] ), which runs Windows PowerShell (version 3 or later) commands or scripts.
Stanza Syntax in inputs.conf [powershell://InstalledSoftware]
script = Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Select-Object DisplayName, DisplayVersion, Publisher, InstallDate
- schedule = 0 0 * * *
- sourcetype = Windows:InstalledSoftware
- index = windows_inventory
- [powershell://QueryADUsers]
- script = . "$SplunkHome\etc\apps\TA-Windows\bin\get_locked_accounts.ps1"
- schedule = */15 * * * *
- sourcetype = ActiveDirectory:LockedUsers
- index = msad
- Key Parameters for PowerShell Inputs
script : Contains either inline PowerShell cmdlets/pipeline expressions or a dot-sourced path to an external .ps1 script file.
schedule : How often to run, in seconds or as a cron schedule. Whichever you choose, the command also runs once when the instance starts . If schedule is not set, it runs only at startup.
sourcetype : Stamped on all resulting output.
PowerShell Execution Policies & Security Controls
Windows enforces execution policies to prevent unauthorized script execution:
Policies: Restricted , AllSigned , RemoteSigned , Unrestricted , Bypass .
Enterprise Group Policy (GPO) Overrides:
Domain Group Policy can enforce machine-wide execution policies, so test scripts under the target machine's effective policy.
Environments that use AppLocker or Windows Defender Application Control (WDAC) may force PowerShell into Constrained Language Mode , which blocks some .NET and COM features that complex scripts use.
Service Account Privileges & Active Directory Data Collection
Since version 9.1 , the Windows universal forwarder installer creates and uses a least-privileged virtual account, NT SERVICE\SplunkForwarder , by default. You can instead choose Local System or a domain account during installation.
Local accounts vs. the domain: Neither the virtual account nor Local System carries a domain user's identity. Both reach network resources as the computer account ( DOMAIN\COMPUTER$ ), which usually cannot query domain controllers or remote systems the way an administrator expects.
Domain Service Account Configuration:
When configuring a forwarder or Heavy Forwarder to harvest Active Directory topology, group membership changes, DNS server records, or Hyper-V metrics across multiple domain systems, the Splunk service must be configured to run as a dedicated Domain Service Account or a Group Managed Service Account (gMSA) .
Following the principle of least privilege, the domain account does not need Domain Admin rights; it should be granted delegated read permissions on required Active Directory organizational units (OUs) and placed in the built-in Event Log Readers and Performance Monitor Users security groups.
Technical Comparison: Scripted Inputs vs. PowerShell Inputs
- Feature Scripted input ( [script://...] ) PowerShell input ( [powershell://<name>] )
- Platforms All platforms Splunk runs on Windows only (Windows PowerShell 3 or later)
What runs An executable or script in an allowed directory, or a .path pointer An inline PowerShell command or a dot-sourced .ps1 script ( script = )
Timing setting interval (seconds or cron; 0 = continuous, -1 = once) schedule (seconds or cron); always runs once at startup too
Output Text written to standard output becomes events PowerShell output objects are turned into events
Splunk REST access passAuth = <user> passes a session key on standard input Supply credentials through your own script logic