Study guide
12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings
Key Takeaways Explain This Topic AI
whitelist and blacklist in a monitor stanza are regular expressions matched against the full file path; if a file matches both, it is not monitored.
Monitor stanzas take a single whitelist and a single blacklist regex; combine alternatives with | (numbered whitelist1-9 settings belong to Windows event log inputs).
Files are identified by a CRC of their first 256 bytes; initCrcLength (256-1048576) lengthens it and crcSalt = <SOURCE> adds the file path.
ignoreOlderThan skips files whose modification time is older than the limit, and ignored files are not checked again until a restart or input reconfiguration, even if they are later updated.
followTail = 1 starts reading at the end of a file the first time it is seen; enable it only temporarily, because it drops existing content of any file that looks new.
Last updated: September 2026
Monitor Input Tuning: Whitelists, Blacklists & CRC Settings
Quick Summary: In large-scale enterprise environments, pointing a monitor input at a busy directory tree without precise tuning can saturate indexing pipelines with irrelevant files or cause silent data loss due to hash collisions. Administrators control file selection using Perl-Compatible Regular Expressions (PCRE) with whitelist and blacklist directives, where blacklist rules strictly supersede whitelists. Furthermore, understanding the 256-byte Cyclic Redundancy Check (CRC) mechanism—and using crcSalt or initCrcLength to eliminate identical header collisions—is critical to ensuring data integrity.
Filtering Files with Regular Expressions: Whitelists & Blacklists
When a monitor input stanza targets a directory (e.g., [monitor:///var/log] ), Splunk attempts to ingest every readable file discovered within that directory structure. To include or exclude specific subsets of files, administrators configure whitelist and blacklist settings in inputs.conf .
Regex Evaluation Rules and Syntax
Both whitelist and blacklist accept Perl-Compatible Regular Expressions (PCRE). A common administrative misconception is that these regular expressions match only the file's base name (e.g., access.log ). In reality, Splunk evaluates the regular expression against the entire fully qualified absolute file path . # Example inputs.conf file filter configuration
- [monitor:///var/log/httpd]
- disabled = false
- index = web_prod
- sourcetype = access_combined
- # Whitelist only .log files
- whitelist = \.log$
- # Blacklist rotated, compressed, or temporary files
- blacklist = (\.gz$|\.bak$|\.tmp$|/archive/)
- Combining Several Patterns
A monitor stanza has one whitelist and one blacklist setting. (Numbered whitelist1 – whitelist9 settings exist only for Windows event log inputs.) To combine alternatives, use regex alternation: [monitor:///opt/data/logs]
- whitelist = \.(log|json)$
- blacklist = (\.archive\.log$|/temp/|\.gz$)
Do not add comments at the end of a setting line, because the text becomes part of the regex.
The Fundamental Precedence Rule: Blacklist ALWAYS Supersedes Whitelist
When both whitelist and blacklist rules are applied within a monitor stanza, Splunk adheres to a strict, non-negotiable evaluation hierarchy: Ingestion Decision = ( Matches Whitelist ) ∧ ¬ ( Matches Blacklist ) \text{Ingestion Decision} = (\text{Matches Whitelist}) \land \neg (\text{Matches Blacklist}) Ingestion Decision = ( Matches Whitelist ) ∧ ¬ ( Matches Blacklist )
If a file matches a whitelist rule but also matches a blacklist rule, the file is unconditionally blacklisted and dropped . Blacklist evaluation takes absolute precedence over whitelist evaluation under all circumstances.
- Whitelist Defined? Blacklist Defined? Matches Whitelist? Matches Blacklist? Action Taken by Splunk
- Yes No Yes N/A Ingested (Matches positive criteria)
- Yes No No N/A Ignored (Fails positive criteria)
- No Yes N/A Yes Ignored (Matched negative criteria)
- No Yes N/A No Ingested (Default accept all non-blacklisted)
- Yes Yes Yes Yes IGNORED (Blacklist supersedes Whitelist)
- Yes Yes Yes No Ingested (Passed whitelist, cleared blacklist)
- Common Regex Traps and Escaping
Unescaped Dots: In regular expressions, a bare dot ( . ) matches any character. Writing whitelist = .log matches syslog , catalog , and dialog.txt . To match a literal file extension, the dot must be escaped: whitelist = \.log$ .
Path Separator Escaping: On Windows systems where backslashes are present, backslashes must be escaped with double backslashes in regexes ( \\ ), or forward slashes should be used in the path definition to avoid backslash escaping issues.
CRC Checking Mechanics & The Fishbucket
When Splunk discovers a file within a monitored directory, it must determine whether the file is brand new, a continuing stream from an existing file, or a duplicate of a file that has already been indexed. It achieves this using a Cyclic Redundancy Check (CRC) hash.
The 256-Byte Default Inspection Window
By default, when splunkd encounters a file, it reads the first 256 bytes and computes a beginning CRC. It then looks the CRC up in the fishbucket ( $SPLUNK_DB/fishbucket/splunk_private_db ):
If the CRC does not exist: the file is new. Splunk reads it from the beginning and records CRCs and seek addresses as it goes.
If the CRC exists and the data at the saved seekAddress still matches its seekCRC: the file was read before. Splunk seeks to the seekAddress and reads only data appended after it.
If the CRC exists but the data at the seekAddress does not match: the file was modified in place, or it is a different file with the same opening bytes. Splunk cannot track both streams under one key, which is the collision problem described next. +-------------------------------------------------------------------------+
- | Monitored Log File |
- | [ Byte 0 ........................................ Byte 255 ] [ Byte 256 ]|
- | <--------- Initial 256 Bytes Hashed for CRC Check ---------> Payload |
- +-------------------------------------------------------------------------+
- Identical Header Collisions & Resolution Strategies
While the 256-byte CRC mechanism works seamlessly for standard dynamic logs, it creates a severe vulnerability in enterprise applications that write standardized boilerplate headers: the Identical Header Collision .
Root Cause of the Collision Defect
Many commercial enterprise software packages, databases, and network appliances prepend a static metadata header to every log file they generate. Consider an enterprise middleware system that creates daily log files with the following initial header: ################################################################
- # Application: Enterprise Financial Transaction Gateway #
- # Version: 14.2.0-RELEASE (Build 9481) #
- # Copyright (C) 2026 Global Banking Technologies Inc. #
- # Format: Standard Common Log Format (UTF-8) #
- ################################################################
If this static banner exceeds 256 bytes, every daily log file created by this system—such as app-2026-09-22.log , app-2026-09-23.log , and app-2026-09-24.log —will have an identical first 256 bytes .
The Consequence
Splunk reads app-2026-09-22.log and stores its beginning CRC with a seekAddress and seekCRC.
The next day, app-2026-09-23.log is created. Its first 256 bytes are identical, so its beginning CRC matches the existing record.
Splunk's documentation explains that because the fishbucket is keyed to the beginning CRC, it cannot track progress independently for the two files . Splunk behaves as if it is dealing with one file that was modified.
Administrators see missing or duplicated data for the daily files, depending on the timing. The documented fix is further configuration: initCrcLength or crcSalt .
Resolution 1: crcSalt = <SOURCE>
The crcSalt directive forces Splunk to alter the data buffer fed into the CRC calculation function. Setting crcSalt = <SOURCE> directs Splunk to append the file's fully qualified source path to the 256-byte header before generating the hash: Buffer = [ First 256 Bytes of File ] + [ Full Filepath String ] \text{Buffer} = [\text{First 256 Bytes of File}] + [\text{Full Filepath String}] Buffer = [ First 256 Bytes of File ] + [ Full Filepath String ] [monitor:///var/log/financial_app/*.log]
- index = finance
- sourcetype = financial:txngateway
- crcSalt = <SOURCE>
Because /var/log/financial_app/app-2026-09-22.log and /var/log/financial_app/app-2026-09-23.log have different filepath strings, their resulting CRC hashes are completely distinct, instantly eliminating the collision.
Caution
The crcSalt = <SOURCE> Rotation Hazard: Never configure crcSalt = <SOURCE> on log files that undergo move-and-recreate rotation (e.g., app.log rolling to app.log.1 ).
When app.log is renamed to app.log.1 , its filepath changes. Because the filepath changed, its CRC signature changes! Splunk fails to recognize app.log.1 as the rotated predecessor of app.log , treats it as a brand-new file, and re-indexes the entire historical file from byte 0 , generating massive duplicate event storms.
Resolution 2: initCrcLength = <bytes>
When log files undergo rotation and share identical headers, crcSalt = <SOURCE> cannot be safely used. The correct architectural remedy is initCrcLength .
Instead of reading only 256 bytes, initCrcLength instructs Splunk to expand the CRC inspection window deeper into the file: [monitor:///var/log/financial_app/*.log]
- index = finance
- sourcetype = financial:txngateway
- # Expand the CRC window to 2048 bytes (past the static banner); allowed range 256-1048576
- initCrcLength = 2048
By expanding the inspection window past the static banner into the variable payload area—where dynamic timestamps, thread IDs, or transaction records begin—each file generates a distinct CRC naturally. Because crcSalt is not used, the file can be safely renamed during log rotation without triggering re-indexing.
Parameter Recommended Use Case Safe with Log Rotation? Impact on Ingestion Performance
Default (256B CRC) Standard logs with dynamic, timestamped headers. Yes (Seamless rotation tracking) Minimal memory and CPU overhead.
crcSalt = <SOURCE> Static, uniquely named historical files or daily date-stamped files that never roll. NO (Triggers massive duplicate indexing on rename) Negligible overhead; completely eliminates header collisions.
initCrcLength = <bytes> Rotating log files with large boilerplate or XML headers. Yes (Safe for rolling logs) Minor additional I/O to read larger initial byte buffer.
Aging Parameters: ignoreOlderThan
When onboarding a legacy server into Splunk, administrators frequently configure a monitor stanza targeting a directory containing years of historical log files (e.g., /var/log/ ). Ingesting gigabytes of stale historical logs can exhaust daily license volume and overwhelm indexing queues.
The ignoreOlderThan directive instructs Splunk to skip files whose last modification timestamp ( mtime ) is older than a specified duration threshold: [monitor:///var/log/.../*.log]
- index = linux_os
- sourcetype = syslog
- # Ignore any file not modified within the last 14 days
- ignoreOlderThan = 14d
- Supported Syntax and Operational Mechanics
Time Span Formats: Accepts standard time units: s (seconds), m (minutes), h (hours), d (days). Examples: 48h , 7d , 30d .
How it is evaluated: the monitor input compares each file's modification time with the current time. If the difference is greater than ignoreOlderThan , the file is put on an ignore list and is not read.
No automatic reactivation: per inputs.conf.spec , files put on the ignore list are not checked again until Splunk restarts or the file monitoring subsystem is reconfigured , even if they become newer later. Choose a value that files you want to read can never reach, allowing for downtime. The spec suggests something like 14d .
Tailing from EOF: followTail = 1
With followTail = 1 , monitoring starts at the end of a file, like tail -f . The input does not read any data that exists in the file when it is first encountered, only data that arrives afterward. The spec calls it an advanced setting, says not to leave it enabled in an ongoing fashion, and says not to use it for rolling log files or files whose names or paths vary. [monitor:///var/log/debug_stream.log]
- index = debug
- sourcetype = app:debug
- followTail = 1
- The Critical Production Danger of followTail
followTail = 1 is easy to misuse because it silently skips data. Initial Boot: File discovered -> Seek to EOF -> Tail new writes (Expected)
Forwarder Restart:
- splunkd stops -> Application writes 500 KB to file while down
- splunkd starts -> File re-evaluated as newly discovered if fishbucket lost
seekptr jumps to NEW EOF -> The 500 KB written during downtime is PERMANENTLY LOST!
Why followTail = 1 Must Never Be Left Permanently Active
Evaluation condition: followTail applies whenever a file is encountered for the first time, meaning it has no fishbucket record.
Failure on Fishbucket Rebuilds: If an administrator cleans the fishbucket ( splunk clean eventdata -index _thefishbucket ) or moves the forwarder to a new storage volume, all files become "newly discovered." With followTail = 1 permanently enabled, Splunk seeks to EOF on every log file on the host, permanently skipping all existing log history.
Failure on Forwarder Downtime: If a monitored file rotates while the forwarder is stopped, the newly rotated active file appears as a fresh file upon forwarder restart. followTail = 1 forces Splunk to seek to EOF, permanently dropping all log entries written between the rotation and the restart.
Important
Operational Protocol for followTail : If followTail = 1 is used to bootstrap a massive log file without indexing historical backlog, follow this three-step procedure:
Add followTail = 1 to the monitor stanza and start Splunk.
Wait long enough for the input to identify the files (new events from them start arriving).
Immediately remove followTail = 1 (or set followTail = 0 ) in inputs.conf and restart Splunk (or remove it through Splunk Web or the CLI, which reconfigures the input). Future startups will then safely rely on the established fishbucket offset.