Study guide

12.1 File & Directory Monitoring Mechanics

Splunk Enterprise Certified Admin SPLK-1003

Track your progress as you read

Finished sections get a check mark.

1 Introduction

About the Splunk Enterprise Certified Admin Exam

2 Chapter 1: Splunk Admin Basics

1.1 Splunk Enterprise Architecture & Core Components

3 Chapter 2: License Management

2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting

4 Chapter 3: Splunk Configuration Files

3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations

5 Chapter 4: Splunk Indexes

4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy

6 Chapter 5: Splunk User Management

5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits

7 Chapter 6: Splunk Authentication Management

6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication

8 Chapter 7: Getting Data In

7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder

9 Chapter 8: Distributed Search

8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)

10 Chapter 9: Getting Data In – Staging

9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options

11 Chapter 10: Configuring Forwarders

10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing

12 Chapter 11: Forwarder Management

11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting

13 Chapter 12: Monitor Inputs

12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives

14 Chapter 13: Network and Scripted Inputs

13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection

15 Chapter 14: Agentless Inputs

14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration

16 Chapter 15: Fine Tuning Inputs

15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data

17 Chapter 16: Parsing Phase and Data

16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview

18 Chapter 17: Manipulating Raw Data

17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue

12.1 File & Directory Monitoring Mechanics

Key Takeaways Explain This Topic AI

[monitor://<path>] inputs in inputs.conf continuously read files and directories (recursively by default) and are handled by the tailing processor.

In monitor paths, * matches within one path segment while ... matches any number of directory levels, including zero.

Read progress is stored in the fishbucket as a beginning CRC of the first 256 bytes, a seekAddress, and a seekCRC; Splunk identifies files by content, not by name or inode.

Rotated copies are recognized because they keep the same beginning CRC and have already been read; compressed rotated archives (.gz, .tar) are the exception and are processed.

followSymlink defaults to true; disabling or deleting a monitor input does not stop files already being read until Splunk restarts.

Last updated: September 2026

File & Directory Monitoring Mechanics

Quick Summary: Ingesting log data from local files and directory hierarchies is the most foundational task in Splunk Enterprise administration. Configured through [monitor://<path>] stanzas in inputs.conf , file monitoring is driven by the splunkd tailingProcessor pipeline. This engine continuously monitors active files, maintains byte-level read offsets in a specialized tracking database known as the fishbucket, identifies log rotations without duplicating data, and navigates complex directory structures using precise single-level and recursive wildcards.

In-Depth Syntax and Structure of [monitor://<path>]

In Splunk Enterprise and the Universal Forwarder, the primary configuration stanza for continuous data ingestion from the filesystem is [monitor://<path>] , defined within inputs.conf . Unlike batch or one-time inputs, a monitor input establishes a continuous watch over the target file or directory tree, ingesting existing contents and tailing newly appended data in near real time.

Stanza Path Conventions and Normalization

When defining monitor paths, administrators must adhere to strict platform syntax and path normalization rules: # Linux / Unix absolute file monitoring

  • [monitor:///var/log/messages]
  • disabled = false
  • index = os
  • sourcetype = syslog
  • # Linux / Unix directory tree monitoring
  • [monitor:///var/log/httpd]
  • disabled = false
  • index = web
  • sourcetype = access_combined
  • # Windows local filesystem monitoring
  • [monitor://C:\inetpub\logs\LogFiles]
  • whitelist = \.log$
  • disabled = false
  • index = iis
  • sourcetype = ms:iis:auto
  • Key Path Rules

Windows Paths: On Windows, the path starts with the drive letter, as in [monitor://C:\inetpub\logs\LogFiles] . Remember that whitelist and blacklist values are regular expressions, so a literal backslash inside them must be escaped as \\ .

Leading Slashes on POSIX Systems: On Linux and Unix platforms, an absolute path must begin with three forward slashes: two slashes indicating the protocol scheme ( monitor:// ) and the third denoting the filesystem root directory ( / ), yielding [monitor:///path/to/log] .

Directory vs. File Targets: If <path> references an individual file, Splunk opens and tails that specific file. If <path> references a directory, Splunk automatically monitors every readable file within that directory and, by default, recurses through all child directories unless constrained by wildcards or tuning settings.

Metadata Assignment: Within each monitor stanza, administrators assign critical indexing and parsing metadata:

index : Specifies the destination target index on the indexers (e.g., index = security ). If omitted, data routes to the default index (typically main ).

sourcetype : Assigns the format classification that governs event breaking, timestamp parsing, and field extraction (e.g., sourcetype = cisco:asa ). If omitted, Splunk assigns a source type automatically, which may not be the one you want.

source : Overrides the default source value, which is the file's full path. Overriding it is rarely needed for monitored files.

host : Sets the originating host value, overriding the server's local hostname when ingesting third-party or relayed telemetry.

disabled : Boolean toggle ( false enables the input; true disables it without deleting the configuration stanza).

Wildcard Syntax: Single-Segment ( * ) vs. Recursive Traversal ( ... )

File systems in enterprise environments frequently partition logs by application instance, date, tenant, or customer ID. Splunk provides two distinct wildcard mechanisms in monitor stanzas to traverse these directory structures: the single asterisk ( * ) and the recursive ellipsis ( ... ). Mastering the precise behavioral difference between these operators is crucial for avoiding unintended data ingestion and performance degradation. Wildcard Comparison:

  • /var/log/*/*.log --> Matches exactly ONE directory level deep
  • Matches: /var/log/apache/access.log
  • Does NOT match: /var/log/apache/ssl/access.log
  • /var/log/.../*.log --> Matches ANY number of nested subdirectories
  • Matches: /var/log/apache/access.log
  • Matches: /var/log/apache/ssl/access.log
  • Matches: /var/log/apache/ssl/external/access.log
  • The Single Asterisk ( * )

The single asterisk * represents a single-segment wildcard . It matches zero or more characters within a single directory segment or within a filename , but it never crosses directory separator boundaries ( / or \ ).

In a filename position: [monitor:///var/log/app_*.log] matches /var/log/app_web.log and /var/log/app_api.log , but does not match /var/log/app_sub/other.log .

In a directory position: [monitor:///var/log/*/catalina.out] matches /var/log/tomcat1/catalina.out and /var/log/tomcat2/catalina.out . However, it strictly refuses to match /var/log/cluster/node1/catalina.out because that path requires crossing two directory separators.

The Recursive Ellipsis ( ... )

The triple-dot ellipsis ... represents a recursive directory traversal wildcard . It matches zero, one, or multiple nested directory levels deep, recursing down through an arbitrary directory tree until it finds files matching the subsequent path pattern.

In an intermediate path position: [monitor:///opt/apps/.../*.log] discovers and monitors .log files in /opt/apps/ , /opt/apps/auth/ , /opt/apps/auth/svc/ , and arbitrarily deep directory hierarchies.

In a trailing position: [monitor:///var/log/audit/...] monitors every file and child directory beneath /var/log/audit/ , regardless of nesting depth.

Wildcard Matching Matrix

To understand how * and ... evaluate real-world filesystem paths, consider the following matrix:

Stanza Pattern in inputs.conf Target File Path Match Result Technical Rationale

[monitor:///var/log/*.log] /var/log/secure.log MATCH File resides directly in /var/log/ and matches pattern.

[monitor:///var/log/*.log] /var/log/audit/audit.log NO MATCH Single * cannot cross the directory boundary into audit/ .

[monitor:///var/log/*/*.log] /var/log/nginx/access.log MATCH Path contains exactly one directory level ( nginx ) between root and file.

[monitor:///var/log/*/*.log] /var/log/nginx/ssl/access.log NO MATCH Path contains two directory levels ( nginx/ssl ), exceeding the single * scope.

[monitor:///var/log/.../*.log] /var/log/nginx/ssl/access.log MATCH Recursive ... matches arbitrary nested subdirectory depths.

[monitor:///var/log/.../*.log] /var/log/firewall.log MATCH Recursive ... matches zero intermediate directory levels as well.

Warning

Using broad recursive monitor stanzas such as [monitor:///var/log/...] without whitelists or blacklists can inadvertently ingest binary system dumps, rotated compressed archives, or rapidly churning temporary files, exhausting indexing licenses and degrading search performance.

Tailing Processor Mechanics: Continuous File Tracking

monitor inputs are handled by the tailing processor ( TailingProcessor in splunkd.log ). It discovers files under the monitored paths, reads new data as it is appended, and records its progress in the fishbucket so that restarts neither lose nor repeat data.

How the Monitor Processor Works

When you name a file , Splunk consumes new data written to it. When you name a directory , it recursively examines the subdirectories ( recursive = true by default) for new files it can read.

A file or directory that does not exist when Splunk starts is checked again every 24 hours from the last restart. Subdirectories of monitored directories are scanned continuously.

Splunk uses memory for every file it tracks, including ignored files. On directories with huge numbers of old files, use a narrower path, a whitelist/blacklist, or ignoreOlderThan .

Monitor inputs may overlap. As long as the stanza names differ, each is independent, and a file is handled by the most specific matching stanza.

Disabling or deleting a monitor input does not stop files already being read. It only stops Splunk from checking those files again. Restart to stop in-process indexing.

Limits: files whose path is longer than 1,024 characters (256 on Windows) are not monitored, and files with a .splunk extension are skipped.

How Progress Is Stored: The Fishbucket

The fishbucket ( $SPLUNK_DB/fishbucket/splunk_private_db , visible as the _thefishbucket index) stores, for each file:

  • a beginning CRC of the first 256 bytes ( initCrcLength ), which is the lookup key,
  • the seekAddress (how many bytes have been read), and

a seekCRC of the data at that position.

After a restart, the processor recomputes each file's beginning CRC, finds the record, checks that the data at the seekAddress still matches the seekCRC, and continues from there.

Log Rotation Detection Mechanics

Log rotation (Linux logrotate , Log4j/Logback rolling) keeps files from growing without limit. Splunk recognizes rotated files by content (CRC), not by file name or inode .

  • Move-and-Recreate (Rename) Rotation Step 1: Active logging to /var/log/app.log
  • Step 2: logrotate renames it: mv /var/log/app.log /var/log/app.log.1
  • Step 3: a new, empty /var/log/app.log is created and the application writes to it

The renamed app.log.1 has the same first 256 bytes as the file Splunk was reading, so its beginning CRC matches an existing fishbucket record. Its content at the seekAddress matches the seekCRC, and it has no data beyond what was read, so Splunk does not read it again .

The new app.log starts with different content, so its beginning CRC is new. Splunk treats it as a new file and reads it from the start.

This works only if the rotated file stays within a monitored path, or if Splunk finished reading it before the rename. Data that was still unread when the file moved out of the monitored path is not collected.

Compressed rotated copies ( .gz , .tar ) are the documented exception: Splunk processes renamed archives. That is why many teams blacklist \.gz$ in directories where rotation compresses old files.

  • Copy-Truncate ( copytruncate ) Rotation Step 1: /var/log/app.log is copied to /var/log/app.log.1
  • Step 2: /var/log/app.log is truncated in place and the application keeps writing to it

The copy app.log.1 has the same beginning CRC as the data already read, so it is recognized and not read again.

The truncated app.log soon holds new opening content with a new beginning CRC, so Splunk reads it as a new file.

Anything written between the copy and the truncate can be lost, and a partially read file can be read twice. Prefer move-and-recreate for important audit logs.

Caution

Do not add crcSalt = <SOURCE> to monitors of rotating files. It puts the path into the CRC, so a renamed file looks brand new and is indexed again.

Symbolic Link Handling: Hazards, Loops & Traversal Risks

Symbolic links (symlinks) are pointers in a filesystem that reference another file or directory path. In inputs.conf , the followSymlink setting controls whether Splunk traverses symlinks encountered during directory monitoring: [monitor:///var/log/apps]

  • followSymlink = false
  • Default Behavior and Operational Hazards

followSymlink defaults to true in inputs.conf . While symlink traversal provides administrative convenience when logs are distributed across multiple storage volumes, it introduces severe operational hazards:

1. Symlinks That Point Back Into the Tree

If a monitored directory contains a symbolic link that points back to itself or to an ancestor directory (for example, /var/log/apps/archive pointing to /var/log/apps ), the same files are reachable through ever-longer paths: /var/log/apps/archive/archive/archive/archive/...

At best this wastes effort walking the same directories; at worst it makes file tracking confusing and hard to troubleshoot. Keep monitored trees free of such links, or set followSymlink = false .

2. Duplicate Data Ingestion

If an administrator configures a monitor stanza for /var/log/httpd/access.log and another monitor stanza for /var/www/logs/ , where /var/www/logs/access.log is a symlink pointing to /var/log/httpd/access.log , Splunk can ingest the same data through both stanzas if crcSalt = <SOURCE> makes the two paths look like different files. This doubles indexing volume, artificially inflates licensing usage, and corrupts search results with duplicate events.

3. Security and Directory Traversal Vulnerabilities

Following symlinks can also expose files outside the intended directory. If an unprivileged application user creates a symlink inside a monitored directory pointing to sensitive system files: ln -s /etc/shadow /var/log/myapp/debug.log

If the forwarder runs as root and followSymlink = true (the default), Splunk follows the link and can index sensitive password hashes. Per the spec, whitelists and blacklists also apply to files at the symlink destination, so a tight whitelist limits the damage.

Recommended Best Practices for Symlinks

Disable Symlink Traversal by Default: Set followSymlink = false on broad directory monitor stanzas unless symlinks are strictly necessary for the application architecture.

Explicit Target Monitoring: If specific symlinked targets must be ingested, create dedicated, tightly scoped [monitor://<explicit_target>] stanzas pointing directly to the real underlying path rather than enabling recursive symlink following on parent directories.

Loading diagram... How the Monitor Processor Decides What to Read