Study guide
2.2 License Violations, Rolling Window Warnings & Troubleshooting
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
2.2 License Violations, Rolling Window Warnings & Troubleshooting
Key Takeaways Explain This Topic AI
License usage is the raw volume of data entering the indexing pipeline, measured midnight to midnight on the license manager's clock; data dropped to nullQueue before indexing does not count.
Internal indexes such as _internal and _introspection, summary indexing, and metric rollup summaries do not count; metric events are counted at up to 150 bytes each.
An Enterprise stack under 100 GB/day is in violation after 45 warnings in a rolling 60 days; 100 GB/day or larger stacks never lose search; Trial, Dev/Test, and Developer violate at 5 warnings in 30 days and Free at 3.
During a violation, searching (including scheduled reports and alerts) is blocked but indexing continues, and the internal indexes remain searchable for diagnosis.
license_usage.log in _internal records type=Usage events (b, s, st, h, idx, pool) and daily type=RolloverSummary totals for troubleshooting spikes.
Last updated: September 2026
License Violations, Rolling Window Warnings & Troubleshooting
In Splunk Enterprise, managing license compliance requires knowing exactly what data consumes daily quota, how overage warnings are calculated, the operational consequences of reaching violation thresholds, and how to rapidly isolate and remediate unexpected ingestion spikes.
Daily License Calculation: What Counts vs. What Is Exempt
Splunk Enterprise calculates license consumption based on the volume of data entering the indexing pipeline during each 24-hour cycle. Understanding the boundary between billable event data and exempt overhead is fundamental to capacity management.
The Calculation Standard: Raw Uncompressed Volume
Splunk measures daily license usage by calculating the uncompressed raw byte volume of incoming data at the moment it is indexed.
Point of Ingestion: Metering happens when data is placed into the indexing pipeline on the instance that indexes it, so events filtered and dropped before indexing never count.
Raw Event Sizing: Splunk calculates the byte size of the raw incoming character stream (the _raw field), including any delimiter or line-breaking characters added during the event-breaking phase.
Index vs. Disk Footprint: License measurement is completely independent of index storage size on disk. Whether indexing overhead increases disk usage by 200% (due to index keys, bloom filters, and dictionary structures) or compression shrinks the raw text to 50% on disk, Splunk charges strictly for the raw, uncompressed event payload.
Re-indexing Consideration: If an administrator re-indexes an existing data set—such as by resetting the fishbucket on a forwarder or re-reading an archived log file—all re-ingested bytes are counted against today's license quota as new data.
Data Accounting Breakdown
Data Category License Impact Technical Justification
Standard Application & System Logs Counts All raw data written to standard, non-internal indexes ( main , web , security , network ).
Metrics Data Counts Each metric event is measured like event data but capped at 150 bytes; events under 150 bytes count as their size plus 18 bytes, up to 150.
Re-indexed Data Counts Any data re-read and passed through the indexing pipeline is metered as new ingest.
Internal Indexes ( _internal , _audit , _introspection , _telemetry ) Exempt Splunk's self-generated diagnostic, administrative, access, and performance monitoring data is free of license charges.
Fishbucket Index ( _thefishbucket ) Exempt Internal tracking pointers and CRC checkpoint records used by inputs to track file offsets do not consume quota.
Summary Indexing & Metric Rollups Exempt Splunk's documentation states that summary indexing and metric rollup summaries do not count against license volume.
Structural Index Metadata ( tsidx , bloom filters, lexicon) Exempt Inverted index keys, postings files, time boundaries, and search structures are purely storage overhead.
Restored (Thawed) Data Exempt Buckets restored from a frozen archive into thaweddb do not count against the license.
Indexer Cluster Replication Traffic Exempt In an indexer cluster, replicated bucket copies sent across peer nodes to satisfy the Replication Factor (RF) are not double-counted. Only the primary raw ingest is metered once.
Forwarded Data (UF / HF transit) Exempt Data parsed and forwarded across the network without being written to a local index on that forwarder is not metered on the forwarder.
- The Daily Reset & Rolling Warning Windows
- The Midnight Reset
Daily license volume is measured from midnight to midnight using the system clock on the license manager .
An indexer in Tokyo or London reports to that manager every minute, but the license day it counts against is the manager's day. Keep every Splunk host on synchronized time (NTP).
If Splunk Web shows a license warning message before midnight, you are already over quota for the day. The warning is recorded when the quota resets at midnight, which gives you time to add license volume before the warning counts.
What Generates a Warning
Splunk's documentation lists these warning conditions:
A license pool reaches its daily license volume limit.
A license stack reaches its daily license volume limit.
A license peer cannot communicate with the license manager.
- Violation Thresholds & Operational Enforcement
- Violation Thresholds by License Type
License Violation rule What happens
Enterprise stack ≥ 100 GB/day Warnings are issued but search is not disabled Keep indexing and searching; buy or rebalance volume
Enterprise stack < 100 GB/day 45 warnings in a rolling 60-day period Search disabled for the affected pool and its members; reset license from Splunk Sales
Enterprise infrastructure (vCPU) Does not currently violate Not applicable
Enterprise Trial 5 or more warnings in a rolling 30 days Search blocked; warnings persist 14 days; no reset license
Dev/Test and Developer 5 or more warnings in a rolling 30 days Search blocked; warnings persist 14 days; reset license available
Free 3 or more warnings in a rolling 30 days Search blocked; warnings persist 14 days; no reset license
Exam trap: Older courseware and many practice sites still quote "5 warnings in 30 days" for every Enterprise license. Splunk's current administration manual gives 45 warnings in 60 days for Enterprise stacks under 100 GB/day and no search blocking for stacks of 100 GB/day or more. The 5-in-30 rule now applies to Trial, Dev/Test, and Developer licenses.
Operational Impact of a License Violation
THE GOLDEN RULE OF SPLUNK LICENSE VIOLATIONS:
Search is blocked; indexing and data ingestion continue.
Search Lockout: For affected stacks and pools, search is blocked, including scheduled reports and alerts . With a license manager, a message appears in the Search app and in the Global Messages bar telling users the license is in violation.
Indexing Never Stops: Indexers keep receiving and indexing data, so a violation causes no data loss. You simply cannot search that data until the violation clears.
Internal Searches Remain Permitted: Searching the internal indexes is not blocked, so you can use the Monitoring Console or searches against _internal to find the cause.
Lost Contact With the License Manager
A license peer that cannot reach the license manager for 72 hours or more is placed in violation and search is blocked on that peer until it reconnects. Indexing continues. Search _internal for LMTracker errors such as "failed to send rows" or "unable to connect".
Investigating and Troubleshooting Ingestion Spikes
When an unexpected data surge threatens a license violation, administrators must quickly identify the offending inputs, sourcetypes, or hosts.
The Key Log: license_usage.log
All licensing telemetry is recorded in $SPLUNK_HOME/var/log/splunk/license_usage.log , indexed automatically into _internal .
Key fields in license_usage.log :
b : Number of raw bytes metered in the reporting event.
s : Source identifier.
st : Sourcetype of the indexed events.
h : Originating host.
idx : Target index.
pool : Assigned license pool.
i : GUID of the license peer that indexed the data.
type : Usage for the per-minute usage records and RolloverSummary for the daily total written at midnight.
- Diagnostic SPL Queries
- 1. Daily License Volume by Pool (Last 30 Days)
Use the midnight rollover summaries to see each pool's daily total: index=_internal source=*license_usage.log type=RolloverSummary
- | eval total_GB = round(b/1024/1024/1024, 2)
- | timechart span=1d sum(total_GB) by pool
- 2. Top License-Consuming Sourcetypes Today
Identify which data types are responsible for current consumption: index=_internal source=*license_usage.log type=Usage
- | stats sum(b) as total_bytes by st
- | eval total_GB = round(total_bytes/1024/1024/1024, 2)
- | sort - total_GB
- | head 10
- 3. Pinpointing High-Volume Hosts and Sources for a Specific Sourcetype
When an anomalous sourcetype (e.g., syslog or cisco:asa ) spikes, isolate the individual host and file path: index=_internal source=*license_usage.log type=Usage st="cisco:asa"
- | stats sum(b) as total_bytes by h, s
- | eval total_MB = round(total_bytes/1024/1024, 2)
- | sort - total_MB
- | head 20
- Common Causes of License Overages
Application Debug Logging: An application team accidentally enables DEBUG or TRACE level logging in production, generating millions of verbose log lines per hour.
Syslog and Network Echo Loops: A misconfigured syslog forwarder, load balancer, or network switch creates a packet reflection loop, repeatedly resending identical messages.
Recursive Forwarder Directory Monitors: A wildcard monitor in inputs.conf (e.g., [monitor:///var/log/...] ) traverses into an active backup or archive directory, indexing compressed .tar.gz or .zip files as binary text.
crcSalt = <SOURCE> on Rotating Logs: Adding crcSalt = <SOURCE> to a monitor input whose files are renamed by log rotation makes every renamed file look new, so Splunk indexes the rotated copies again. Splunk's documentation warns against using crcSalt = <SOURCE> with rotated logs.
Remediation Strategies
Reset License: For an Enterprise stack under 100 GB/day that has lost search, request a reset license from Splunk Sales. Dev/Test customers use the Dev/Test request form, and Developer license holders email Splunk developer support. Trial and Free licenses have no reset license.
License Pool Adjustments: If one pool is collecting warnings because of organizational growth, adjust pool allocations under Settings > Licensing to shift unused capacity from development pools to production pools.
Pre-Indexing Ingestion Filters: Configure routing and filtering rules on Heavy Forwarders or Indexers using props.conf and transforms.conf to discard repetitive or useless events ( DEST_KEY = queue, FORMAT = nullQueue ) before they enter the indexing pipeline and consume quota.
Stack Expansion: If genuine business data growth consistently exceeds capacity, install additional stacking Enterprise license keys to permanently expand the Enterprise stack.