ExamFinch.com My Study

Study guide

4.1 Index Structure: Directories, Buckets & Bucket Files

Practice this exam

Splunk Enterprise Certified Admin SPLK-1003

Track your progress as you read

Finished sections get a check mark.

1 Introduction

About the Splunk Enterprise Certified Admin Exam

2 Chapter 1: Splunk Admin Basics

1.1 Splunk Enterprise Architecture & Core Components

3 Chapter 2: License Management

2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting

4 Chapter 3: Splunk Configuration Files

3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations

5 Chapter 4: Splunk Indexes

4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy

6 Chapter 5: Splunk User Management

5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits

7 Chapter 6: Splunk Authentication Management

6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication

8 Chapter 7: Getting Data In

7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder

9 Chapter 8: Distributed Search

8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)

10 Chapter 9: Getting Data In – Staging

9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options

11 Chapter 10: Configuring Forwarders

10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing

12 Chapter 11: Forwarder Management

11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting

13 Chapter 12: Monitor Inputs

12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives

14 Chapter 13: Network and Scripted Inputs

13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection

15 Chapter 14: Agentless Inputs

14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration

16 Chapter 15: Fine Tuning Inputs

15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data

17 Chapter 16: Parsing Phase and Data

16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview

18 Chapter 17: Manipulating Raw Data

17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue

4.1 Index Structure: Directories, Buckets & Bucket Files

Key Takeaways Explain This Topic AI

A Splunk index is a set of bucket directories under $SPLUNK_DB/<index>: hot and warm buckets in db (homePath), cold buckets in colddb (coldPath), and restored buckets in thaweddb (thawedPath).

Warm, cold, and thawed buckets are named db_<newest_time> <oldest_time> <localid>; hot buckets are named hot_v1_<localid>, and replicated cluster copies use the rb_ prefix.

Each bucket holds a compressed rawdata journal (zstd by default) plus tsidx index files, a bloom filter, and Hosts/Sources/SourceTypes.data metadata files.

Splunk's sizing rule of thumb: rawdata is about 15% of raw volume and index files about 35%, so a searchable bucket copy is roughly half the raw size.

Only rawdata is needed to recreate a bucket; splunk rebuild regenerates the tsidx and metadata files from the journal.

Last updated: September 2026

Index Structure: Directories, Buckets, and Files

In Splunk Enterprise, an index is not a monolithic relational database table or flat log repository. Instead, an index is a logical collection of discrete filesystem directories called buckets . Each bucket encapsulates a self-contained slice of time, containing both the compressed raw event text and the inverted search indexes that allow Splunk to query billions of events in seconds.

Understanding bucket architecture and lifecycle progression is essential for designing resilient storage tiers, managing disk capacity, meeting compliance retention mandates, and maintaining peak search performance.

Anatomy of an Index: Bucket Tiers & Storage Paths

When splunkd indexes incoming event streams, it organizes buckets into five sequential lifecycle tiers based on the age and write state of the data:

Hot Tier: Actively being written to by the indexing pipeline. Read/write.

Warm Tier: Rolled from hot; no longer receives new data. Read-only.

Cold Tier: Moved from warm to lower-cost, high-capacity storage. Read-only.

Frozen Tier: Aged out from cold storage; removed from search indexes. Either archived or permanently deleted.

Thawed Tier: Reconstructed from archived frozen data back into an active, searchable state. $SPLUNK_DB/<index_name>/

  • ├── db/ <-- Specified by homePath (Hot & Warm Buckets)
  • │ ├── hot_v1_0/ <-- Active Hot bucket (read/write)
  • │ ├── hot_v1_1/ <-- Active Hot bucket (read/write)
  • │ ├── db_1695427200_1695340800_1/ <-- Warm bucket (read-only)
  • │ └── db_1695340799_1695254400_0/ <-- Warm bucket (read-only)
  • ├── colddb/ <-- Specified by coldPath (Cold Buckets)
  • │ ├── db_1695254399_1695168000_2/ <-- Cold bucket (read-only)
  • │ └── db_1695167999_1695081600_3/ <-- Cold bucket (read-only)
  • └── thaweddb/ <-- Specified by thawedPath (Restored Buckets)
  • └── db_1694000000_1693000000_99/ <-- Thawed bucket (read-only, manual cleanup)
  • Bucket Tier Characteristics
  • Lifecycle Tier Filesystem Path Setting State Read / Write Searchable Typical Storage Media
  • Hot homePath Active Read / Write Yes Fast NVMe / PCIe SSD
  • Warm homePath Static Read-Only Yes Fast NVMe / Enterprise SSD
  • Cold coldPath Static Read-Only Yes High-Density HDD / Shared NAS
  • Frozen coldToFrozenDir / Script Archived None No Tape / Object Storage (S3 / Blob)
  • Thawed thawedPath Restored Read-Only Yes Standard Disk / Temporary Scratch

Important

Both hot and warm buckets reside in the exact same directory specified by the homePath parameter (by default, $SPLUNK_DB/<index_name>/db ). Cold buckets reside in a separate directory path defined by coldPath (by default, $SPLUNK_DB/<index_name>/colddb ), allowing administrators to place older, less frequently accessed data on less expensive, high-capacity storage tiers.

Deep Dive into Bucket Files

A bucket directory is a self-contained database unit. A warm or cold bucket on a standalone indexer contains files like these (exact names vary by version, and Splunk warns that bucket internals can change): db_1695427200_1695340800_1/

  • ├── rawdata/
  • │ ├── journal.zst <-- Compressed raw events (zstd by default; gzip/lz4 are alternatives)
  • │ └── l1Hashes / l2Hash <-- Present only when data integrity control is enabled
  • ├── *.tsidx <-- Time-series index files (lexicon + postings)
  • ├── bloomfilter <-- Lets searches skip buckets that cannot contain a term
  • ├── Hosts.data <-- Hosts seen in the bucket, with counts and time ranges
  • ├── Sources.data <-- Sources seen in the bucket
  • ├── SourceTypes.data <-- Source types seen in the bucket
  • └── bucket_info.csv <-- Bucket-level metadata
  • 1. Rawdata Journal ( rawdata/journal.zst )

The rawdata directory holds the compressed journal of every event indexed into the bucket, together with each event's index-time metadata ( _time , host , source , sourcetype ).

journalCompression in indexes.conf defaults to zstd . It affects only new buckets, and Splunk can search buckets compressed with different algorithms side by side.

The journal is the "source of truth". Everything else in the bucket can be regenerated from it with splunk rebuild , which is why archiving keeps only rawdata/ .

2. Time-Series Index Files ( .tsidx )

The .tsidx files are Splunk's inverted index. They contain the lexicon (every indexed term) and postings that point from each term to the events that contain it. A search first uses the tsidx files to find candidate events, so it avoids decompressing the whole journal.

A hot bucket that is actively receiving data creates many small tsidx files.

The splunk-optimize process merges those small files into fewer, larger ones as the bucket grows and when it rolls.

3. Metadata Files ( *.data )

Hosts.data , Sources.data , and SourceTypes.data record which hosts, sources, and source types appear in the bucket, with counts and first/last times. The metadata search command reads them, which is why | metadata type=hosts index=web answers quickly without scanning events.

4. Bloom Filters and Other Files

bloomfilter : A compact structure that lets a search rule a bucket out quickly. If the bloom filter says a term is not present, the bucket is skipped.

bucket_info.csv : Bucket-level metadata used by the indexer.

Integrity hash files: When enableDataIntegrityControl = true , Splunk writes l1Hashes and l2Hash files in the bucket's rawdata directory (see the data integrity section).

How Much Disk Does a Bucket Use?

Splunk's long-standing sizing rule of thumb is that compressed rawdata is about 15% of the original raw volume and index files are about 35% , so a searchable bucket copy is roughly 50% of the raw data it holds. The index files are the larger share, which is why archiving only rawdata/ saves so much space.