Study guide
15.2 Character Set Encoding & Handling Binary Data
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
15.2 Character Set Encoding & Handling Binary Data
Key Takeaways Explain This Topic AI
CHARSET in props.conf tells Splunk the source encoding so it can convert to UTF-8; it works in [<sourcetype>] or [source::] stanzas but not [host::], and defaults to AUTO on Windows and UTF-8 elsewhere.
An invalid CHARSET name causes further input from that stanza to be discarded, while characters that are invalid in a valid encoding are escaped as hex, such as \xF3 .
CHARSET and NO_BINARY_CHECK apply at input time, where data is first read, so for monitored files they must be on the forwarder with the input, including a universal forwarder.
Splunk ignores files it classifies as binary by default; UTF-16 text without a proper encoding declaration can look binary because of its null bytes.
NO_BINARY_CHECK = true makes Splunk process binary files for one [<sourcetype>] or [source::] stanza; never set it in [default], where it removes protection for every input.
Last updated: September 2026
Character Set Encoding & Handling Binary Data
Quick Summary: Splunk stores and searches text as UTF-8. When a source writes another encoding, such as UTF-16LE from Windows tools or ISO-8859-1 from legacy systems, set CHARSET in props.conf so that Splunk converts it correctly. Splunk also skips files it classifies as binary unless NO_BINARY_CHECK = true . The key exam point: both settings apply at input time, where the data is first read . For monitored files, that is the forwarder with the input, including a universal forwarder.
- CHARSET in props.conf [<sourcetype>] # or [source::<spec>]; not [host::<spec>]
- CHARSET = <encoding>
Per props.conf.spec :
Splunk assumes input for the stanza is in the named encoding and converts it to UTF-8.
It can be set only in [<sourcetype>] or [source::<spec>] stanzas, not [host::<spec>] .
Valid names are those listed by iconv -l on most *nix systems.
An invalid encoding name logs a warning during initial configuration, and further input from that stanza is discarded .
Characters that are invalid in a valid encoding are escaped as hex, for example \xF3 .
CHARSET = AUTO makes Splunk try to detect the encoding automatically and convert it to UTF-8.
Defaults: AUTO on Windows machines and UTF-8 everywhere else.
Where it applies: at input time, when data is first read, such as on a forwarder that has configured inputs acquiring the data .
What Goes Wrong Without It
When non-UTF-8 bytes are read as UTF-8:
Accented and non-Latin characters appear as escaped hex or replacement characters.
Timestamps whose month or day names contain non-ASCII characters may not be recognized, so events get a fallback time.
Field extractions and searches for those values stop matching.
With UTF-16 in particular, every other byte of ordinary text is a null byte. The data looks like binary, and line-breaking patterns may not match as expected.
Common Encodings
Encoding Typical sources Characteristics
UTF-8 Modern Linux and UNIX logs, JSON, web servers Superset of ASCII; 1 to 4 bytes per character; Splunk's internal format
UTF-16LE Many Windows tools and exports, such as PowerShell transcripts or SQL Server exports 2 bytes per basic character; ASCII text alternates with 0x00 ; BOM FF FE
UTF-16BE Some Java, mainframe, and appliance exports Big-endian UTF-16; BOM FE FF
ISO-8859-1 (Latin-1) Legacy UNIX daemons, older Western European applications Single-byte; 0x80 – 0xFF hold accented letters
- Windows-1252 Older Windows text files Like Latin-1, plus characters such as € and smart quotes in 0x80 – 0x9F
- SHIFT-JIS , EUC-JP , GB18030 Japanese and Chinese systems Multi-byte national encodings
A byte order mark (BOM) at the start of a file ( EF BB BF for UTF-8, FF FE for UTF-16LE, FE FF for UTF-16BE) helps detection. However, network streams and many rotated files do not start with one. For known sources, name the encoding explicitly instead of relying on detection.
Where to Put CHARSET : Input Time
Topology Where CHARSET must be
Universal forwarder monitoring a file, sending to indexers On the universal forwarder , in an app next to the input (for example, the same add-on that holds inputs.conf )
- Heavy forwarder with the input On the heavy forwarder
- Indexer or standalone instance reading the data itself On that instance
- Network input (TCP/UDP) or HEC on an indexer or heavy forwarder On the instance that receives the data
This differs from most parsing settings, such as LINE_BREAKER , TIME_FORMAT , and TRANSFORMS . Those run on the first full Splunk instance (a heavy forwarder or an indexer), not on a universal forwarder. CHARSET and NO_BINARY_CHECK belong to the small set of input-time props.conf settings. Many add-ons therefore install the same props.conf on forwarders and indexers.
Binary File Detection
Monitored directories often contain files that are not text, such as executables, core dumps, database files, and compressed files that are not handled as archives. Indexing them would waste license and disk space on meaningless terms. Splunk therefore examines each new file and ignores files it classifies as binary . props.conf.spec describes the default as NO_BINARY_CHECK = false (binary files are ignored) .
A rejected file leaves a warning from the file classifier in splunkd.log , typically stating that the file is invalid with the reason binary . To find such messages: index=_internal sourcetype=splunkd component=FileClassifierManager
The UTF-16 False Positive
In UTF-16LE, the text ABC is stored as 41 00 42 00 43 00 . A file full of null bytes looks binary, so a UTF-16 log without a correct encoding declaration can be skipped entirely. The fix is to declare the encoding, and, if the file is still classified as binary, to allow it: # props.conf on the universal forwarder that monitors the transcripts
- [powershell:transcript]
- CHARSET = UTF-16LE
- NO_BINARY_CHECK = true
For UTF-16 data, the spec also advises detect_trailing_nulls = false , so that null bytes that are part of the text are not trimmed.
NO_BINARY_CHECK
NO_BINARY_CHECK = true makes Splunk process binary files for that stanza.
It can be set only in [<sourcetype>] or [source::<source>] stanzas, not [host::<host>] .
Default: false .
Applies at input time , on the forwarder or instance that reads the data.
Splunk's own props.conf uses it for data that only looks binary. One example is [source::....Z(.\d+)?] , which decompresses .Z files with unarchive_cmd and sets NO_BINARY_CHECK = true .
Legitimate Uses
UTF-16 or other multi-byte encodings that are otherwise classified as binary.
Text records with a small binary header or null padding.
Why Not Set It Broadly
Setting NO_BINARY_CHECK = true in [default] or on a broad [source::...] pattern removes the protection everywhere:
- Risk Consequence
- License use Executables, dumps, and other binaries are indexed and count against the license
- Index size Random byte sequences produce huge numbers of useless terms in the .tsidx files
- Parsing Binary data rarely contains sensible line breaks, producing enormous or truncated events
- Search quality Garbage events clutter results and slow broad searches
Best practice: set NO_BINARY_CHECK = true only in a specific [<sourcetype>] or narrow [source::...] stanza, together with the correct CHARSET , and deploy it where the input runs.