Study guide
17.2 Data Masking, Redaction & SEDCMD
Splunk Enterprise Certified Admin SPLK-1003
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Splunk Enterprise Certified Admin Exam
2 Chapter 1: Splunk Admin Basics
1.1 Splunk Enterprise Architecture & Core Components
3 Chapter 2: License Management
2.1 License Types, Stacks, Pools & Licensing Architecture 2.2 License Violations, Rolling Window Warnings & Troubleshooting
4 Chapter 3: Splunk Configuration Files
3.1 Splunk Configuration Directory Structure 3.2 Configuration Layering, Precedence & Resolution Rules 3.3 Using btool to Inspect & Debug Configurations
5 Chapter 4: Splunk Indexes
4.1 Index Structure: Directories, Buckets & Bucket Files 4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle 4.3 Checking Index Data Integrity 4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes 4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files 4.6 Applying a Data Retention Policy
6 Chapter 5: Splunk User Management
5.1 Splunk Roles, Capabilities & Custom Role Design 5.2 User Account Administration, Search Restrictions & Limits
7 Chapter 6: Splunk Authentication Management
6.1 Integrating Splunk with LDAP 6.2 Other Authentication Options & Enabling Multifactor Authentication
8 Chapter 7: Getting Data In
7.1 CLI Administration & Basic Data Onboarding Workflows 7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder
9 Chapter 8: Distributed Search
8.1 How Distributed Search Works 8.2 Roles of the Search Head and Search Peers 8.3 Configuring Search Peers & Distributed Search Groups 8.4 Search Head Scaling & Search Head Clustering (SHC)
10 Chapter 9: Getting Data In – Staging
9.1 Ingestion Pipeline Stages: Input, Parsing, Merging, Typing & Indexing 9.2 Pipeline Component Placement & Queue Management 9.3 Splunk Input Options
11 Chapter 10: Configuring Forwarders
10.1 Forwarder Setup & outputs.conf Load Balancing 10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing
12 Chapter 11: Forwarder Management
11.1 Deployment Management & the Deployment Server 11.2 Deployment Apps & Deployment Clients 11.3 Server Classes, Client Grouping & App Deployment Strategies 11.4 Forwarder Management Monitoring & Troubleshooting
13 Chapter 12: Monitor Inputs
12.1 File & Directory Monitoring Mechanics 12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings 12.3 Deploying a Remote Monitor Input, Network Shares & Archives
14 Chapter 13: Network and Scripted Inputs
13.1 Creating Network Inputs: TCP & UDP 13.2 Optional Settings for Network Inputs 13.3 Scripted Inputs & PowerShell Data Collection
15 Chapter 14: Agentless Inputs
14.1 Windows Management Instrumentation (WMI) Inputs 14.2 HTTP Event Collector (HEC) Architecture & Administration
16 Chapter 15: Fine Tuning Inputs
15.1 Input-Phase Processing & Default Metadata Assignment 15.2 Character Set Encoding & Handling Binary Data
17 Chapter 16: Parsing Phase and Data
16.1 Line Breaking & Event Segmentation 16.2 Timestamp Extraction, Time Zones & Clock Skew Handling 16.3 Validating Event Parsing with Data Preview
18 Chapter 17: Manipulating Raw Data
17.1 Transformation Mechanics: props.conf & transforms.conf 17.2 Data Masking, Redaction & SEDCMD 17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue
17.2 Data Masking, Redaction & SEDCMD
Key Takeaways Explain This Topic AI
SEDCMD-<class> = s/<regex>/<replacement>/<flags> in props.conf rewrites _raw at index time; the y/string1/string2/ form substitutes characters one for one.
SEDCMD flags are g to replace every match or a number to replace only that match; without a flag, only the first match is replaced, and back-references use \n with a single digit.
SEDCMD runs in the typing pipeline (regex replacement) on heavy forwarders or indexers, so masked values are never written to the rawdata journal or .tsidx files; it has no effect on a universal forwarder by default.
Masking at index time keeps sensitive values out of Splunk's stored data, unlike search-time masking, but data indexed before the rule existed stays unmasked.
A transforms.conf rewrite with DEST_KEY = _raw works too but needs two files and replaces one match per transform (REPEAT_MATCH is ignored for _raw); regexes with nested quantifiers can backtrack catastrophically and block the pipelines.
Last updated: September 2026
Data Masking, Redaction & SEDCMD
Quick Summary: SEDCMD in props.conf masks data at index time using sed syntax ( s/<regex>/<replacement>/g ). It rewrites the event's _raw text in the typing pipeline on a heavy forwarder or indexer, before the event is written. Sensitive values such as card numbers, Social Security numbers, and tokens therefore never reach the rawdata journal or the .tsidx files. This helps organizations meet storage rules such as PCI DSS.
Regulatory Imperatives for Ingestion Redaction
Regulated organizations must limit where sensitive personal and financial data is stored. Once such data is indexed in plain text, removing it is painful: it means deleting events (the delete command only hides them from search), cleaning indexes, or letting buckets age out.
Core Regulatory Standards
PCI-DSS 4.0 (Payment Card Industry Data Security Standard):
Requirement 3: Mandates the protection of stored cardholder data. Primary Account Numbers (PANs) must be masked when displayed (showing at most the first six and last four digits). Furthermore, Sensitive Authentication Data (SAD)—including card verification codes (CVV2, CVC2, CID), full magnetic stripe / chip data, and PINs— must never be stored after authorization , even if encrypted.
HIPAA (Health Insurance Portability and Accountability Act):
Requires safeguards for electronic protected health information (ePHI). Keeping identifiers such as Social Security numbers and medical record numbers out of operational logs reduces how much of the logging platform must be treated as holding ePHI.
GDPR (General Data Protection Regulation):
Article 5 (Data Minimization): Personal data must be adequate, relevant, and limited to what is necessary in relation to the purposes for which they are processed.
Article 25 (Data Protection by Design and by Default): names pseudonymisation as an example of an appropriate technical measure. Masking at collection is one way to apply it.
Ingestion Redaction vs. Search-Time Masking
Evaluation Criterion Search-Time Masking (eval, sed, field masks) Ingestion Redaction via SEDCMD
Mechanism Applied when a search runs (for example rex mode=sed or eval replace() ) Applied in the typing pipeline on the heavy forwarder or indexer, before writing
What is stored Plain text. The sensitive value is in the rawdata journal and .tsidx files Masked. Only the replacement text is written
Exposure Anyone who can search without the mask, export buckets, or read the index files can see the value The value cannot be recovered from Splunk's stored data (the original log file on the source host is unaffected)
Compliance Does not prevent storage of prohibited data such as sensitive authentication data Keeps the data out of Splunk storage, supporting PCI DSS, HIPAA, and GDPR obligations
Search Overhead Consumes search-time CPU cycles on every single query invocation. Zero search-time overhead; data is already stored in masked format.
SEDCMD Syntax and Operational Mechanics
SEDCMD is configured in props.conf under a source type, source, or host stanza, on the heavy forwarder or indexer that parses the data. The spec says it is only used at index time. It needs no transforms.conf stanza.
- Syntax Specification [<stanza_name>]
- SEDCMD-<class> = s/<regex>/<replacement>/[flags]
SEDCMD-<class> : The directive identifier. <class> is an arbitrary string that must be unique within the stanza (e.g., SEDCMD-mask_cc , SEDCMD-mask_ssn ).
s : the replace command. Splunk supports only a subset of sed : replace ( s ) and character substitution ( y/string1/string2/ ) , which replaces each character of string1 with the character at the same position in string2. A sed script can be a space-separated list of commands.
<regex> : A Perl-Compatible Regular Expression (PCRE) pattern matching the sensitive text.
<replacement> : the replacement string. It can contain literal text (e.g., XXXX ) and back-references written \n , where n is a single digit ( \1 , \2 ), to keep non-sensitive captured parts.
[flags] : either g , which replaces all matches in the event, or a number , which replaces only that match (for example 2 for the second). With no flag, only the first match is replaced.
Where SEDCMD Runs in the Pipeline
Parsing pipeline: the data is broken into events ( LINE_BREAKER ).
Merging pipeline: timestamps are extracted, and lines merged if configured. Timestamp extraction therefore sees the original text.
Typing pipeline: the regex-replacement processor applies SEDCMD (and TRANSFORMS ) to _raw .
Index pipeline: the masked _raw is written to the rawdata journal ( journal.zst on current versions), and the .tsidx files are built only from the masked text.
Because masking happens after timestamp extraction, a SEDCMD rule cannot break timestamp recognition. However, it does affect every later index-time step, such as TRANSFORMS classes that match on the masked text.
- Practical SEDCMD Masking Recipes
- 1. Masking Credit Card Numbers (Preserving Last 4 Digits)
To mask a 16-digit credit card number while retaining the final four digits for account identification: # props.conf
- [payment:gateway:api]
- SEDCMD-mask_card = s/(\b\d{4}[- ]?)\d{4}[- ]?\d{4}[- ]?(\d{4}\b)/\1XXXX-XXXX-\2/g
- Input: User tx card=4111-2222-3333-4444 approved
- Output Written to Disk: User tx card=4111-XXXX-XXXX-4444 approved
Analysis: Group 1 ( \1 ) captures the first 4 digits, while Group 2 ( \2 ) captures the final 4 digits. The middle 8 digits are replaced with XXXX-XXXX- .
- 2. Redacting Social Security Numbers (SSNs)
- To scrub standard 9-digit U.S. Social Security Numbers formatted as ###-##-#### : # props.conf
- [hr:employee:onboarding]
- SEDCMD-redact_ssn = s/\b\d{3}-\d{2}-\d{4}\b/XXX-XX-XXXX/g
- Input: Employee record SSN=123-45-6789 onboarded successfully
- Output Written to Disk: Employee record SSN=XXX-XX-XXXX onboarded successfully
- 3. Redacting Bearer Tokens and Authorization Headers
- To mask JSON Web Tokens (JWT) or API bearer tokens in HTTP authorization headers: # props.conf
- [nginx:ingress:access]
- SEDCMD-mask_bearer = s/(?i)(Authorization:\s*Bearer\s+)[A-Za-z0-9_\-\.]+/\1[REDACTED_TOKEN]/g
Input: GET /v1/account Authorization: Bearer eyJhbGciOi...
- Output Written to Disk: GET /v1/account Authorization: Bearer [REDACTED_TOKEN]
- 4. Redacting Password and Secret Fields in Key-Value Pairs
- To mask passwords regardless of capitalization or quoting: # props.conf
- [app:auth:trace]
- SEDCMD-mask_pw = s/(?i)(password|passwd|secret)\s*[:=]\s*["']?[^"'\s,;]+["']?/\1=****************/g
- Input: login failed user=admin password="SuperSecretP@ss!" host=db01
- Output Written to Disk: login failed user=admin password=**************** host=db01
- SEDCMD in props.conf vs. Regex Replacement in transforms.conf
Administrators can also rewrite _raw using transforms.conf by setting DEST_KEY = _raw . Understanding the operational differences between these two approaches is essential: # Alternative approach using transforms.conf:
- # props.conf
- [app:auth:trace]
- TRANSFORMS-mask_pw = redact_pw_transform
- # transforms.conf
- [redact_pw_transform]
- REGEX = ^(.*password=)"[^"]+"(.*)$
- FORMAT = $1"********"$2
- DEST_KEY = _raw
- Architectural Comparison
Technical Attribute SEDCMD (props.conf) transforms.conf (DEST_KEY = _raw)
Configuration Overhead Single file ( props.conf ). No extra stanza mapping. Two files ( props.conf + transforms.conf ). Requires cross-referencing stanzas.
Pipeline Location Typing pipeline (regex replacement) Typing pipeline (regex replacement)
Processing Speed Splunk's documentation describes it as slightly faster Slightly slower; the whole event is rebuilt from capture groups
Global Multi-Matches g flag replaces every match in the event One match per transform; REPEAT_MATCH is ignored when DEST_KEY = _raw
Typical Use Simple masking of patterns anywhere in the event Rewrites that need SOURCE_KEY , lookahead control, or ordering with other transforms
Regular Expression Performance & Catastrophic Backtracking Hazards
Because SEDCMD executes at index time on every incoming event in real time, regex efficiency is paramount. A poorly constructed regular expression can cripple indexer ingestion throughput.
The Mechanism of Catastrophic Backtracking
Splunk uses a Non-Deterministic Finite Automaton (NFA) regex engine. When a regex contains ambiguous, overlapping, or nested quantifiers, the engine must explore every combination of branches before determining that a pattern fails.
Consider an anti-pattern regex intended to find numbers enclosed by words: (x+x+)+y
If an event contains xxxxxxxxxxxxxxxxxxxxxxxxxxxx (without the trailing y ), the engine tests exponential combinations ( 2 n 2^n 2 n ). For a string of just 30 characters, this can require over one billion evaluation steps .
Real-World Catastrophic Regex Anti-Patterns
Nested Quantifiers:
Bad: s/(\b\d+[- ]?)+/XXX/g
Why it fails: The outer quantifier ()+ and the inner quantifier \d+ compete for the same characters, triggering massive backtracking on invalid numbers.
Optimized: s/\b(?:\d{4}[- ]?){4}\b/XXXX-XXXX-XXXX-XXXX/g
Unanchored Wildcards with Optional Matches:
Bad: s/.*credit_card=(.*)/cc=XXXX/g
Why it fails: The leading .* scans to the end of the event, then steps backward character-by-character to satisfy the rest of the expression.
- Optimized: s/credit_card=[0-9]{13,19}/credit_card=XXXX/g
- Operational Impact on Splunk Ingestion
When an indexer encounters catastrophic backtracking:
The pipeline thread running the regex keeps a CPU core busy on each problem event.
typingQueue fills, then the queues before it ( aggQueue , parsingQueue , and the input queues) fill in turn.
The instance stops accepting data quickly, so forwarders sending to it slow down or switch to other indexers, and ingestion lag grows.