3.2 Ingest Pipelines, Processors & Normalization Workflows

Elastic Certified SIEM Analyst Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & Study Strategy

1.1 Elastic Certified SIEM Analyst Exam Facts, Objectives & Study Plan

2 Chapter 2: Elastic Stack Architecture & SIEM Foundations

2.1 Elastic Stack Architecture for Security Operations 2.2 Telemetry Ingestion Architecture & Data Pipelines 2.3 Security Data Storage, Data Streams & ILM

3 Chapter 3: Elastic Common Schema (ECS) & Ingestion Normalization

3.1 ECS Structure, Core Principles & Essential Field Sets 3.2 Ingest Pipelines, Processors & Normalization Workflows 3.3 Fleet Server & Elastic Agent Integration Management

4 Chapter 4: Discover, Data Views & Security Query Languages

4.1 Kibana Discover, Security Data Views & Runtime Fields 4.2 Kibana Query Language (KQL) & Lucene for SecOps 4.3 ES|QL for Multi-Stage Security Analysis

5 Chapter 5: Security Visualizations & Threat Analytics

5.1 Metric, Aggregation & Histogram Visualizations for SOC 5.2 Data Tables, Heatmaps & Specialized Security Charts 5.3 Elastic Maps & Geospatial Telemetry Analysis

6 Chapter 6: Kibana Lens for Security Workflows

6.1 Lens Architecture & Multi-Layer Telemetry Visualizations 6.2 Lens Formulas, Ratios & Anomaly Baseline Metrics 6.3 Operationalizing Lens in SOC Triage & Drilldowns

7 Chapter 7: Security Dashboards & SOC Operations

7.1 Security Dashboard Architecture & SOC Layout Design 7.2 Interactive Controls, Time Sync & Cross-Filtering 7.3 Dashboard Exporting, Reporting & Spaces Sharing

8 Chapter 8: The Elastic Security App: Capabilities, Explore & Dashboards

8.1 Security App Capabilities & the Explore Pages (Hosts, Network, Users) 8.2 Monitoring with the Security App Dashboards

9 Chapter 9: Elastic Security Detection Engine & Core Rules

9.1 Detection Engine Architecture & Prebuilt Rule Management 9.2 Custom Query & Threshold Detection Rules 9.3 Event Correlation Rules with Event Query Language (EQL)

10 Chapter 10: Advanced Detection Rules, ML & Rule Tuning

10.1 Threat Intelligence & Indicator Match Rules 10.2 New Terms, ES|QL & Machine Learning Detection Rules 10.3 Detection Rule Tuning, Exceptions & Alert Suppression

11 Chapter 11: Alert Triage, Timelines & Investigation Workflows

11.1 Alert Triage Workflow & the Alert Details Flyout 11.2 Elastic Security Timeline & Timeline Templates 11.3 Visual Event Analyzer & Session View Forensics

12 Chapter 12: Case Management, Integrations & AI Assistant

12.1 Elastic Security Case Management & Triage Workflows 12.2 Case Connectors & External Incident Escalation 12.3 Elastic AI Assistant for Security & Attack Discovery

13 Chapter 13: Security Administration, RBAC & Kibana Spaces

13.1 Kibana Spaces for SOC Multitenancy & Segmentation 13.2 Role-Based Access Control, Feature Privileges & Index Security 13.3 Elastic Defend Administration & Endpoint Policy Controls

14 Chapter 14: End-to-End Threat Hunting & Incident Investigation Capstone

14.1 Hypothesis-Driven Threat Hunting & Baselining in Elastic 14.2 Investigating Lateral Movement & Credential Access Campaigns 14.3 Data Exfiltration, C2 Detection & Incident Reconstruction

3.2 Ingest Pipelines, Processors & Normalization Workflows

Key Takeaways Explain This Topic AI

Ingest pipelines execute pre-indexing transformations directly on Elasticsearch ingest nodes, converting raw log strings into structured ECS documents.

The dissect processor provides high-throughput delimiter splitting for structured logs, whereas grok uses regular expression patterns to parse complex, variable log formats.

Conditional processor execution via Painless 'if' expressions ensures targeted execution and prevents runtime exceptions on missing fields.

Production SecOps pipelines require defensive design, utilizing 'ignore_missing: true' and 'on_failure' handlers to avoid dropping unparsed documents.

The _simulate API enables safe, in-memory validation of ingest pipelines against sample log payloads prior to production deployment.

Last updated: September 2026

Ingest Pipeline Architecture in the Elastic Stack

In modern SecOps architectures, log parsing and schema normalization occur directly within Elasticsearch using Ingest Pipelines . An ingest pipeline is a configured sequence of processors running on Elasticsearch ingest nodes that intercept, parse, enrich, and transform documents before they are written to a data stream or index.

An Ingest Pipeline definition contains three core elements:

description : Operational context explaining the pipeline's purpose.

processors : An ordered array of individual transformation steps executed sequentially on each incoming document.

on_failure : A fallback array of processors executed if an unhandled exception occurs in the primary processor sequence. PUT _ingest/pipeline/logs-cisco_asa-normalization

• {
• "description": "Parses and normalizes Cisco ASA firewall syslog to ECS",
• "processors": [
• {
• "grok": {
• "field": "message",

"patterns": ["%ASA-%{INT:log.syslog.severity.code}-%{INT:cisco.asa.message_id}: %{GREEDYDATA:cisco.asa.message}"]

• }
• },
• {
• "set": {
• "field": "event.category",
• "value": "network"
• }
• }
• ],
• "on_failure": [
• {
• "append": {
• "field": "tags",
• "value": ["_cisco_asa_parse_failure"]
• }
• }
• ]
• }
• Conditional Execution with Painless ( if Clauses)

Processors can execute conditionally using the if parameter, which evaluates an inline Painless script. The document context is accessible via the ctx map:

• Field existence checks : "if": "ctx.host?.name != null"
• Dataset matching : "if": "ctx.event?.dataset == 'cisco.asa' && ctx.cisco?.asa?.message_id == '106023'"
• Regex pattern evaluation : "if": "ctx.message =~ /^(?i)failed password/"

Using if conditions prevents unnecessary processing overhead, eliminates runtime exceptions caused by missing keys, and enables modular, multi-source pipeline routing.

Pipeline Validation via the _simulate API

Before attaching a pipeline to a production data stream, engineers validate its behavior using the _simulate API. This runs the pipeline in memory against test payloads without writing documents to disk: POST _ingest/pipeline/_simulate

• {
• "pipeline": {
• "processors": [
• {
• "dissect": {
• "field": "message",
• "pattern": "%{syslog_timestamp} %{host.name} %{process.name}[%{process.pid}]: %{log.message}"
• }
• }
• ]
• },
• "docs": [
• {
• "_source": {

"message": "Sep 30 10:14:22 srv-dc01 sshd[4192]: Failed password for invalid user admin from 192.168.10.50 port 54822"

• }
• }
• ]
• }

The API response returns the resulting transformed document or highlights the exact processor that failed, including the underlying error message and stack trace.

Core Processors in SecOps Normalization

Security pipelines leverage specialized processors to convert raw strings into validated ECS fields.

Parsing: dissect vs. grok

Parsing unstructured log lines is the most resource-intensive phase of ingestion. Analysts must evaluate the trade-offs between dissect and grok :

• Characteristic dissect Processor grok Processor
• Mechanism Delimiter-based string splitting (no regex) Regular expression pattern matching with regex libraries

Resource Utilization Extremely low CPU and memory footprint Moderate to high CPU overhead due to regex backtracking

Format Flexibility Strict; requires predictable, fixed delimiter structures High; handles variable tokens, optional fields, and alternate formats

Type Coercion Extracts all matched tokens as strings Supports inline data typing (e.g., %{NUMBER:source.port:long} )

Primary Use Cases Structured syslog, CEF, W3C web access logs, CSV Complex multi-format logs, Linux auditd, arbitrary app logs

• dissect Syntax : Matches field names enclosed in %{} separated by static characters: {
• "dissect": {
• "field": "message",
• "pattern": "%{timestamp} %{host.name} %{process.name}: %{event.original}"
• }
• }
• grok Syntax : Utilizes built-in ECS pattern libraries: {
• "grok": {
• "field": "message",
• "patterns": [
• "%{IP:source.ip}:%{POSINT:source.port:long} -> %{IP:destination.ip}:%{POSINT:destination.port:long}",
• "Connection from %{IP:source.ip}"
• ]
• }
• }
• Field Transformation Processors
• set : Hardcodes ECS fields or copies values between attributes. Supports dynamic Mustache templating: {
• "set": {
• "field": "event.kind",
• "value": "event"
• }
• }

rename : Maps proprietary vendor keys to standard ECS naming. Always configure "ignore_missing": true to handle optional vendor fields: {

• "rename": {
• "field": "src_ip",
• "target_field": "source.ip",
• "ignore_missing": true
• }
• }
• convert : Coerces data types (e.g., converting port strings to integers or byte counters to long): {
• "convert": {
• "field": "destination.port",
• "type": "long",
• "ignore_missing": true
• }
• }
• remove : Strips intermediate staging fields, redundant raw payloads, or sensitive authentication tokens: {
• "remove": {
• "field": ["temp_token", "unparsed_header"],
• "ignore_missing": true
• }
• }
• Enrichment Processors: date , geoip , and user_agent

date : Normalizes vendor timestamps into the standardized ISO-8601 @timestamp field. If omitted, @timestamp reflects indexing time rather than event occurrence, distorting timeline analysis: {

• "date": {
• "field": "vendor_time",
• "target_field": "@timestamp",
• "formats": ["yyyy-MM-dd HH:mm:ss", "UNIX_MS", "ISO8601"],
• "timezone": "UTC"
• }
• }

geoip : Enriches IP addresses with geographic coordinates, country, city, and Autonomous System (ASN) metadata from GeoLite2 databases that Elasticsearch downloads and keeps updated automatically: {

• "geoip": {
• "field": "source.ip",
• "target_field": "source.geo",
• "ignore_missing": true
• }
• }

user_agent : Parses raw HTTP User-Agent strings, extracting browser engine, operating system, and device classifications into user_agent.* .

Custom Attributes: labels.* vs. tags

SecOps workflows frequently require custom metadata during ingestion:

labels : A key-value dictionary of string pairs (e.g., labels.environment: "production" , labels.compliance_tier: "pci-dss" ). Used for structured filtering and organizational grouping.

tags : A flat array of keyword strings used as operational markers (e.g., tags: ["forwarded", "vpn_gateway", "_grokparsefailure"] ).

Fault Tolerance and Pipeline Error Handling

In high-throughput SOC ingestion, malformed log lines are inevitable. An unhandled exception stops the pipeline, and the document is not indexed. Dropping security logs creates critical visibility blind spots.

Defensive pipeline architecture incorporates two layers of error management:

Processor-Level on_failure : Catches errors from a single processor so the rest of the pipeline can continue (the example tags any geoip lookup error rather than failing the document): {

• "geoip": {
• "field": "source.ip",
• "target_field": "source.geo",
• "on_failure": [
• {
• "append": {
• "field": "tags",
• "value": ["_geoip_lookup_failure"]
• }
• }
• ]
• }
• }

Pipeline-Level on_failure : Captures any unhandled processor failure, extracts the error message, tags the document, and guarantees indexing so the raw message ( event.original ) remains retrievable for forensic investigation: "on_failure": [

• {
• "set": {
• "field": "error.message",
• "value": "{{ _ingest.on_failure_message }}"
• }
• },
• {
• "append": {
• "field": "tags",
• "value": ["_pipeline_failure"]
• }
• }
• ]

Loading diagram... Ingest Pipeline Transformation & Error Handling Workflow