Study guide

4.1 Kibana Discover, Security Data Views & Runtime Fields

Elastic Certified SIEM Analyst Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & Study Strategy

1.1 Elastic Certified SIEM Analyst Exam Facts, Objectives & Study Plan

2 Chapter 2: Elastic Stack Architecture & SIEM Foundations

2.1 Elastic Stack Architecture for Security Operations 2.2 Telemetry Ingestion Architecture & Data Pipelines 2.3 Security Data Storage, Data Streams & ILM

3 Chapter 3: Elastic Common Schema (ECS) & Ingestion Normalization

3.1 ECS Structure, Core Principles & Essential Field Sets 3.2 Ingest Pipelines, Processors & Normalization Workflows 3.3 Fleet Server & Elastic Agent Integration Management

4 Chapter 4: Discover, Data Views & Security Query Languages

4.1 Kibana Discover, Security Data Views & Runtime Fields 4.2 Kibana Query Language (KQL) & Lucene for SecOps 4.3 ES|QL for Multi-Stage Security Analysis

5 Chapter 5: Security Visualizations & Threat Analytics

5.1 Metric, Aggregation & Histogram Visualizations for SOC 5.2 Data Tables, Heatmaps & Specialized Security Charts 5.3 Elastic Maps & Geospatial Telemetry Analysis

6 Chapter 6: Kibana Lens for Security Workflows

6.1 Lens Architecture & Multi-Layer Telemetry Visualizations 6.2 Lens Formulas, Ratios & Anomaly Baseline Metrics 6.3 Operationalizing Lens in SOC Triage & Drilldowns

7 Chapter 7: Security Dashboards & SOC Operations

7.1 Security Dashboard Architecture & SOC Layout Design 7.2 Interactive Controls, Time Sync & Cross-Filtering 7.3 Dashboard Exporting, Reporting & Spaces Sharing

8 Chapter 8: The Elastic Security App: Capabilities, Explore & Dashboards

8.1 Security App Capabilities & the Explore Pages (Hosts, Network, Users) 8.2 Monitoring with the Security App Dashboards

9 Chapter 9: Elastic Security Detection Engine & Core Rules

9.1 Detection Engine Architecture & Prebuilt Rule Management 9.2 Custom Query & Threshold Detection Rules 9.3 Event Correlation Rules with Event Query Language (EQL)

10 Chapter 10: Advanced Detection Rules, ML & Rule Tuning

10.1 Threat Intelligence & Indicator Match Rules 10.2 New Terms, ES|QL & Machine Learning Detection Rules 10.3 Detection Rule Tuning, Exceptions & Alert Suppression

11 Chapter 11: Alert Triage, Timelines & Investigation Workflows

11.1 Alert Triage Workflow & the Alert Details Flyout 11.2 Elastic Security Timeline & Timeline Templates 11.3 Visual Event Analyzer & Session View Forensics

12 Chapter 12: Case Management, Integrations & AI Assistant

12.1 Elastic Security Case Management & Triage Workflows 12.2 Case Connectors & External Incident Escalation 12.3 Elastic AI Assistant for Security & Attack Discovery

13 Chapter 13: Security Administration, RBAC & Kibana Spaces

13.1 Kibana Spaces for SOC Multitenancy & Segmentation 13.2 Role-Based Access Control, Feature Privileges & Index Security 13.3 Elastic Defend Administration & Endpoint Policy Controls

14 Chapter 14: End-to-End Threat Hunting & Incident Investigation Capstone

14.1 Hypothesis-Driven Threat Hunting & Baselining in Elastic 14.2 Investigating Lateral Movement & Credential Access Campaigns 14.3 Data Exfiltration, C2 Detection & Incident Reconstruction

4.1 Kibana Discover, Security Data Views & Runtime Fields

Key Takeaways Explain This Topic AI

Kibana Discover is the primary interactive investigation workspace in the Elastic Stack for inspecting raw security telemetry, analyzing temporal event distributions, and tracking process lineage.

Data views bind Discover and the Security app to target data streams (such as logs-* or the .alerts-security.alerts-<space-id> alias) and define the @timestamp time field and custom field formatters.

The View surrounding documents feature in Discover shows the events immediately before and after a selected document in the same data view, in time order; adding a host filter focuses it on one entity.

Runtime fields execute Painless scripts at query time to extract or compute values on the fly without requiring reindexing or modifying underlying Elasticsearch index mappings.

While runtime fields provide maximum agility for threat hunting and rapid schema prototyping, they consume query-time CPU cycles and must be evaluated against pre-indexed fields for high-frequency detection rules.

Last updated: September 2026

In modern security operations centers (SOCs), rapid triage depends on an analyst's ability to pivot seamlessly from macroscopic threat indicators to microscopic raw event records. Kibana Discover serves as the ground-truth investigation workspace within the Elastic Stack, enabling security engineers and analysts to inspect unstructured and structured telemetry, uncover event spikes across time boundaries, and formulate exploratory threat hypotheses.

Understanding how Discover interacts with Elasticsearch indices through Security Data Views and dynamically parses incoming data via Runtime Fields is essential for effective security monitoring, rapid incident response, and performance-conscious query design.

Navigating Kibana Discover in Security Operations

Discover exposes the raw event stream indexed into Elasticsearch. While curated dashboards and the Elastic Security app offer specialized analytical abstractions, Discover provides unfiltered access to fields defined in the Elastic Common Schema (ECS). +---------------------------------------------------------------------------------------+

  • | KIBANA DISCOVER UI |
  • +---------------------------------------------------------------------------------------+
  • | [Data View: logs-* v] [KQL: host.name: "srv-web01" and event.outcome: "failure" ] |
  • | [Time Picker: Last 24 Hours (Absolute: 2026-09-29T00:00:00Z to 2026-09-30T00:00:00Z)] |
  • +---------------------------------------------------------------------------------------+
  • | EVENT DISTRIBUTION HISTOGRAM (Binned by 30-minute intervals) |
  • | || ||| ||||||||||| (Anomaly Spike: 14:30 UTC) || |
  • | || ||| .. ||||||||||| .. || .. |
  • +---------------------------------------------------------------------------------------+
  • | FIELD SIDEBAR | CUSTOM DOCUMENT TABLE |
  • | Selected Fields: | @timestamp | host.name | user.name | process.name |
  • | - @timestamp | 2026-09-29T14:30:12Z | srv-web01 | root | sshd |
  • | - host.name | 2026-09-29T14:30:14Z | srv-web01 | admin | sshd |
  • | - user.name | 2026-09-29T14:30:17Z | srv-web01 | test | sshd |
  • | - process.name | |
  • | Available Fields... | [> Expand Doc] -> Flyout (JSON / Single-Doc / Surrounding Docs) |
  • +---------------------------------------------------------------------------------------+
  • Time Picker Mechanics

Accurate temporal scoping prevents query timeouts and cluster resource exhaustion while ensuring analysts capture relevant evidence:

Quick Intervals : Pre-configured dynamic windows (e.g., Last 15 minutes , Last 24 hours , Last 7 days ). These automatically calculate relative offsets from the current execution moment.

Relative Time : Configured with custom lookbacks, such as now-4h to now , or rounding boundaries like now-1d/d (looking back one full calendar day rounded to midnight).

Absolute Time : Fixed ISO 8601 start and end boundaries (e.g., 2026-09-29T08:00:00.000Z to 2026-09-29T12:00:00.000Z ). Absolute ranges are mandatory when reconstructing forensic evidence during incident containment.

Auto-Refresh : Live-polling interval (e.g., refresh every 5 seconds or 1 minute) used on SOC overview monitors tracking active intrusion scenarios.

Analyst Tip on Timezones: Elasticsearch natively stores all dates internally in Coordinated Universal Time (UTC) formatted as ISO 8601. Kibana translates timestamps into the browser's local timezone by default. In multi-region incident response, analysts should standardize Kibana's Advanced Settings ( dateFormat:tz ) to UTC to avoid temporal misalignments across distributed investigation teams.

The Event Distribution Histogram

The histogram positioned directly above the document table plots event frequency over the selected time range. It relies on an Elasticsearch date_histogram aggregation, automatically binning counts into intervals (seconds, minutes, hours, or days) determined by the total query span.

Spike Detection : Sudden vertical surges often indicate high-velocity events such as brute-force authentication attacks, port scans, or network fuzzing.

Ingestion Troughs & Blackouts : Complete drops in event counts indicate ingestion pipeline failures, log shipper crashes, or network partitioning between endpoints and Fleet Server.

Interactive Temporal Zooming : Clicking and dragging across an anomalous histogram peak immediately constrains the Discover time picker to that exact micro-window, filtering millions of background records down to the intrusion burst.

Document Table Customization

By default, Discover renders matching documents with the @timestamp column and the full unparsed _source document. In high-volume triage, this creates visual noise. Analysts customize the view by:

Adding Columns from the Field Sidebar : Hovering over ECS fields (e.g., host.name , user.name , event.action , process.name , source.ip , destination.ip ) and selecting Add creates a clean, tabular presentation.

Column Sorting : Clicking column headers toggles ascending or descending sort orders. Elasticsearch executes this sorting against the indexed values of keyword , date , or numeric fields.

Field Statistics : Selecting a field in the sidebar shows its top values and their share of the current documents, allowing analysts to spot outlier usernames or unusual destination ports immediately.

Document Flyout & Surrounding Documents

Clicking the expand icon next to any event opens the Document Flyout , displaying structured key-value pairs categorized by ECS domain, as well as the raw JSON payload in _source .

From the flyout, analysts can apply instantaneous inclusion filters ( + ) or exclusion filters ( - ), check field mapping definitions, or copy field values. Crucially, the flyout offers View Surrounding Documents :

In incident triage, an alert typically fires on a single isolated event (e.g., an unauthorized PowerShell execution).

Selecting View surrounding documents opens a context view showing a configurable number of documents immediately before and after the selected event, in time order, from the same data view.

The context view does not automatically limit results to the same host. Add a filter such as host.name: "srv-dc01" to focus it on one entity. Kibana sorts by the data view's time field and uses tiebreaker fields (the context:tieBreakerFields advanced setting, _doc by default) to keep a stable order when events share a timestamp.

Security Data Views Architecture

A Data View (formerly known as an Index Pattern in older Elastic Stack versions) informs Kibana which Elasticsearch indices, data streams, or index aliases contain the telemetry an analyst wishes to explore.

Stream Matching and Wildcards

Data Views use glob-style wildcards to target specific subsets of security data:

logs-* : Matches all general log data streams across Windows, Linux, network appliances, and cloud workloads.

logs-endpoint.events.* : Targets endpoint sensor activity generated by the Elastic Defend integration.

.alerts-security.alerts-default : The hidden alias holding detection alerts for the default space (one alias per Kibana space, .alerts-security.alerts-<space-id> ).

auditbeat-*,packetbeat-*,filebeat-* : Targets legacy Beat shippers.

Multiple comma-separated streams can be combined in a single Data View, enabling cross-source correlation across disparate infrastructure layers.

The Security app uses its own default data view ( security-solution-<space-id> ), built from the securitySolution:defaultIndex advanced setting. By default it includes Beats and Elastic Agent patterns such as logs-* , filebeat-* , auditbeat-* , packetbeat-* , winlogbeat-* and endgame-* , plus -*elastic-cloud-logs-* , which excludes Elastic Cloud's own logs.

Default Timestamp Configuration

Every time-based Data View requires a primary time field, almost universally configured as @timestamp in ECS-compliant environments. This field governs:

The chronological axis of the Discover histogram.

The range filter applied by the Kibana time picker.

Time-based shard skipping: Elasticsearch can skip shards whose time range cannot match the selected window, which keeps long-retention searches fast.

Field Formatters

Data Views allow administrators to define presentation rules for fields without modifying underlying documents:

String Formatters : Enforcing uppercase, lowercase, or regex substring transforms.

URL Templates : Transforming IP addresses, domain names, or file hashes into clickable hyperlinks directing analysts to internal asset databases or external threat intelligence lookups (e.g., VirusTotal or internal ticketing systems).

Bytes & Duration Formatters : Formatting raw integer values (e.g., 1048576 bytes to 1 MB , or 3500 ms to 3.5s ) for rapid operational interpretation.

Runtime Fields in Security Operations

A Runtime Field is an Elasticsearch field evaluated dynamically at query time using a Painless script. Unlike indexed fields, which are parsed and written to immutable inverted indices or BKD trees during ingestion, runtime fields calculate their values on the fly whenever a query executes. +---------------------------------------------------------------------------------+

  • | INDEXED FIELDS VS RUNTIME FIELDS |
  • +---------------------------------------------------------------------------------+
  • | INGEST-TIME PATH (Indexed Fields): |
  • | Log Ingestion -> Ingest Pipeline -> Inverted Index / Doc Values -> Shard Storage |
  • | [Disk footprint high; Ingest CPU cost; Instantaneous query execution at scale] |
  • +---------------------------------------------------------------------------------+
  • | QUERY-TIME PATH (Runtime Fields): |
  • | Query Execution -> Shard Scan -> Painless Script Evaluated -> Results Emitted |
  • | [Zero disk storage; Zero reindexing; Compute CPU overhead during search] |
  • +---------------------------------------------------------------------------------+
  • Scopes: Ephemeral vs. Saved Runtime Fields

Saved Runtime Fields in Data Views : Configured directly in Kibana under Stack Management > Data Views . Once saved, the field appears in the Discover sidebar and can be queried using KQL, visualized in Kibana Lens, and referenced in detection rules just like any pre-indexed ECS field.

Index-Level Runtime Fields : Defined directly inside the Elasticsearch index mapping ( _mapping ). Any query hitting that index across any Kibana space or API client can consume the field.

Ephemeral Runtime Fields : Defined dynamically within the body of a specific search request using the runtime_mappings parameter. These exist exclusively for the duration of that single query execution, perfect for ad-hoc hunting scripts.

Painless Scripting for Ad-Hoc Parameter Parsing

When new attack techniques emerge or unmapped logs arrive without proper ingest pipelines, waiting for reindexing or pipeline redeployment impedes urgent threat containment. Runtime fields allow SOC analysts to extract critical indicators immediately.

Example 1: Extracting DNS Subdomains

Consider a scenario where dns.question.name contains subdomain.attacker-c2.com , but the SOC needs to aggregate queries by the top-level domain or extract high-entropy subdomains used for DNS tunneling. // Painless script to extract the first subdomain label

  • String domain = doc['dns.question.name'].value;
  • if (domain != null) {
  • int firstDot = domain.indexOf('.');
  • if (firstDot > 0) {
  • emit(domain.substring(0, firstDot));
  • } else {
  • emit(domain);
  • }
  • }
  • Example 2: Calculating Session Duration

Calculating session length in seconds across authentication sessions where event.start and event.end exist as separate timestamps: if (doc.containsKey('event.start') && doc.containsKey('event.end') &&

  • !doc['event.start'].empty && !doc['event.end'].empty) {
  • long durationMs = doc['event.end'].value.toEpochMilli() - doc['event.start'].value.toEpochMilli();
  • emit(durationMs / 1000.0);
  • }
  • Architectural Tradeoffs in Enterprise SOC Clusters

While runtime fields provide unprecedented agility, deploying them across high-volume security data streams introduces serious performance considerations:

Architectural Dimension Indexed Fields (ECS Ingest Pipeline) Runtime Fields (Painless / Data View)

Query Execution Speed Extremely fast (sub-millisecond to seconds); leverages Lucene inverted indices and columnar doc values. Slower; execution speed scales linearly with document count as the Painless script executes per record.

Disk Storage Footprint Higher; data is written into immutable segment files and auxiliary indexing structures. Zero additional disk storage; values are calculated in-memory and discarded post-query.

Indexing Throughput Lower; ingest node CPU must execute pipeline parsers prior to document serialization. Maximum throughput; ingestion bypasses parsing and writes raw payloads directly.

Mapping Flexibility Rigid; modifying fields requires updating templates and reindexing historical indices. Flexible; script changes take effect instantly across historical data without reindexing.

Aggregations & Sorting Highly optimized via native columnar doc values. High CPU consumption; aggregating millions of documents requires on-the-fly script evaluation.

Recommended Use Case High-cardinality detection rules, dashboard visualizations, long-term threat baselines. Rapid incident prototyping, temporary threat hunting pivots, parsing obscure legacy fields.

Best Practice Lifecycle: Treat runtime fields as an exploratory staging ground. When a runtime field proves indispensable for automated alerting or high-frequency dashboards, migrate the parsing logic upstream into an Elastic Ingest Pipeline processor ( dissect , grok , or script ) so future logs are indexed natively.