Study guide
9.2 Custom Query & Threshold Detection Rules
Elastic Certified SIEM Analyst Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & Study Strategy
1.1 Elastic Certified SIEM Analyst Exam Facts, Objectives & Study Plan
2 Chapter 2: Elastic Stack Architecture & SIEM Foundations
2.1 Elastic Stack Architecture for Security Operations 2.2 Telemetry Ingestion Architecture & Data Pipelines 2.3 Security Data Storage, Data Streams & ILM
3 Chapter 3: Elastic Common Schema (ECS) & Ingestion Normalization
3.1 ECS Structure, Core Principles & Essential Field Sets 3.2 Ingest Pipelines, Processors & Normalization Workflows 3.3 Fleet Server & Elastic Agent Integration Management
4 Chapter 4: Discover, Data Views & Security Query Languages
4.1 Kibana Discover, Security Data Views & Runtime Fields 4.2 Kibana Query Language (KQL) & Lucene for SecOps 4.3 ES|QL for Multi-Stage Security Analysis
5 Chapter 5: Security Visualizations & Threat Analytics
5.1 Metric, Aggregation & Histogram Visualizations for SOC 5.2 Data Tables, Heatmaps & Specialized Security Charts 5.3 Elastic Maps & Geospatial Telemetry Analysis
6 Chapter 6: Kibana Lens for Security Workflows
6.1 Lens Architecture & Multi-Layer Telemetry Visualizations 6.2 Lens Formulas, Ratios & Anomaly Baseline Metrics 6.3 Operationalizing Lens in SOC Triage & Drilldowns
7 Chapter 7: Security Dashboards & SOC Operations
7.1 Security Dashboard Architecture & SOC Layout Design 7.2 Interactive Controls, Time Sync & Cross-Filtering 7.3 Dashboard Exporting, Reporting & Spaces Sharing
8 Chapter 8: The Elastic Security App: Capabilities, Explore & Dashboards
8.1 Security App Capabilities & the Explore Pages (Hosts, Network, Users) 8.2 Monitoring with the Security App Dashboards
9 Chapter 9: Elastic Security Detection Engine & Core Rules
9.1 Detection Engine Architecture & Prebuilt Rule Management 9.2 Custom Query & Threshold Detection Rules 9.3 Event Correlation Rules with Event Query Language (EQL)
10 Chapter 10: Advanced Detection Rules, ML & Rule Tuning
10.1 Threat Intelligence & Indicator Match Rules 10.2 New Terms, ES|QL & Machine Learning Detection Rules 10.3 Detection Rule Tuning, Exceptions & Alert Suppression
11 Chapter 11: Alert Triage, Timelines & Investigation Workflows
11.1 Alert Triage Workflow & the Alert Details Flyout 11.2 Elastic Security Timeline & Timeline Templates 11.3 Visual Event Analyzer & Session View Forensics
12 Chapter 12: Case Management, Integrations & AI Assistant
12.1 Elastic Security Case Management & Triage Workflows 12.2 Case Connectors & External Incident Escalation 12.3 Elastic AI Assistant for Security & Attack Discovery
13 Chapter 13: Security Administration, RBAC & Kibana Spaces
13.1 Kibana Spaces for SOC Multitenancy & Segmentation 13.2 Role-Based Access Control, Feature Privileges & Index Security 13.3 Elastic Defend Administration & Endpoint Policy Controls
14 Chapter 14: End-to-End Threat Hunting & Incident Investigation Capstone
14.1 Hypothesis-Driven Threat Hunting & Baselining in Elastic 14.2 Investigating Lateral Movement & Credential Access Campaigns 14.3 Data Exfiltration, C2 Detection & Incident Reconstruction
9.2 Custom Query & Threshold Detection Rules
Key Takeaways Explain This Topic AI
Custom Query rules evaluate KQL or Lucene query syntax against defined index patterns on a scheduled recurring interval, generating one alert per matching document or grouped alert batch.
The effective search window for scheduled rules is calculated as the rule execution interval plus additional look-back time (W = interval + look-back), creating an overlapping buffer that prevents missed alerts due to network latency and ingestion pipeline delays.
Threshold rules identify volume-based anomalies by aggregating matching events within the look-back window and triggering an alert only when the event count reaches or exceeds a defined threshold value.
Group by fields (up to three, such as source.ip or user.name) make threshold rules count per entity rather than globally, and the optional Count setting adds a unique-value (cardinality) condition.
Inappropriate look-back intervals or missing grouping fields lead to severe alert degradation—either generating redundant duplicate alerts on overlapping windows or obscuring distributed low-and-slow brute force attacks.
Last updated: September 2026
Core Detection Rule Paradigms in Elastic Security
While prebuilt rules provide comprehensive coverage against common commodity malware and recognized threat actor tactics, every enterprise presents a unique attack surface. Proprietary internal applications, bespoke network topologies, specialized administrative scripts, and organization-specific compliance mandates require SIEM analysts to engineer custom detection logic. Elastic Security provides several specialized rule types to meet these demands, with Custom Query Rules and Threshold Rules forming the fundamental workhorses of scheduled telemetry analysis.
Understanding the mathematical relationships between rule execution schedules, ingestion latency, search windows, and aggregation grouping is critical for authoring rules that reliably surface genuine security threats while eliminating false positives.
Custom Query Rules: Architecture and Formulation
Custom Query rules are the most widely deployed rule type in Elastic Security. They evaluate a search query against one or more index patterns on a recurring schedule. When documents matching the query criteria are returned, the Detection Engine generates an alert for each hit (or bundles them into an alert group). +-------------------------------------------------------------+
- | Custom Query Rule |
- | Query: process.name : "certutil.exe" and |
- | process.args : ("-urlcache" or "-split") |
- | Indices: logs-endpoint.events.*, winlogbeat-* |
- +-------------------------------------------------------------+
- |
- v
- Elasticsearch Evaluates Query DSL
- |
- +---------------------+---------------------+
- | |
- [ Document 1 Matches ] [ Document 2 Matches ]
- | |
- v v
- [ Alert Signal Created ] [ Alert Signal Created ]
- (Entity: host-01, PID: 3412) (Entity: host-09, PID: 8812)
- 1. Query Languages: KQL vs. Lucene
When authoring Custom Query rules, analysts can write logic in either Kibana Query Language (KQL) or standard Lucene syntax :
Kibana Query Language (KQL) :
Default and recommended language for most query rules.
Provides intuitive syntax, automated field completion, and seamless handling of nested ECS fields.
Example: event.category: "process" and process.name: "powershell.exe" and process.args: ("-enc" or "-encodedcommand")
Safe execution: KQL treats missing fields gracefully without throwing runtime exceptions.
Lucene Query Syntax :
Required when detection logic demands advanced search features unsupported by KQL, such as regular expressions, fuzzy matching, or field proximity searches.
- Regex matching syntax: process.command_line: /.*-e(nc|ncodedcommand).*/
- Range queries: destination.port: [1024 TO 65535]
- 2. Scoping Index Patterns and Data Streams
Custom Query rules must explicitly declare the index patterns or data streams they inspect. A common performance mistake is scoping rules against overly broad wildcards (e.g., * or logs-* ).
Best practice: Scope queries strictly to the relevant ECS data streams, such as logs-endpoint.events.process-* for process execution, logs-network.* for firewall traffic, or logs-system.security-* for Windows event logs.
Scoping to targeted data streams reduces shard-level execution overhead and dramatically speeds up query evaluation.
Ingestion Delay Mitigation and the Search Window Formula
A critical challenge in real-time threat detection is accounting for ingestion delay (also known as telemetry latency). Understanding how the Detection Engine handles temporal windows prevents critical alerts from being missed.
The Latency Challenge
Consider a security event occurring on an endpoint at exactly 12:00:00 . That timestamp is permanently stamped on the event as @timestamp . However, the event must traverse multiple architectural stages:
The endpoint agent buffers and flushes the log.
The log transmits across the WAN to an ingestion queue (e.g., Apache Kafka or Logstash).
Ingest pipelines parse, grok, and enrich the event.
Elasticsearch indexes the document into a data stream shard.
If this pipeline takes 90 seconds, the document is indexed into Elasticsearch at 12:01:30 . If a detection rule executes on a strict 1-minute interval searching only from [now - 1m TO now] , the query executing at 12:01:00 will look from 12:00:00 to 12:01:00 . At 12:01:00 , the event has not yet arrived in Elasticsearch! When the next run executes at 12:02:00 , it queries 12:01:00 to 12:02:00 . The event timestamped at 12:00:00 falls outside both search windows and is never evaluated , resulting in a missed detection.
The Look-Back Buffer Solution
To solve ingestion delay, the Detection Engine incorporates an Additional Look-Back Time parameter. The rule searches a rolling window that extends further back in time than the run interval.
The effective temporal search window ( W W W ) evaluated by the rule is calculated as:
W = interval + look-back W = \text{interval} + \text{look-back} W = interval + look-back Timeline:
- | [==================== Effective Search Window (W) ====================] |
- | |
- +---------------------------------------+---------------------------------+
- | Additional Look-Back Time | Rule Run Interval |
- | (e.g., 2m) | (e.g., 5m) |
- +---------------------------------------+---------------------------------+
- ^ ^
- now - (5m + 2m) = now - 7m now
For example, if a rule runs every 5 minutes ( interval = 5m ) with an additional look-back of 2 minutes ( look-back = 2m ), each execution searches a 7-minute historical window ( W = 5 m + 2 m = 7 m W = 5\text{m} + 2\text{m} = 7\text{m} W = 5 m + 2 m = 7 m ):
Run at 12:05 : Evaluates documents with @timestamp between 11:58:00 and 12:05:00 .
Run at 12:10 : Evaluates documents with @timestamp between 12:03:00 and 12:10:00 .
Notice that the 2-minute window between 12:03:00 and 12:05:00 is searched twice. To prevent duplicate alerts for documents identified in multiple runs, the Detection Engine applies an automated Signal Deduplication Engine based on internal document IDs and rule signatures. If an alert has already been generated for a specific underlying document, subsequent overlapping runs ignore it. Elastic recommends at least 1 minute of additional look-back, and a rule that runs every 5 minutes with 1 minute of look-back searches the last 6 minutes on each run.
Threshold Detection Rules: Architecture and Mechanics
Many attack techniques cannot be identified by examining a single isolated event. A single failed SSH login is a routine operational occurrence; 50 failed SSH logins within 3 minutes targeting a single server is an active brute force attack. Threshold rules address this challenge by aggregating events over a specified time window and alerting only when the document count reaches or exceeds a designated numerical threshold. +--------------------------------------------------------------------------+
- | Threshold Rule: SSH Brute Force Detection |
- | Query: event.category: "authentication" and event.outcome: "failure" |
- | Group By: source.ip |
- | Threshold: count() >= 10 |
- | Window: 5 minutes |
- +--------------------------------------------------------------------------+
- |
- v
- [ Aggregation by source.ip over [now - (5m + 1m) TO now] ]
- |
- +---------------------------+---------------------------+
- | |
- source.ip: 10.0.0.15 source.ip: 198.51.100.82
- Count: 2 failures Count: 42 failures
- Threshold >= 10? [ NO ] Threshold >= 10? [ YES ]
- | |
- (No Alert Fired) [ ALERT SIGNAL FIRED ]
- Entity: 198.51.100.82
- Alert Count: 42 events
- 1. The Group By Field: Entity Cardinality and Partitioning
A fundamental requirement of threshold rule engineering is defining the Group by field(s). The group-by field determines how the aggregation buckets are segmented:
Without Group By (Global Threshold) : The query calculates a single global count across all matching documents in the enterprise. If the threshold is set to 20 failed logins, and 20 different users across 5,000 corporate workstations each mistype their password once, the rule fires. This produces massive false positives.
With Single Group By (Entity Partitioning) : Grouping by source.ip partitions the count per unique originating IP address. Grouping by user.name partitions the count per target account. An alert fires only when an individual entity crosses the threshold.
Composite / Multi-Field Group By : Grouping by multiple fields (up to three, e.g., source.ip AND user.name ) identifies specific attacker-target pairings. This detects when a specific host attempts multiple logins against a specific user account.
Count (Cardinality) Condition : A threshold rule can also require a minimum number of unique values of another field. For example, Group by source.ip , Threshold ≥ 20 and Count user.name ≥ 10 alerts only when one IP produced at least 20 matching events against at least 10 different accounts, the typical shape of password spraying.
Synthetic Alerts : A threshold alert summarizes many events, so it contains only the Group by fields and counts rather than every field of the source documents. Sending the alert to Timeline lists all the matching events, including ones that did not reach the threshold.
2. Threshold Alert Suppression and Noise Control
When an ongoing attack generates hundreds or thousands of matching events within a short timeframe, an unmanaged threshold rule could generate an alert on every single scheduled execution. To manage analyst fatigue, Elastic Security allows threshold rules to configure Alert Suppression (a Platinum feature, in technical preview for threshold rules in 8.15):
When enabled, once an alert is generated for an entity (e.g., source.ip: 198.51.100.82 ), further alerts for that same entity are suppressed for a designated duration (e.g., 1 hour), even if subsequent executions continue to detect counts exceeding the threshold.
- Practical SecOps Threat Detection Scenarios
- Scenario A: RDP / Active Directory Brute Force Attack
Objective : Detect adversaries attempting rapid credential guessing against Windows Domain Controllers.
- Rule Type : Threshold Rule
- Index Pattern : winlogbeat-* , logs-system.security-*
Query : event.category: "authentication" and event.action: "logon-failed" and winlog.logon.type: "RemoteInteractive"
- Group By : source.ip , user.name
- Threshold : Count ≥ 10 \ge 10 ≥ 10
Schedule : Run every 5 minutes, Additional look-back: 2 minutes.
Scenario B: Reconnaissance & Port Scanning
Objective : Surface external network scanners probing perimeter firewalls.
- Rule Type : Threshold Rule
- Index Pattern : logs-firewall.* , logs-suricata.*
- Query : event.category: "network" and network.transport: "tcp" and event.type: "denied"
- Group By : source.ip
- Threshold : Count ≥ 100 \ge 100 ≥ 100
Schedule : Run every 2 minutes, Additional look-back: 1 minute.
Scenario C: Ransomware Mass File Modification Canary
Objective : Detect rapid file tampering or mass renaming indicative of active ransomware encryption routines.
- Rule Type : Threshold Rule
- Index Pattern : logs-endpoint.events.file-*
Query : event.category: "file" and event.action: ("rename" or "deletion") and file.extension: ("locked" or "crypted" or "enc")
- Group By : host.id , process.entity_id
- Threshold : Count ≥ 50 \ge 50 ≥ 50
Schedule : Run every 1 minute, Additional look-back: 1 minute.
The "Low-and-Slow" Evasion Challenge
A known limitation of threshold rules is vulnerability to "low-and-slow" attacks. If an adversary knows an organization alerts on ≥ 10 \ge 10 ≥ 10 failed authentications within 5 minutes, the attacker can pace attempts to occur once every 2 minutes (2.5 attempts per 5 minutes). The threshold is never crossed. SIEM analysts counter low-and-slow tactics by deploying complementary Machine Learning anomaly detection rules or expanding threshold windows to hours or days with lower proportionate thresholds.
Custom Query Rules vs. Threshold Rules Comparison
The following table contrasts the technical, architectural, and operational characteristics of Custom Query rules versus Threshold rules:
Specification Custom Query Rule Threshold Rule
Primary Purpose Detect specific, high-fidelity single events or signatures Detect volume-based anomalous aggregations over time
Query Structure KQL or Lucene search expression KQL or Lucene search expression + Aggregation filter
Execution Engine Standard Elasticsearch Query DSL (hits search) Elasticsearch Terms / Date Histogram Aggregations
Alert Generation One alert per matching document (or 1 grouped alert) One alert per grouped entity bucket exceeding count ( N N N )
Group By Field Optional (used for alert suppression) Optional, but usually essential for entity-level thresholds (up to 3 fields)
Search Window ( W W W ) W = interval + look-back W = \text{interval} + \text{look-back} W = interval + look-back W = interval + look-back W = \text{interval} + \text{look-back} W = interval + look-back
Typical Threat Use Cases LOLBin execution, credential dumping, malware hashes Brute force logins, port scans, DDoS, ransomware file churn
Primary False Positive Risk Benign administrative scripts matching broad queries High baseline activity crossing global un-partitioned thresholds
Primary Evasion Risk Adversary alters command-line parameters or filenames Adversary slows attack rate below threshold window ("low-and-slow")