Study guide

11.2 Centralized Observability with Amazon OpenSearch & Athena

AWS Certified DevOps Engineer – Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CI/CD Pipeline Architecture & Source Integration

1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models

2 Chapter 2: Automated Testing, Test Gates & Quality Verification

2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates

3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies

3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda

4 Chapter 4: CloudFormation, CDK & Reusable IaC Components

4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance

5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower

5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation

6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration

6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig

7 Chapter 7: High Availability & Multi-Region Resiliency

7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR

8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation

8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery

9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation

9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray

10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps

11 Chapter 11: Automated Observability & Event-Driven Monitoring

11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries

12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines

12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling

13 Chapter 13: Automated Remediation & Deployment Troubleshooting

13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures

14 Chapter 14: Identity, Access Management & Secrets at Scale

14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager

15 Chapter 15: Security Automation, Data Protection & Compliance Auditing

15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config

11.2 Centralized Observability with Amazon OpenSearch & Athena

Key Takeaways Explain This Topic AI

Amazon OpenSearch Service delivers real-time indexing, sub-second full-text log search, and Kibana/OpenSearch Dashboards visualization, whereas Amazon Athena provides serverless, ANSI SQL queries over petabyte-scale historical log archives in Amazon S3 at lower cost.

OpenSearch Serverless completely decouples compute (OpenSearch Compute Units - OCUs) from cloud storage, automatically scaling indexing and search independently without manual cluster sizing, dedicated master nodes, or shard rebalancing.

OpenSearch Ingestion (OSI) is a fully managed, serverless log pre-processor based on Data Prepper that enriches, mutates, and parses telemetry payloads before indexing, routing unparseable records to dead-letter queues (DLQs) in S3.

Amazon Kinesis Data Firehose with Dynamic Partitioning parses streaming log payloads using JQ expressions or inline Lambda transformations, dynamically partitioning S3 keys (e.g., year/month/day/service) to optimize downstream analytical queries.

Glue Data Catalog Partition Projection eliminates the high latency and throttling bottlenecks of MSCK REPAIR TABLE and Glue Crawlers by computing partition locations algorithmically in-memory during Athena query planning.

Last updated: September 2026

Centralized Observability: Operational Search vs. Historical Data Lake

Enterprise DevOps architectures generate massive volumes of heterogeneous log telemetry: application stdout logs, ALB access logs, VPC Flow Logs, AWS WAF inspection logs, and AWS CloudTrail audit events. A central responsibility of the DevOps Engineer Professional is architecting a cost-effective, high-performance telemetry pipeline that satisfies two distinct operational requirements:

Real-Time Operational Investigation : Near-real-time ingestion, low-latency search, live tailing, anomaly detection, and operational dashboards for incident triage and active debugging.

Historical Ad-Hoc Analytics & Compliance : Cost-effective retention of terabytes to petabytes of structured log data queried periodically for security retrospectives, audit compliance, and capacity planning.

Architectural Attribute Amazon OpenSearch Service / Serverless Amazon Athena + Amazon S3

Primary Paradigm Inverted index, distributed full-text search engine. Serverless distributed SQL query engine (Presto / Trino).

Ingestion Latency Near real time; end-to-end delay varies with the ingestion path and buffering. Batch or micro-batch; delay varies with Firehose buffering or export scheduling.

Query Response Time Interactive search; latency depends on index design, shard health, data volume, and query complexity. Seconds to minutes, depending on data scanned and partitioning.

Visualization Tool OpenSearch Dashboards (Kibana), live tailing, alerts. Amazon QuickSight, Jupyter Notebooks, or custom BI tools.

Cost Model Billed per instance-hour + EBS volume (or per OCU-hour). Billed $5.00 per TB of data scanned by queries + S3 storage.

Best Suited For Active triage (last 7 to 30 days of application/access logs). Long-term archives (90 days to 7+ years of raw telemetry).

Amazon OpenSearch Service Architecture & Ingestion

Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) is a managed distributed search and analytics suite. In enterprise deployments, OpenSearch clusters are architected into distinct node tiers to ensure cluster stability under heavy indexing workloads.

Cluster Topologies and Shard Optimization

Dedicated cluster manager nodes (legacy/API term: dedicated master nodes): Perform cluster coordination, index creation, shard routing, and cluster-state management without serving data queries. For production domains that use dedicated cluster managers, three nodes across three Availability Zones is the standard quorum topology.

Data Nodes : Store indexed shards and process search and indexing requests. Configured across multiple AZs (Multi-AZ with Standby provides a 99.99% SLA with automatic failover).

Storage Tiers & Cost Optimization :

Hot Tier : Utilizes high-performance local NVMe SSDs or Amazon EBS gp3 volumes for active write and high-frequency read operations.

UltraWarm Tier : Backed by Amazon S3 with warm compute caches. Reduces storage costs by up to 90% compared to hot EBS storage while retaining read-only search capabilities.

Cold Storage Tier : Fully decouples compute from storage; data resides purely on S3. When historical indices are needed, they are attached to compute on-demand.

Index State Management (ISM) : Automates lifecycle transitions. For example, an ISM policy can automatically roll over an active index when it reaches 50 GB or 1 day, migrate it to UltraWarm after 7 days, transition to Cold after 30 days, and permanently delete it after 365 days. {

  • "policy": {
  • "description": "Hot-UltraWarm-Delete Lifecycle Policy",
  • "default_state": "hot",
  • "states": [
  • {
  • "name": "hot",
  • "actions": [{"rollover": {"min_index_age": "1d", "min_primary_shard_size": "40gb"}}],
  • "transitions": [{"state_name": "warm", "conditions": {"min_index_age": "7d"}}]
  • },
  • {
  • "name": "warm",
  • "actions": [{"warm_migration": {}}],
  • "transitions": [{"state_name": "delete", "conditions": {"min_index_age": "90d"}}]
  • },
  • {
  • "name": "delete",
  • "actions": [{"delete": {}}]
  • }
  • ]
  • }
  • }

Important

Shard Sizing Rule of Thumb : Shards that are too small result in excessive Lucene index overhead and JVM memory pressure (the "over-sharding" disaster). Shards that are too large cause slow rebalancing and recovery. Aim for 10 GB to 30 GB per shard for search-heavy workloads and 30 GB to 50 GB per shard for time-series log ingestion .

OpenSearch Serverless & OpenSearch Ingestion (OSI)

For variable or unpredictably scaling workloads, OpenSearch Serverless eliminates cluster provisioning, cluster manager node configuration, and shard sizing:

Compute is decoupled from storage and metered in OpenSearch Compute Units (OCUs) , where 1 OCU provides 6 GiB of RAM and corresponding vCPU.

Data is stored durably in Amazon S3; compute capacity scales up during traffic surges and scales down automatically during idle periods.

Collections are specialized by workload type: Timeseries (optimized for logs and metrics), Search (catalog and e-commerce search), or VectorSearch (semantic AI embeddings).

OpenSearch Ingestion (OSI) is a fully managed, serverless log pre-processor based on Data Prepper. It replaces self-hosted Logstash clusters on EC2:

Receives logs from Amazon Kinesis, CloudWatch Logs, Fluent Bit, or S3.

Executes inline parsing (Grok filters), timestamp extraction, field mutations, and data masking (PII redaction).

Natively sinks enriched documents directly into OpenSearch domains or Serverless collections, with automatic Dead-Letter Queue (DLQ) routing to Amazon S3 for unparseable payloads.

Streaming Pipelines: CloudWatch Subscriptions & Kinesis Firehose Dynamic Partitioning

To move logs from source environments (EC2, ECS, Lambda, EKS) into OpenSearch and S3, AWS provides native streaming integrations centered on CloudWatch Logs subscription filters and Amazon Kinesis Data Firehose.

Subscription Filters

A CloudWatch Logs Subscription Filter processes matching events as they are ingested and forwards them to a supported destination:

Amazon Data Firehose for buffered delivery to a configured destination.

Amazon Kinesis Data Streams for fan-out and multi-consumer streaming analytics.

AWS Lambda for custom transformation or forwarding.

Amazon OpenSearch Service , using the CloudWatch Logs subscription integration and its Lambda forwarding function.

A log group can have multiple subscription filters, so independent delivery paths can feed the hot OpenSearch tier and the S3 archive.

Kinesis Data Firehose with Dynamic Partitioning

Historically, Firehose delivered data to S3 using a flat, time-based prefix: s3://bucket/YYYY/MM/DD/HH/ . When logs from multiple microservices were aggregated, analytical queries with Amazon Athena were forced to scan all services simultaneously, resulting in massive query costs.

Dynamic Partitioning solves this by evaluating the contents of each streaming JSON record using JQ expressions or an inline Lambda function to construct granular S3 prefix paths: Incoming JSON Record:

  • {"timestamp": "2026-09-11T14:32:00Z", "service": "payments", "level": "ERROR", "message": "Timeout"}
  • │
  • ▼

Firehose JQ Expression Extraction:

  • .service = "payments"
  • .timestamp = "2026-09-11"
  • │
  • ▼

Dynamic S3 Key Output:

s3://enterprise-logs/service=payments/year=2026/month=09/day=11/payments-2026-09-11-uuid.parquet

Inline Format Conversion : Firehose can automatically convert incoming JSON records into columnar Apache Parquet or Apache ORC formats before writing to S3, using an AWS Glue Data Catalog table schema.

Destination Constraint : Dynamic partitioning is supported only when Amazon S3 is the Firehose stream destination, and it must be enabled when the stream is created. A separate delivery path is required to index the same events in OpenSearch.

Serverless Log Analytics with Amazon Athena & S3

Amazon Athena allows DevOps engineers to execute standard ANSI SQL queries directly against structured and semi-structured log files in Amazon S3, without provisioning or managing database infrastructure.

Native AWS Log Ingestion Targets

Athena is widely used to analyze AWS service logs written directly to S3:

ALB Access Logs : High-volume HTTP request logs capturing client IP, latency, backend target status code, user agent, and TLS cipher.

AWS CloudTrail Logs : API auditing logs recording caller identity, IAM role ARN, source IP, event name, and request parameters.

Amazon VPC Flow Logs : Network metadata capturing source/destination IP, port, protocol, packet count, and ACCEPT/REJECT status.

AWS WAF Logs : Deep inspection logs showing rule group evaluations, terminating actions, and inspected web request headers.

Query Performance Optimization Rules

To minimize Athena query costs and optimize query execution time:

Use Columnar Formats (Parquet / ORC) : Athena charges $5.00 per TB scanned. If an ALB log table has 30 columns and a query selects only client_ip and target_status_code , Parquet allows Athena to scan only those two column stripes, reducing data scanned by up to 85–90% compared to uncompressed raw text.

Compress Data : Use Snappy compression with Parquet or Gzip with raw text to minimize S3 I/O transfer volume.

Partition by Query Predicates : Partition data by high-cardinality search dimensions (e.g., year , month , day , and service ). Queries with WHERE year = '2026' AND month = '09' prune all other partition directories completely.

Glue Data Catalog & Partition Projection

When new logs are continuously delivered to S3, Athena must know about newly created partition directories before it can query them. In traditional setups, engineers used one of two methods:

MSCK REPAIR TABLE <table_name> : Instructs Athena to scan the entire S3 bucket prefix tree using the S3 ListObjectsV2 API to discover new partition folders.

AWS Glue Crawlers : Scheduled batch jobs that crawl S3 paths and register new partitions in the Glue Data Catalog.

The Failure of Traditional Partition Discovery at Scale

In high-volume environments where logs are partitioned by minute or hour across dozens of services, a bucket accumulates hundreds of thousands of S3 prefix directories:

MSCK REPAIR TABLE scans the S3 prefix tree sequentially. With millions of objects, the command takes several hours , hits S3 API rate limits ( 503 Slow Down ), and frequently exceeds Athena query timeout limits.

Glue Crawlers incur high operational costs, introduce a 5- to 15-minute metadata lag, and require complex orchestration.

Partition Projection Architecture

Partition Projection completely eliminates MSCK REPAIR TABLE and Glue Crawlers by shifting partition calculation from static storage metadata to in-memory algorithmic generation: Traditional Query Workflow:

Athena Query ──> Queries Glue Catalog ──> Sequential S3 ListObjects ──> Extreme Latency / Timeouts

Partition Projection Workflow:

  • Athena Query (WHERE date = '2026/09/11')
  • │
  • ▼
  • Reads Table Properties: projection.date.range = 2024/01/01,NOW
  • │
  • ├─> Algorithmic In-Memory Calculation (Computes S3 URIs instantly)
  • ▼
  • Direct S3 Object Fetch: s3://logs/year=2026/month=09/day=11/
  • (Zero S3 ListObjects discovery calls, zero crawler costs, immediate execution)

Partition projection is configured directly in the table properties of the AWS Glue Data Catalog: CREATE EXTERNAL TABLE alb_access_logs (

  • type string,
  • time string,
  • elb string,
  • client_ip string,
  • target_status_code string,
  • request_url string
  • )
  • PARTITIONED BY (date string)
  • ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
  • WITH SERDEPROPERTIES ('input.regex' = '([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ...')
  • LOCATION 's3://my-enterprise-alb-logs-bucket/AWSLogs/111122223333/elasticloadbalancing/us-east-1/'
  • TBLPROPERTIES (
  • 'projection.enabled' = 'true',
  • 'projection.date.type' = 'date',
  • 'projection.date.range' = '2024/01/01,NOW',
  • 'projection.date.format' = 'yyyy/MM/dd',
  • 'projection.date.interval' = '1',
  • 'projection.date.interval.unit' = 'DAYS',

'storage.location.template' = 's3://my-enterprise-alb-logs-bucket/AWSLogs/111122223333/elasticloadbalancing/us-east-1/${date}'

);

When a query executes WHERE date = '2026/09/11' , Athena uses the projection rules to construct the exact S3 prefix .../2026/09/11/ instantly, reading the data with zero S3 directory listing overhead.

DOP-C02 Exam Watchouts & Troubleshooting

Scenario / Symptom Root Cause Solution

Athena query returns 0 records even though new log files are confirmed present in S3 partition folders New partitions have not been registered in the Glue Data Catalog Enable Partition Projection on the Glue table properties, or run ALTER TABLE ADD PARTITION dynamically via Lambda on S3 upload.

OpenSearch cluster health transitions to RED during peak log ingestion hours Heavy indexing load caused unassigned shards due to primary shard failure or JVM out-of-memory (OOM) For a production domain using dedicated cluster managers, use the recommended three-node quorum topology; verify shard sizes are between 30–50 GB; scale up data node EBS volume throughput.

Athena queries on ALB access logs take over 20 minutes and cost hundreds of dollars Logs are stored in raw text/gzip format with no date partitioning, forcing full-table scans across all historical logs Implement Kinesis Data Firehose with dynamic partitioning to convert logs into Parquet and partition by date.

CloudWatch Logs subscription filter fails to send logs to Kinesis Data Firehose The IAM role used by the subscription filter lacks firehose:PutRecord permissions or trust relationship with logs.<region>.amazonaws.com Update the IAM execution role trust policy to allow logs.amazonaws.com and grant firehose:PutRecord on the Firehose delivery stream ARN.

OpenSearch Ingestion (OSI) drops unparseable log records silently Data Prepper pipeline lacks a dead-letter queue (DLQ) configuration Configure an S3 sink in the OSI pipeline definition under the dlq block to persist failed and malformed events.

Loading diagram... Centralized Multi-Tier Log Analytics & Streaming Architecture