ExamFinch.com My Study

Study guide

9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

Practice this exam

AWS Certified DevOps Engineer – Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: CI/CD Pipeline Architecture & Source Integration

1.1 Multi-Stage & Multi-Account CodePipeline Design 1.2 CodeBuild Projects, Build Environments & Pipeline Secrets 1.3 Source Control Workflows, Webhooks & Branching Models

2 Chapter 2: Automated Testing, Test Gates & Quality Verification

2.1 Automated Unit, Integration & Acceptance Testing in Pipelines 2.2 Load Testing, Performance Benchmarking & Synthetic Canaries 2.3 Security Scanning, SAST/DAST & Approval Gates

3 Chapter 3: Artifact Management & Multi-Platform Deployment Strategies

3.1 Artifact Lifecycle Management with CodeArtifact, ECR & S3 3.2 Automated Image Baking with EC2 Image Builder 3.3 In-Place, Rolling & Immutable Deployments on EC2 3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda

4 Chapter 4: CloudFormation, CDK & Reusable IaC Components

4.1 Advanced CloudFormation: Nested Stacks, Change Sets & Custom Resources 4.2 Cross-Account & Multi-Region StackSets & Drift Detection 4.3 Modern IaC: AWS CDK, SAM & Service Catalog Governance

5 Chapter 5: Multi-Account Architecture, Organizations & Control Tower

5.1 AWS Organizations Hierarchy, OUs & Account Provisioning 5.2 Service Control Policies (SCPs) & Multi-Account Guardrails 5.3 AWS Control Tower, Account Factory & Landing Zone Automation

6 Chapter 6: Systems Manager & Large-Scale Fleet Configuration

6.1 SSM Agent, Hybrid Activation & Session Manager 6.2 Systems Manager Run Command, State Manager & Associations 6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig

7 Chapter 7: High Availability & Multi-Region Resiliency

7.1 Designing Resilient Multi-AZ & Multi-Region Architectures 7.2 Route 53 Traffic Flow, Health Checks & DNS Failover 7.3 Data Replication: Aurora Global DB, DynamoDB Global Tables & S3 CRR

8 Chapter 8: Scalability, Auto Scaling & Disaster Recovery Automation

8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools 8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway 8.3 Disaster Recovery Strategies: RTO, RPO & Architecture Tiers 8.4 Automated Backup & Recovery with AWS Backup & Elastic Disaster Recovery

9 Chapter 9: CloudWatch Metrics, Agent Collection & Log Aggregation

9.1 CloudWatch Metrics, Dimensions, Metric Math & High Resolution 9.2 Unified CloudWatch Agent & Custom Metric Collection 9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

10 Chapter 10: Log Analysis, Dashboards & Distributed Tracing with X-Ray

10.1 CloudWatch Logs Insights, Filter Syntax & Query Optimization 10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection 10.3 Distributed Tracing with AWS X-Ray & Service Maps

11 Chapter 11: Automated Observability & Event-Driven Monitoring

11.1 Application & Infrastructure Health Checks with Route 53 & ALB 11.2 Centralized Observability with Amazon OpenSearch & Athena 11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries

12 Chapter 12: Event-Driven Incident Detection & Notification Pipelines

12.1 AWS Health, CloudTrail Events & Operational Notification 12.2 EventBridge Event Routing, Patterns & Multi-Account Buses 12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling

13 Chapter 13: Automated Remediation & Deployment Troubleshooting

13.1 AWS Config Managed Rules & Automated Remediation with SSM 13.2 Event-Driven Infrastructure Self-Healing with Lambda & Step Functions 13.3 Troubleshooting CI/CD Pipeline & Deployment Failures 13.4 Root Cause Analysis for Auto Scaling & Container Failures

14 Chapter 14: Identity, Access Management & Secrets at Scale

14.1 IAM Policy Evaluation Logic, NotAction & Explicit Denies 14.2 IAM Permissions Boundaries & Delegated Administration 14.3 IAM Identity Center, Cross-Account Roles & STS Session Policies 14.4 Automated Secrets Management & Secret Rotation with Secrets Manager

15 Chapter 15: Security Automation, Data Protection & Compliance Auditing

15.1 Network Security Automation: Security Groups, NACLs, Network Firewall & WAF 15.2 Cryptographic Controls & Key Management with AWS KMS & ACM 15.3 Automated Threat Detection: GuardDuty, Inspector & Security Hub 15.4 Continuous Compliance Auditing & Log Integrity with CloudTrail & Config

9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

Key Takeaways Explain This Topic AI

Amazon CloudWatch Logs organizes telemetry into Log Groups and Log Streams; log retention periods (from 1 day to 3,653 days / 10 years, or Never Expire) control lifecycle storage costs, while AWS KMS CMKs enforce customer-controlled encryption at rest.

Metric filters evaluate streaming log events in near-real time during ingestion, extracting numerical data points (e.g., latency, response codes) using JSON or space-delimited patterns without retaining expensive log search queries.

When defining metric filters for alarms, configuring DefaultValue=0 is critical to prevent missing data anomalies and alarm state thrashing when zero error events occur.

Subscription filters deliver real-time push streaming of log events to AWS Lambda, Amazon Kinesis Data Streams, or Amazon Data Firehose, supporting up to five subscription filters per log group.

Cross-account log aggregation architecture pairs a CloudWatch Logs Destination in a central Security/Logging account with an IAM role and resource policy, allowing member accounts to stream log data directly to a central Kinesis Data Stream.

Last updated: September 2026

CloudWatch Logs Architecture: Groups, Streams, and Retention Lifecycles

Amazon CloudWatch Logs is a highly available, scalable service for centralizing operating system, application, and AWS service logs (such as Lambda execution logs, VPC Flow Logs, and API Gateway execution logs).

Log Organizational Hierarchy

Log Event : A record containing an exact timestamp (milliseconds since epoch) and a raw UTF-8 string message payload (maximum size: 1,024 KB / 1 MB per event; a single PutLogEvents batch is also capped at 1 MB).

Log Stream : A sequence of log events emitted by the same source instance, container task, or Lambda execution environment. Log streams represent an ephemeral unit of concurrency.

Log Group : A logical administrative collection of log streams that share the same access control policies, monitoring configurations, metric filters, subscription filters, and retention rules.

Ingestion Quotas and Retention Policies

By default, CloudWatch Logs retains log events indefinitely (Never Expire) . In enterprise organizations, this creates mounting storage costs for ephemeral diagnostic logs that lose business value after days or weeks.

Retention Windows : The only accepted values are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, and 3653 days, or Never Expire (achieved by deleting the retention policy).

Cost-Optimization Architecture : Set log group retention to a short operational window (e.g., 14 to 30 days) for active CloudWatch Logs Insights querying and metric alarms. For long-term audit compliance (e.g., PCI-DSS or HIPAA requiring 7-year retention), export logs to Amazon S3 and transition them to S3 Glacier Flexible Retrieval or Glacier Deep Archive via S3 Lifecycle rules.

Metric Filters: Real-Time Log Parsing and Numerical Extraction

Metric filters monitor log events as they are ingested into CloudWatch Logs in near-real time. They evaluate incoming log records against a filter pattern and publish custom metric data points without persisting or modifying the underlying log events.

Filter Pattern Syntax: JSON vs. Space-Delimited

CloudWatch Logs supports two distinct filter syntaxes:

JSON Metric Filter Patterns : Used when applications emit structured JSON logs. Supports boolean logic, numerical comparisons, and nested attribute inspection: { ($.statusCode >= 400 && $.statusCode < 500) || $.errorType = "ValidationException" }

Space-Delimited Metric Filter Patterns : Used for legacy unstructured or space-separated formats (such as Apache, NGINX, or custom text logs): [ip, user, username, timestamp, request, statusCode = 5*, bytes, responseTime > 1000]

Metric Transformation and the DefaultValue Rule

When a metric filter matches a log event, CloudWatch publishes a metric data point according to its Metric Transformation :

Metric Namespace : The target custom namespace (e.g., CustomApp/WebErrors ).

Metric Name : The name of the metric (e.g., 4xxErrorCount , ExecutionLatency ).

Metric Value : The numerical value to publish. Can be a static integer (e.g., 1 for counting events) or a dynamic extracted token (e.g., $.durationMs or $responseTime ).

Default Value : Specifies the value published when a reporting period contains no matching log events .

Important

DOP-C02 Exam Trap : If you create a CloudWatch Alarm on a metric filter that counts errors (e.g., 5xxErrorCount ) and leave DefaultValue blank ( null ), the metric will publish no data points during periods where zero errors occur. The alarm will transition to INSUFFICIENT_DATA or fail to clear. Setting DefaultValue: 0 ensures that CloudWatch publishes 0 during error-free periods, allowing the alarm to evaluate properly and transition back to OK .

Subscription Filters: Real-Time Event Streaming Destinations

While metric filters extract scalar numbers, Subscription Filters extract the complete, raw log event payload and stream it in near-real time to external processing systems.

Subscription Filter Limits and Destinations

Each CloudWatch Log Group supports up to five subscription filters . Subscription filters push log data to three supported AWS targets:

Target Service Integration Mechanism Primary Enterprise Use Case

AWS Lambda Direct synchronous invocation with base64-encoded, gzip-compressed payload Real-time security parsing, Slack/Teams alerting, SIEM webhook forwarding

Amazon Kinesis Data Streams Direct stream record injection ( kinesis:PutRecords ) High-throughput, ordered distributed processing with custom consumer fleets

Amazon Kinesis Data Firehose Direct delivery stream injection ( firehose:PutRecordBatch ) Automated buffering and delivery to Amazon S3 data lakes, Amazon OpenSearch Service, Snowflake, or Datadog

Note : Do not confuse Subscription Filters with CreateExportTask . CreateExportTask is an asynchronous batch API that dumps logs from CloudWatch Logs to Amazon S3. It cannot stream in real-time, can take hours to complete, and is limited to one active export task per account per region.

Enterprise Cross-Account Log Centralization Architecture

Large enterprises operate multi-account AWS Organizations with strict governance boundaries. Standard security architecture mandates consolidating all application, security, and audit logs from hundreds of Member Accounts into a single, hardened Central Logging Account . [ Member Account (222222222222) ]

  • CloudWatch Log Group
  • │
  • ▼ (Subscription Filter)
  • [ Central Logging Account (111111111111) ]
  • CloudWatch Logs Destination (CentralLogDestination)
  • │ (Access Policy: Allows Member Account 222222222222)
  • ▼
  • IAM Role (Assumed by CloudWatch Logs Service)
  • │ (Policy: kinesis:PutRecord* on Kinesis Stream)
  • ▼
  • Amazon Kinesis Data Stream
  • │
  • ▼
  • Amazon Kinesis Data Firehose ──> Amazon S3 / OpenSearch
  • Step 1: Central Logging Account Configuration

Kinesis Stream : Provision an Amazon Kinesis Data Stream (e.g., CentralSecurityLogStream ) in the central account ( 111111111111 ).

IAM Execution Role : Create an IAM role ( CWLogsToKinesisRole ) that the CloudWatch Logs service assumes to write records into the stream: {

  • "Version": "2012-10-17",
  • "Statement": [
  • {
  • "Effect": "Allow",
  • "Principal": {
  • "Service": "logs.us-east-1.amazonaws.com"
  • },
  • "Action": "sts:AssumeRole"
  • }
  • ]
  • }
  • Permissions Policy attached to the role: {
  • "Version": "2012-10-17",
  • "Statement": [
  • {
  • "Effect": "Allow",
  • "Action": ["kinesis:PutRecord", "kinesis:PutRecords"],
  • "Resource": "arn:aws:kinesis:us-east-1:111111111111:stream/CentralSecurityLogStream"
  • }
  • ]
  • }

CloudWatch Logs Destination : Create the destination resource using PutDestination : aws logs put-destination \

  • -destination-name "CentralLogDestination" \
  • -target-arn "arn:aws:kinesis:us-east-1:111111111111:stream/CentralSecurityLogStream" \
  • -role-arn "arn:aws:iam::111111111111:role/CWLogsToKinesisRole"

Destination Access Policy : Attach a resource policy ( PutDestinationPolicy ) granting member accounts permission to subscribe to the destination. You can grant access to individual accounts or an entire AWS Organization: {

  • "Version": "2012-10-17",
  • "Statement": [
  • {
  • "Sid": "AllowOrganizationMemberAccounts",
  • "Effect": "Allow",
  • "Principal": "*",
  • "Action": "logs:PutSubscriptionFilter",
  • "Resource": "arn:aws:logs:us-east-1:111111111111:destination:CentralLogDestination",
  • "Condition": {
  • "StringEquals": {
  • "aws:PrincipalOrgID": "o-abcdef1234"
  • }
  • }
  • }
  • ]
  • }
  • Step 2: Member Account Configuration

In each member account ( 222222222222 ), configure a Subscription Filter on the local Log Group pointing to the central destination ARN: aws logs put-subscription-filter \

  • -log-group-name "/aws/ec2/production/application" \
  • -filter-name "StreamToCentralSecurity" \
  • -filter-pattern "" \
  • -destination-arn "arn:aws:logs:us-east-1:111111111111:destination:CentralLogDestination"

Note : Setting --filter-pattern "" captures all log events. Passing a specific pattern filters events before cross-account transmission.

KMS CMK Encryption at Rest for CloudWatch Logs

By default, CloudWatch Logs encrypts stored log data using AWS-managed keys. For regulatory compliance, organizations must enforce customer control using AWS KMS Customer Managed Keys (CMKs) .

  • Associating a KMS Key with a Log Group
  • Associate a KMS CMK via CLI or CloudFormation: aws logs associate-kms-key \
  • -log-group-name "/aws/ec2/production/application" \
  • -kms-key-id "arn:aws:kms:us-east-1:111111111111:key/12345678-1234-1234-1234-123456789012"
  • KMS Key Policy Delegation Requirements

Caution

Critical Key Policy Rule : The KMS CMK key policy must explicitly grant the regional CloudWatch Logs service principal permissions to perform cryptographic operations. CloudWatch Logs uses an Encryption Context that contains the ARN of the log group. {

  • "Version": "2012-10-17",
  • "Statement": [
  • {
  • "Sid": "AllowCloudWatchLogsEncryption",
  • "Effect": "Allow",
  • "Principal": {
  • "Service": "logs.us-east-1.amazonaws.com"
  • },
  • "Action": [
  • "kms:Encrypt*",
  • "kms:Decrypt*",
  • "kms:ReEncrypt*",
  • "kms:GenerateDataKey*",
  • "kms:DescribeKey"
  • ],
  • "Resource": "*",
  • "Condition": {
  • "ArnEquals": {

"kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:us-east-1:111111111111:log-group:/aws/ec2/production/application"

  • }
  • }
  • }
  • ]
  • }
  • DOP-C02 Troubleshooting & Operational Watchouts

Issue Root Cause Resolution

Metric filter alarm triggers false alerts or stays in INSUFFICIENT_DATA during low-traffic periods DefaultValue was not configured in the metric transformation, leaving gaps during periods with 0 errors Update the metric filter transformation to include DefaultValue: 0

Member account fails to create subscription filter: InvalidParameterException: Could not deliver message to destination Destination policy in central account does not grant logs:PutSubscriptionFilter to member account ID or Org ID Verify the central account PutDestinationPolicy allows the member account principal and matches the correct region

Kinesis stream in central account throttles ( WriteProvisionedThroughputExceeded ) during peak traffic High volume of log events across member accounts saturates Kinesis shard write capacity (1 MB/sec or 1,000 records/sec per shard) Switch Kinesis Data Stream capacity mode from Provisioned to On-Demand , or scale shard count dynamically

AWS Lambda subscription target fails to process events; records appear corrupted CloudWatch Logs compresses subscription payloads with gzip and encodes them in base64 In the Lambda handler, decode the base64 string, decompress the gzip buffer using zlib.gunzip , and parse the resulting JSON string

Cross-Account Subscription Debugging Sequence

When cross-account subscriptions fail, verify in this strict sequence:

Region Symmetry : The member account Log Group, the central CloudWatch Logs Destination, and the Kinesis Data Stream must all reside in the exact same AWS Region (e.g., us-east-1 ). CloudWatch Logs Destinations do not support cross-region subscription delivery.

Destination Trust Policy : The IAM role referenced in PutDestination must have a trust policy allowing logs.<region>.amazonaws.com (or logs.amazonaws.com ).

Destination Resource Policy : The PutDestinationPolicy must explicitly allow the member account's AWS account ID (or Organization ID) to call logs:PutSubscriptionFilter against the destination resource.

Loading diagram... Cross-Account CloudWatch Log Centralization Architecture