Study guide

2.4 Troubleshooting Security Telemetry & Log Ingestion

AWS Certified Security – Specialty SCS-C03

Track your progress as you read

Finished sections get a check mark.

1 Chapter 0: Getting Started with the SCS-C03 Exam

0.1 SCS-C03 Exam Format, Domains & Study Strategy

2 Chapter 1: Security Monitoring, Alerting & Threat Detection Services

1.1 Workload Monitoring & Health Check Strategies 1.2 Amazon GuardDuty Architecture & Finding Types 1.3 AWS Security Hub & Automated Conformance Assessment 1.4 Amazon Macie & Automated Data Classification

3 Chapter 2: Enterprise Logging Architecture & Analysis

2.1 Multi-Account CloudTrail & Organization Trails 2.2 Amazon Security Lake & OCSF Log Normalization 2.3 Network Telemetry: VPC Flow Logs & Route 53 Resolver Logs 2.4 Troubleshooting Security Telemetry & Log Ingestion 2.5 Analyzing, Normalizing & Correlating Security Logs

4 Chapter 3: Incident Readiness, Response Planning & Automated Runbooks

3.1 Incident Response Planning & Blast Radius Mitigation 3.2 Testing & Validating Incident Response Plans 3.3 Automated Incident Remediation with EventBridge & Step Functions 3.4 Systems Manager OpsCenter & Incident Manager Runbooks

5 Chapter 4: Digital Forensics, Evidence Preservation & Root Cause Analysis

4.1 Forensic Artifact Capture & Chain of Custody 4.2 Threat Containment & Network Quarantine Techniques 4.3 Correlating Events & Root Cause Analysis with Amazon Detective

6 Chapter 5: Edge Protection, WAF & DDoS Mitigation

5.1 CloudFront Security Headers & Origin Access Control 5.2 AWS WAF Architecture & Advanced Rule Sets 5.3 AWS Shield Advanced & DDoS Defense Architectures 5.4 Edge Integrations: Managed Rule Groups, OCSF Ingestion, IoT Policies & CORS

7 Chapter 6: Compute Workload Hardening, Vulnerability Management & GenAI Security

6.1 Golden AMIs & Container Image Hardening Pipelines 6.2 Amazon Inspector Vulnerability Management & Patch Automation 6.3 Generative AI Security: Amazon Bedrock Guardrails & Model Protection 6.4 Instance Profiles, Service Roles & Execution Roles for Compute 6.5 Secure Administrative Access: Session Manager & EC2 Instance Connect

8 Chapter 7: Network Security, Segmentation, Private Connectivity & Encryption in Transit

7.1 Layered Traffic Filtering: SGs, NACLs & AWS Network Firewall 7.2 Hybrid Connectivity Security: VPN, Direct Connect & MACsec 7.3 Zero-Trust Access & Network Access Analysis 7.4 Enforcing Encryption in Transit: ELB Security Policies, TLS & HTTPS-Only Access 7.5 Private Access: VPC Endpoints, PrivateLink, Client VPN & Verified Access 7.6 Inter-Resource Encryption in Transit: Nitro, EMR, EKS & SageMaker AI

9 Chapter 8: Enterprise Authentication, Federation & Directory Services

8.1 AWS IAM Identity Center & Enterprise IdP Integration 8.2 Amazon Cognito User Pools & Identity Pools 8.3 AWS Security Token Service (STS) & Temporary Credential Workflows 8.4 Troubleshooting Identity & Authentication Failures

10 Chapter 9: Advanced Authorization Architecture & Policy Evaluation

9.1 Complete AWS Policy Evaluation Logic & Decision Flow 9.2 Permission Boundaries & Safe Admin Delegation 9.3 Attribute-Based Access Control (ABAC) with IAM Tags

11 Chapter 10: Least Privilege, Policy Validation & External Access Auditing

10.1 IAM Roles Anywhere & Amazon Verified Permissions 10.2 Auditing External & Unused Access with IAM Access Analyzer 10.3 Analyzing Authorization Failures & Policy Simulation

12 Chapter 11: Cryptographic Services: KMS Architecture & Key Governance

11.1 AWS KMS Architecture, Key Policies & Key Types 11.2 KMS Grants, Encryption Context & Multi-Region Keys 11.3 Imported Key Material (BYOK) & CloudHSM Key Stores

13 Chapter 12: Secrets Management, Sensitive Data Masking & PKI

12.1 AWS Secrets Manager & Systems Manager Parameter Store 12.2 Sensitive Data Masking in Telemetry & Messaging 12.3 AWS Certificate Manager (ACM) & Private CA Architecture

14 Chapter 13: Storage Encryption, Integrity, Lifecycle & Backup Protection

13.1 S3 Encryption Modes & S3 Bucket Keys 13.2 Data Integrity Protection: Object Lock, Vault Lock & Code Signing 13.3 Secure Data Lifecycle, Cross-Account Replication & Backup Vault Lock 13.4 Lifecycle, Backup & Secure Replication Beyond S3: EFS, FSx, DLM, DataSync & Air-Gapped Vaults

15 Chapter 14: Multi-Account Architecture, Organizations & Control Tower Governance

14.1 AWS Organizations Structure, Hierarchy & Resource Control 14.2 AWS Control Tower Landing Zone & Guardrail Framework 14.3 Delegated Administrator Architecture & Root User Governance 14.4 Centralized Resource Sharing with RAM & Service Catalog

16 Chapter 15: Compliance Automation, Infrastructure as Code Security & Auditing

15.1 Infrastructure as Code Security & Multi-Account Deployment 15.2 AWS Config, Conformance Packs & Automated Remediation 15.3 Audit Evidence Collection & Well-Architected Security Reviews

2.4 Troubleshooting Security Telemetry & Log Ingestion

Key Takeaways Explain This Topic AI

The CloudWatch Unified Agent requires explicit IAM instance profile permissions (logs:CreateLogStream, logs:PutLogEvents, logs:DescribeLogStreams) and network line-of-sight to regional CloudWatch endpoints.

Amazon API Gateway logging fails silently unless an IAM role with the AmazonAPIGatewayPushToCloudWatchLogs managed policy is configured in the global regional API Gateway Account Settings.

AWS Lambda execution roles require logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents permissions, and KMS key policies on encrypted log groups must explicitly grant access to logs.<region>.amazonaws.com .

VPC Flow Logs delivered to S3 require the delivery.logs.amazonaws.com service principal with aws:SourceAccount conditions, whereas CloudTrail requires cloudtrail.amazonaws.com with bucket-owner-full-control ACL enforcement.

CloudFront standard logging (legacy) to S3 needs a bucket with ACLs enabled (not Bucket owner enforced); SSE-KMS works only with a customer managed key whose key policy lets delivery.logs.amazonaws.com call kms:GenerateDataKey*, never with aws/s3.

Last updated: September 2026

Systematic Troubleshooting Methodology for Security Telemetry

Security telemetry pipelines fail for predictable architectural reasons: IAM authorization failures, service principal mismatches in resource policies, network line-of-sight blockages, and incompatible cryptographic configurations. For the AWS Certified Security – Specialty examination, you must possess the ability to rapidly isolate the root cause when logs fail to deliver. [ Log Source ] ──> (1. IAM / Trust) ──> (2. Network / Endpoints) ──> (3. Service Principal) ──> (4. KMS Encryption) ──> [ Destination ]

When a telemetry stream ceases or fails to initialize, evaluate these four layers systematically:

Identity & Execution Role : Does the entity producing the logs have permissions to call PutLogEvents or assume the logging role?

Network Path : Can the client establish TLS sessions to logs.<region>.amazonaws.com through NAT or VPC Endpoints?

Resource / Bucket Policy : Does the receiving S3 bucket policy trust the exact AWS service principal and satisfy all condition keys?

Cryptographic Key Policy : Does the destination KMS key policy grant kms:GenerateDataKey* and kms:DescribeKey to the delivering service with the proper encryption context?

Diagnosing CloudWatch Unified Agent Failures

The Amazon CloudWatch Unified Agent is installed on EC2 instances and on-premises servers to collect system metrics and log files (e.g., /var/log/secure , /var/log/audit/audit.log , Windows Event Logs). When log streams fail to appear in CloudWatch Logs, investigate the following failure modes:

1. Missing or Inadequate IAM Permissions

The EC2 instance profile must have an attached IAM policy granting CloudWatch Logs write operations. Attach the AWS-managed policy CloudWatchAgentServerPolicy or a scoped custom policy containing: {

  • "Version": "2012-10-17",
  • "Statement": [
  • {
  • "Effect": "Allow",
  • "Action": [
  • "logs:CreateLogGroup",
  • "logs:CreateLogStream",
  • "logs:PutLogEvents",
  • "logs:DescribeLogStreams"
  • ],
  • "Resource": "arn:aws:logs:*:*:*"
  • }
  • ]
  • }
  • 2. Host-Level File Permissions

The Unified Agent typically runs as a dedicated non-root service account ( cwagent ). If target operating system log files (such as /var/log/audit/audit.log ) have file permissions set to 0600 owned strictly by root:root , the cwagent process cannot read the file. The local agent log ( /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log ) will report open /var/log/audit/audit.log: permission denied .

3. Network Line-of-Sight & VPC Endpoints

EC2 instances residing in private subnets without an internet gateway or NAT gateway cannot reach the public CloudWatch Logs regional API. You must provision an Interface VPC Endpoint ( com.amazonaws.<region>.logs ). Common pitfalls include:

Security Group on the VPC Endpoint : The security group attached to the Interface Endpoint must permit inbound TCP port 443 from the private subnet CIDR.

Private DNS : Private DNS must be enabled on the VPC Endpoint so that SDK and agent calls to logs.<region>.amazonaws.com resolve automatically to the endpoint's private IP addresses.

API Gateway Execution & Access Logging Failures

Amazon API Gateway supports two types of logging:

Access Logging : Logs client caller identity, request path, IP address, and response latency.

Execution Logging : Logs detailed internal execution steps, authorizer execution, request/response transformation, and integration backend errors.

The Global Account Settings Trap

A frequent point of failure is that enabling execution or access logging on an individual API stage appears to succeed in the console, but no log groups or log streams are ever created in CloudWatch Logs.

Root Cause : API Gateway requires an IAM role configured globally at the API Gateway Account Settings level for that region. API Gateway cannot assume a role specified per-stage; it relies on a regional account-level role. [ API Gateway Console / API ]

  • └── Settings
  • └── CloudWatch log role ARN: arn:aws:iam::111122223333:role/APIGatewayLoggingRole
  • The IAM role must have a trust relationship allowing apigateway.amazonaws.com : {
  • "Version": "2012-10-17",
  • "Statement": [
  • {
  • "Effect": "Allow",
  • "Principal": { "Service": "apigateway.amazonaws.com" },
  • "Action": "sts:AssumeRole"
  • }
  • ]
  • }

And it must have the managed policy AmazonAPIGatewayPushToCloudWatchLogs attached, granting logs:CreateLogGroup , logs:CreateLogStream , logs:PutLogEvents , and logs:DescribeLogStreams .

Full Request/Response Data Tracing Security Risk

When enabling Execution Logging, administrators can enable Log full requests/responses data . While invaluable for debugging, this logs entire request bodies, query strings, and headers into plaintext CloudWatch Logs. If requests contain passwords, API keys, or credit card numbers, this creates an unmasked compliance violation.

AWS Lambda Execution Role Permissions

When an AWS Lambda function executes, it streams execution outputs ( stdout , stderr , and runtime errors) to a default log group named /aws/lambda/<function-name> .

Required IAM Actions

The Lambda function's execution role must contain:

  • logs:CreateLogGroup
  • logs:CreateLogStream

logs:PutLogEvents These are included in the AWS-managed policy AWSLambdaBasicExecutionRole .

Encrypted Log Groups with Customer Managed Keys (CMK)

If the CloudWatch Log Group is encrypted with an AWS KMS Customer Managed Key, who needs KMS permissions? This is a classic exam question.

The Lambda execution role does NOT require KMS permissions to write logs. The Lambda runtime passes log data to the CloudWatch Logs service.

The CloudWatch Logs Service Principal ( logs.<region>.amazonaws.com ) MUST be granted KMS permissions in the KMS Key Policy: {

  • "Sid": "AllowCloudWatchLogsKeyAccess",
  • "Effect": "Allow",
  • "Principal": { "Service": "logs.us-east-1.amazonaws.com" },
  • "Action": [
  • "kms:Encrypt*",
  • "kms:Decrypt*",
  • "kms:ReEncrypt*",
  • "kms:GenerateDataKey*",
  • "kms:DescribeKey"
  • ],
  • "Resource": "*",
  • "Condition": {
  • "ArnLike": {
  • "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:us-east-1:111122223333:log-group:*"
  • }
  • }
  • }
  • S3 Bucket Policy & KMS Key Policies: CloudTrail vs. VPC Flow Logs

When log services deliver directly to Amazon S3, you must use the correct service principal. Using the wrong service principal in the bucket policy causes delivery to fail with Access Denied .

Log Service Correct Service Principal Required S3 Bucket Actions Critical Condition Keys

AWS CloudTrail cloudtrail.amazonaws.com s3:GetBucketAcl , s3:PutObject s3:x-amz-acl: bucket-owner-full-control , aws:SourceArn or aws:SourceAccount

VPC Flow Logs (to S3) delivery.logs.amazonaws.com s3:GetBucketAcl , s3:PutObject aws:SourceAccount

VPC Flow Logs (to CloudWatch) vpc-flow-logs.amazonaws.com Assumes an IAM Role with logs:PutLogEvents Trust policy on IAM role for vpc-flow-logs.amazonaws.com

Exam Trap : A common distractor on the exam proposes configuring vpc-flow-logs.amazonaws.com in an S3 bucket policy. This will fail! When VPC Flow Logs writes to Amazon S3, AWS uses the Log Delivery service, whose service principal is delivery.logs.amazonaws.com .

Confused Deputy Prevention with aws:SourceArn

To prevent an adversary from abusing an AWS service to write unauthorized data to your audit bucket, always include aws:SourceArn or aws:SourceAccount in your bucket policy condition blocks.

CloudFront Logging Debugging: Standard vs. Real-Time Logs

Amazon CloudFront provides two distinct logging mechanisms with fundamentally different operational constraints.

Standard Access Logs (S3 Delivery)

Standard logs are delivered as compressed gzip files within minutes to an hour or so. The original S3 delivery is now labelled standard logging (legacy) ; the newer standard logging (v2) can also send the same records to CloudWatch Logs or Amazon Data Firehose.

The two bucket traps in legacy S3 delivery :

ACLs must be enabled. CloudFront grants the awslogsdelivery account FULL_CONTROL through the bucket ACL. New buckets default to S3 Object Ownership Bucket owner enforced , which disables ACLs, so logs never arrive until ACLs are enabled on that bucket.

KMS key choice. SSE-S3 always works. SSE-KMS works only with a customer managed key whose key policy allows the delivery.logs.amazonaws.com service principal to call kms:GenerateDataKey* (plus kms:Decrypt if S3 Bucket Keys are enabled). The AWS managed key aws/s3 cannot be used, because its policy can't be edited.

Real-Time Logs (Kinesis Data Streams Delivery)

Real-time logs deliver configurable access data within seconds to Amazon Kinesis Data Streams .

Requires an IAM role that CloudFront assumes to write records.

The IAM role trust policy must grant sts:AssumeRole to cloudfront.amazonaws.com .

The IAM permissions policy must grant kinesis:PutRecord and kinesis:PutRecords on the stream ARN.

If the Kinesis data stream uses SSE-KMS , the IAM role that CloudFront assumes must also be allowed to use that key (for example, kms:GenerateDataKey ).

Loading diagram... Enterprise Telemetry Ingestion Troubleshooting Decision Tree