Study guide

2.2 Amazon Security Lake & OCSF Log Normalization

AWS Certified Security – Specialty SCS-C03

Track your progress as you read

Finished sections get a check mark.

1 Chapter 0: Getting Started with the SCS-C03 Exam

0.1 SCS-C03 Exam Format, Domains & Study Strategy

2 Chapter 1: Security Monitoring, Alerting & Threat Detection Services

1.1 Workload Monitoring & Health Check Strategies 1.2 Amazon GuardDuty Architecture & Finding Types 1.3 AWS Security Hub & Automated Conformance Assessment 1.4 Amazon Macie & Automated Data Classification

3 Chapter 2: Enterprise Logging Architecture & Analysis

2.1 Multi-Account CloudTrail & Organization Trails 2.2 Amazon Security Lake & OCSF Log Normalization 2.3 Network Telemetry: VPC Flow Logs & Route 53 Resolver Logs 2.4 Troubleshooting Security Telemetry & Log Ingestion 2.5 Analyzing, Normalizing & Correlating Security Logs

4 Chapter 3: Incident Readiness, Response Planning & Automated Runbooks

3.1 Incident Response Planning & Blast Radius Mitigation 3.2 Testing & Validating Incident Response Plans 3.3 Automated Incident Remediation with EventBridge & Step Functions 3.4 Systems Manager OpsCenter & Incident Manager Runbooks

5 Chapter 4: Digital Forensics, Evidence Preservation & Root Cause Analysis

4.1 Forensic Artifact Capture & Chain of Custody 4.2 Threat Containment & Network Quarantine Techniques 4.3 Correlating Events & Root Cause Analysis with Amazon Detective

6 Chapter 5: Edge Protection, WAF & DDoS Mitigation

5.1 CloudFront Security Headers & Origin Access Control 5.2 AWS WAF Architecture & Advanced Rule Sets 5.3 AWS Shield Advanced & DDoS Defense Architectures 5.4 Edge Integrations: Managed Rule Groups, OCSF Ingestion, IoT Policies & CORS

7 Chapter 6: Compute Workload Hardening, Vulnerability Management & GenAI Security

6.1 Golden AMIs & Container Image Hardening Pipelines 6.2 Amazon Inspector Vulnerability Management & Patch Automation 6.3 Generative AI Security: Amazon Bedrock Guardrails & Model Protection 6.4 Instance Profiles, Service Roles & Execution Roles for Compute 6.5 Secure Administrative Access: Session Manager & EC2 Instance Connect

8 Chapter 7: Network Security, Segmentation, Private Connectivity & Encryption in Transit

7.1 Layered Traffic Filtering: SGs, NACLs & AWS Network Firewall 7.2 Hybrid Connectivity Security: VPN, Direct Connect & MACsec 7.3 Zero-Trust Access & Network Access Analysis 7.4 Enforcing Encryption in Transit: ELB Security Policies, TLS & HTTPS-Only Access 7.5 Private Access: VPC Endpoints, PrivateLink, Client VPN & Verified Access 7.6 Inter-Resource Encryption in Transit: Nitro, EMR, EKS & SageMaker AI

9 Chapter 8: Enterprise Authentication, Federation & Directory Services

8.1 AWS IAM Identity Center & Enterprise IdP Integration 8.2 Amazon Cognito User Pools & Identity Pools 8.3 AWS Security Token Service (STS) & Temporary Credential Workflows 8.4 Troubleshooting Identity & Authentication Failures

10 Chapter 9: Advanced Authorization Architecture & Policy Evaluation

9.1 Complete AWS Policy Evaluation Logic & Decision Flow 9.2 Permission Boundaries & Safe Admin Delegation 9.3 Attribute-Based Access Control (ABAC) with IAM Tags

11 Chapter 10: Least Privilege, Policy Validation & External Access Auditing

10.1 IAM Roles Anywhere & Amazon Verified Permissions 10.2 Auditing External & Unused Access with IAM Access Analyzer 10.3 Analyzing Authorization Failures & Policy Simulation

12 Chapter 11: Cryptographic Services: KMS Architecture & Key Governance

11.1 AWS KMS Architecture, Key Policies & Key Types 11.2 KMS Grants, Encryption Context & Multi-Region Keys 11.3 Imported Key Material (BYOK) & CloudHSM Key Stores

13 Chapter 12: Secrets Management, Sensitive Data Masking & PKI

12.1 AWS Secrets Manager & Systems Manager Parameter Store 12.2 Sensitive Data Masking in Telemetry & Messaging 12.3 AWS Certificate Manager (ACM) & Private CA Architecture

14 Chapter 13: Storage Encryption, Integrity, Lifecycle & Backup Protection

13.1 S3 Encryption Modes & S3 Bucket Keys 13.2 Data Integrity Protection: Object Lock, Vault Lock & Code Signing 13.3 Secure Data Lifecycle, Cross-Account Replication & Backup Vault Lock 13.4 Lifecycle, Backup & Secure Replication Beyond S3: EFS, FSx, DLM, DataSync & Air-Gapped Vaults

15 Chapter 14: Multi-Account Architecture, Organizations & Control Tower Governance

14.1 AWS Organizations Structure, Hierarchy & Resource Control 14.2 AWS Control Tower Landing Zone & Guardrail Framework 14.3 Delegated Administrator Architecture & Root User Governance 14.4 Centralized Resource Sharing with RAM & Service Catalog

16 Chapter 15: Compliance Automation, Infrastructure as Code Security & Auditing

15.1 Infrastructure as Code Security & Multi-Account Deployment 15.2 AWS Config, Conformance Packs & Automated Remediation 15.3 Audit Evidence Collection & Well-Architected Security Reviews

2.2 Amazon Security Lake & OCSF Log Normalization

Key Takeaways Explain This Topic AI

Amazon Security Lake automatically aggregates and normalizes security telemetry across multi-account, multi-region environments into the Open Cybersecurity Schema Framework (OCSF).

Native sources (CloudTrail management and S3/Lambda data events, EKS audit logs, Route 53 Resolver query logs, Security Hub CSPM findings, VPC Flow Logs, and AWS WAF logs) are normalized to OCSF and stored as partitioned Apache Parquet.

Regional roll-up architecture enables multi-region global organizations to consolidate security telemetry into designated target regions, reducing data transfer costs while maintaining centralized visibility.

Security Lake manages AWS Lake Formation permissions and AWS Glue Data Catalog tables, empowering security analysts to execute federated SQL queries in Amazon Athena without data transformation overhead.

Third-party SIEM and analytics platforms integrate as subscribers via two architectural patterns: query-based access using Lake Formation or direct data access via SQS event notifications.

Last updated: September 2026

The Enterprise Telemetry Fragmentation Problem

In large-scale enterprise environments spanning dozens of AWS accounts, multiple regions, and hybrid on-premises data centers, security teams confront severe telemetry fragmentation . Each logging system produces records in disparate, proprietary formats:

CloudTrail emits nested JSON structures.

VPC Flow Logs emit space-delimited text strings.

Route 53 Resolver logs emit distinct JSON schemas.

GuardDuty findings use the Amazon GuardDuty finding schema.

Security Hub findings use the AWS Security Finding Format (ASFF).

On-premises firewalls and identity providers generate syslog, CEF, or custom key-value pairs.

Historically, Security Operations Center (SOC) teams had to build, maintain, and scale complex Extract, Transform, Load (ETL) pipelines using AWS Lambda, Amazon Kinesis, and AWS Glue to parse, re-index, and normalize these streams before ingesting them into a Security Information and Event Management (SIEM) tool. These pipelines introduce significant compute costs, processing latency, and operational fragility.

Amazon Security Lake solves this challenge by automatically centralizing security data from multiple AWS accounts and regions into a purpose-built security data lake, normalizing all telemetry into the open-source Open Cybersecurity Schema Framework (OCSF) .

The Open Cybersecurity Schema Framework (OCSF)

OCSF is an open-source, vendor-agnostic standard initiated by AWS and leading cybersecurity vendors. It defines a standardized, hierarchical taxonomy for security events, eliminating the need for custom schema parsing. [OCSF Root]

  • └── Categories (e.g., Network Activity, System Activity, Finding)
  • └── Event Classes (e.g., DNS Activity, Network Activity, Security Finding)
  • └── Standardized Attributes (e.g., src_endpoint, dst_endpoint, actor, severity_id)
  • Core OCSF Concepts

Categories : High-level groupings of event domains (e.g., Category 1: System Activity, Category 2: Findings, Category 3: Identity & Access Management, Category 4: Network Activity).

Event Classes : Specific event implementations within a category. For example, Event Class 4001: Network Activity represents connection flows, while Event Class 4003: DNS Activity represents DNS queries and resolutions.

Standardized Data Types and Dictionaries : OCSF standardizes IP addresses ( ip ), port numbers ( port ), MAC addresses ( mac ), hostnames ( hostname ), and status codes. For example, whether a connection was allowed or denied is mapped to standard disposition_id integers (e.g., 1: Allowed , 2: Blocked ).

Unified Severity : Threat severity across different security products is normalized to a severity_id integer (1 Informational, 2 Low, 3 Medium, 4 High, 5 Critical, 6 Fatal, plus 0 Unknown and 99 Other), enabling consistent correlation across GuardDuty, third-party endpoint detection and response (EDR) agents, and network firewalls.

Amazon Security Lake Core Architecture

Security Lake is deployed across an entire AWS Organization via a designated Delegated Administrator Account (typically the enterprise Security Tooling or Central Security account). [ AWS Accounts & Regions ]

  • ├── CloudTrail Management & Data Events
  • ├── VPC Flow Logs
  • ├── Route 53 Resolver Query Logs === (Native Extraction) ===> [ Amazon Security Lake ]
  • ├── Amazon GuardDuty Findings │
  • └── AWS Security Hub Findings ▼
  • [ Normalization Engine ]
  • │
  • ▼
  • [ Apache Parquet in S3 ]
  • (Partitioned by Source,
  • Region, Account, Date)
  • Automatically Collected Native Sources

When enabled, Security Lake collects its natively supported AWS sources without agents or custom Lambda triggers:

  • AWS CloudTrail Management Events
  • AWS CloudTrail Data Events (for Amazon S3 and AWS Lambda)
  • Amazon VPC Flow Logs
  • Amazon Route 53 Resolver Query Logs

AWS Security Hub CSPM Findings (which carry GuardDuty, Inspector, Macie, and IAM Access Analyzer findings, converted from ASFF to OCSF)

  • Amazon EKS Audit Logs
  • AWS WAFv2 Logs (web ACL traffic logs, which is how edge telemetry reaches an OCSF data lake)
  • Storage Mechanics: Apache Parquet & Partitioning

Security Lake stores all ingested and normalized data in Amazon S3 using Apache Parquet format. Parquet is an open-source, columnar storage format optimized for analytical queries:

Columnar Layout : Queries that examine specific columns (such as src_endpoint.ip or actor.user.name ) only read the relevant columns from disk, avoiding costly full-row table scans.

Compression : Parquet is a compressed columnar format, so the stored volume is far smaller than the equivalent raw JSON.

Deterministic Partitioning : Data is partitioned by source, Region, account ID, and event day, so object prefixes end in a pattern such as region=us-east-1/accountId=111122223333/eventDay=20260929/ (custom sources are written under an ext/ prefix).

Partition pruning allows analytical query engines to scan only the exact temporal and structural slices required.

Multi-Region Roll-Up Architecture

Global enterprises operating across multiple AWS regions face architectural trade-offs between centralized visibility and cross-region data transfer costs. Security Lake resolves this via Regional Roll-ups .

Roll-Up Mechanics

Contributing Regions : Regions where workloads run and telemetry originates (e.g., eu-west-1 , ap-southeast-1 , us-west-2 ).

Roll-up Regions : Designated target regions (e.g., us-east-1 ) that automatically receive replicated security data from contributing regions.

Security Lake uses Amazon S3 cross-region replication under the hood to consolidate telemetry into the roll-up region. This enables security analysts and SIEM systems to execute queries against a single regional endpoint while complying with corporate retention and cross-region egress budgets.

Exam Tip : Data residency compliance laws (such as GDPR in Europe) may prohibit transferring security logs outside their originating jurisdiction. In such scenarios, do NOT configure roll-up replication for regulated regions; leave them as standalone regional data stores and query them locally.

Querying Security Lake via AWS Lake Formation & Amazon Athena

Security Lake registers all normalized Parquet tables in the AWS Glue Data Catalog and enforces data access controls through AWS Lake Formation .

Security analysts can immediately run standard SQL queries in Amazon Athena against OCSF tables without configuring table DDLs or repair scripts: -- Investigate high-volume blocked outbound traffic across all VPCs

  • SELECT
  • time_dt,
  • src_endpoint.ip AS source_ip,
  • src_endpoint.port AS source_port,
  • dst_endpoint.ip AS destination_ip,
  • dst_endpoint.port AS destination_port,
  • traffic.bytes AS bytes_transferred,
  • disposition
  • FROM
  • amazon_security_lake_glue_db_us_east_1.amazon_security_lake_table_us_east_1_vpc_flow_2_0
  • WHERE
  • disposition = 'Blocked'
  • AND traffic.bytes > 10000000
  • AND eventDay = '20260929'
  • ORDER BY
  • traffic.bytes DESC
  • LIMIT 100;

Because the underlying data is Parquet, Athena scans only the requested columns and matching date partitions, dramatically accelerating query execution while minimizing query costs (charged at $5.00 per TB scanned).

Subscriber Integration Models: Query Access vs. Data Access

Security Lake decouples log collection from security consumption. Downstream consumers (e.g., Splunk, Datadog, Snowflake, OpenSearch, internal SOC tools) register as Subscribers .

Security Lake supports two subscriber consumption models:

Subscriber Access Model Architectural Flow Primary Use Case Security & Authorization Mechanism

Query Access The subscriber queries data in place using Amazon Athena or Redshift Spectrum. No data is copied out of S3. Interactive threat hunting, ad-hoc forensics, BI dashboards, Snowflake federated queries. AWS Lake Formation grants fine-grained, column-level and row-level permissions to the subscriber's IAM principal.

Data Access Security Lake sends S3 event notifications via Amazon SQS when new Parquet files are written. The subscriber pulls objects directly from S3. Ingestion into external SIEMs (Splunk Enterprise, Datadog, Sumo Logic) that require local index storage. Security Lake configures an Amazon SQS queue and S3 bucket policy allowing cross-account IAM role assumption with an sts:ExternalId . [ Security Lake S3 Bucket ]

  • │
  • ├── (Direct Query via Athena) ─────────> [ Query Subscriber (Snowflake / BI) ]
  • │ (Lake Formation Permissions)
  • │
  • ├── New Parquet Object Created
  • │ │
  • │ ▼
  • ├──> [ Amazon SQS Queue ] ─────────────> [ Data Subscriber (Splunk / Datadog) ]
  • │ │ (Polls SQS, assumes cross-account
  • └────────┴── Fetches Parquet File ──────── IAM role with ExternalId)
  • Ingesting Custom & Third-Party Log Sources

Enterprise security requires holistic visibility across cloud and on-premises infrastructure. Security Lake supports ingesting Custom Sources (such as Palo Alto firewall logs, Okta authentication logs, or bespoke container application logs):

Define Source : The administrator registers a custom source in Security Lake, specifying the source name and OCSF event class.

S3 Ingestion Location : Security Lake provisions a dedicated S3 bucket prefix and an IAM execution role.

Transformation : The customer's ingestion pipeline (e.g., AWS Lambda, AWS Glue ETL, or Amazon Data Firehose) transforms raw log events into OCSF-compliant Apache Parquet format and writes the objects to the custom source prefix.

Cataloging : Security Lake invokes an AWS Glue Crawler to update partition definitions in the Glue Data Catalog, making custom data instantly queryable alongside native AWS telemetry.

Loading diagram... Amazon Security Lake Multi-Region Ingestion & Subscriber Architecture