Study guide
8.3 Centralized Audit Logging with AWS CloudTrail
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
8.3 Centralized Audit Logging with AWS CloudTrail
Key Takeaways Explain This Topic AI
AWS CloudTrail records supported AWS API and account activity according to event type and configuration, providing identity, source, time, and request context for audit and investigation.
Management Events capture control-plane operations (e.g., Glue CreateJob, Redshift ModifyCluster), while Data Events record high-volume data-plane operations (e.g., S3 GetObject/PutObject, Lambda Invoke).
AWS Organizations integration allows creation of an Organization Trail that automatically aggregates audit logs across all member accounts into a single KMS-encrypted S3 bucket in a dedicated Log Archive account.
CloudTrail log file integrity validation uses SHA-256 hashes and signed digest files to detect modification or deletion in the delivered log chain; prevention and immutable retention require separate S3 controls.
CloudTrail Lake provides managed SQL analysis with configurable multi-year retention; select the retention period required by policy and supported by the current event data store settings.
Last updated: August 2026
8.3 Centralized Audit Logging with AWS CloudTrail
While CloudWatch provides visibility into application metrics and operational execution logs, AWS CloudTrail provides governance, compliance, and operational auditing across the entire AWS API ecosystem. CloudTrail records supported API activity selected by a trail or event data store—whether invoked via the console, SDKs, CLI, or service integrations—along with fields such as identity, time, source IP, request parameters, and response information. Management events are included by default in a trail; high-volume data events must be selected explicitly.
For data engineers, CloudTrail is critical for auditing data access, tracking infrastructure changes, enforcing security compliance, and performing post-incident forensic investigations on data lakes and processing pipelines.
- CloudTrail Event Categories
- CloudTrail categorizes API activity into three distinct event types: ┌─────────────────────────┐
- │ AWS CloudTrail │
- └────────────┬────────────┘
- │
- ┌──────────────────────────────────┼──────────────────────────────────┐
- ▼ ▼ ▼
- [ Management Events ] [ Data Events ] [ Insight Events ]
- (Control Plane) (Data Plane) (Anomaly Detection)
- CreateBucket - S3 GetObject / PutObject - API Call Spikes
- StartJobRun (Glue) - Lambda Invoke - Rate Limit Throttling
- ModifyCluster (Redshift) - DynamoDB GetItem - Unusual User Activity
- 1. Management Events (Control Plane Operations)
Management events capture operations performed on control-plane resources. Examples include:
Creating or deleting S3 buckets ( CreateBucket , DeleteBucket ).
Configuring IAM roles, policies, and permissions ( AttachRolePolicy ).
Modifying database cluster configurations ( ModifyCluster in Redshift).
Launching or stopping ETL jobs ( StartJobRun , BatchStopJobRun in AWS Glue).
Default State : CloudTrail automatically records Management Events across all services free of charge in the Event History (90-day retention).
2. Data Events (Data Plane Operations)
Data events record high-volume resource operations executed within or against underlying data assets. Examples include:
Amazon S3 object-level operations ( GetObject , PutObject , DeleteObject ).
AWS Lambda function invocations ( Invoke ).
Amazon DynamoDB item-level operations ( GetItem , PutItem ).
Operational Significance : Data events generate massive event volumes. They are disabled by default to prevent unexpected storage costs. Data engineers must selectively enable Data Events on specific security-sensitive S3 data lake prefixes or Lambda functions.
3. CloudTrail Insights Events
Insights events analyze CloudTrail Management Events using machine learning baselines to detect unusual operational behavior. CloudTrail automatically alerts when it detects anomalous patterns such as:
Sudden spikes in API call volume (e.g., unexpected burst of DeleteBucket or StopJobRun calls).
Widespread API throttling errors ( RateExceeded ).
Centralized Multi-Account Audit Architecture
In enterprise environment architectures using AWS Organizations , security best practices mandate centralizing all audit logs into a single, dedicated Log Archive Account isolated from workload development and production accounts. [ Account A (Dev) ] ───┐
- ├──> [ Organization Trail ] ──> [ Central Log Archive Account ]
- [ Account B (Prod) ] ───┤ │
- │ ├──> KMS key Encrypted S3 Bucket
- [ Account C (Data) ] ───┘ └──> SHA-256 Digest Validation
Architectural Components of Centralized Logging:
Organization Trail : Configured in the AWS Organizations management account (or delegated administrator account). Creating an Organization Trail automatically creates matching trails in all child accounts across the organization. Member accounts cannot delete or disable the organization trail.
Central S3 Log Bucket : A hardened S3 bucket located in the Log Archive account. The bucket policy grants cloudtrail.amazonaws.com write access via s3:PutObject with a condition requiring aws:SourceOrgID enforcement to prevent unauthorized cross-account log injection.
KMS Key Encryption : CloudTrail encrypts log files using Server-Side Encryption with AWS KMS customer managed KMS keys (SSE-KMS). The KMS key policy must grant CloudTrail permissions ( kms:GenerateDataKey* , kms:DescribeKey ) and allow cross-account decryption access strictly for authorized compliance auditors.
Log File Integrity Validation : To prove audit logs have not been tampered with or modified after creation, CloudTrail log file integrity validation is enabled. CloudTrail delivers cryptographic digest files containing SHA-256 hashes of all delivered log files, signed using RSA private keys. Auditors verify digest signatures to detect any log deletion or alteration.
Querying CloudTrail Logs for Audit Analysis
Raw CloudTrail logs are delivered to S3 as gzipped JSON files structured by account, region, date, and hour. Analyzing audit logs requires dedicated query engines.
Option 1: Querying via Amazon Athena
Data engineers can create an external partitioned Athena table over the S3 CloudTrail log bucket to execute ad-hoc SQL queries.
- DDL Schema Definition: CREATE EXTERNAL TABLE cloudtrail_logs (
- eventVersion STRING,
- userIdentity STRUCT<
- type: STRING,
- principalId: STRING,
- arn: STRING,
- accountId: STRING,
- sessionContext: STRUCT<
- attributes: STRUCT<mfaAuthenticated: STRING, creationDate: STRING>,
- sessionIssuer: STRUCT<type: STRING, principalId: STRING, arn: STRING, userName: STRING>
- >
- >,
- eventTime STRING,
- eventSource STRING,
- eventName STRING,
- awsRegion STRING,
- sourceIPAddress STRING,
- userAgent STRING,
- errorCode STRING,
- errorMessage STRING,
- requestParameters STRING,
- responseElements STRING
- )
- PARTITIONED BY (region STRING, year STRING, month STRING, day STRING)
- ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
- LOCATION 's3://central-log-archive-bucket/AWSLogs/o-1234567890/';
- Security Analysis Query 1: Identifying Unauthorized Data Access Attempts SELECT
- eventTime,
- userIdentity.arn AS Principal,
- sourceIPAddress,
- eventName,
- errorCode,
- errorMessage
- FROM cloudtrail_logs
- WHERE eventSource = 's3.amazonaws.com'
- AND errorCode IN ('AccessDenied', 'AllAccessDisabled')
- AND year = '2026' AND month = '08'
- ORDER BY eventTime DESC
- LIMIT 50;
Explanation : Identifies unauthorized attempts to read or modify S3 data lake assets, revealing potential permission misconfigurations or compromised IAM principals.
- Security Analysis Query 2: Tracking AWS Glue Job Configuration Modifications SELECT
- eventTime,
- userIdentity.arn AS ModifyingUser,
- sourceIPAddress,
- eventName,
- json_extract_scalar(requestParameters, '$.jobName') AS JobName
- FROM cloudtrail_logs
- WHERE eventSource = 'glue.amazonaws.com'
- AND eventName IN ('UpdateJob', 'DeleteJob', 'BatchStopJobRun')
- AND year = '2026';
Explanation : Audits administrative modifications or manual terminations of production ETL pipelines.
Option 2: CloudTrail Lake
CloudTrail Lake is a managed security data lake that allows data teams to run immutable SQL queries directly over audit events without managing S3 storage, Glue Crawlers, or Athena schemas.
Feature CloudTrail + Athena CloudTrail Lake
Setup & Maintenance Requires manually creating S3 buckets, Glue crawlers, and Athena partition projection schema DDLs. Fully managed; create an Event Data Store (EDS) with one click.
Data Retention Dependent on S3 Lifecycle Rules. Event data stores can use fixed retention up to 2,557 days or extendable retention up to 3,653 days.
Query Engine Standard Athena Presto/Trino engine querying raw JSON log files. Managed, optimized SQL engine built for event structures.
Multi-Account Aggregation Requires complex multi-account S3 bucket policies and cross-account KMS permissions. Built-in native support for AWS Organizations Event Data Stores.