Study guide
11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies
Key Takeaways Explain This Topic AI
AWS KMS envelope encryption protects large payloads with local data keys while a KMS key protects the data-key ciphertext; the plaintext key should exist only in protected memory for the required operation.
Enabling Amazon S3 Bucket Keys can reduce KMS GenerateDataKey and Decrypt API calls by up to 99%, lowering cost and throttling risk; it does not eliminate every KMS request or the need for retries and quota monitoring.
Enforcement of data in transit encryption requires S3 Bucket Policies with an explicit Deny statement for requests where aws:SecureTransport evaluates to false.
SSE-KMS uses AWS KMS for key management and audit logging in CloudTrail, while SSE-S3 uses Amazon-managed keys with zero KMS API charges, and SSE-C offloads key management entirely to the client.
KMS Key Policies form the foundational access control layer for Customer managed KMS keys; without explicit delegation in the Key Policy, identity-based IAM policies cannot grant permissions to use the key.
Last updated: August 2026
Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies
Data encryption is a fundamental requirement for regulatory compliance (HIPAA, GDPR, PCI-DSS, SOC 2) and corporate data governance. In AWS data engineering pipelines, data must be protected both at rest (stored on disk in S3, Redshift, EBS, DynamoDB, EMR) and in transit (moving over the network between services). AWS Key Management Service (KMS) serves as the central security service for cryptographic key management.
AWS KMS Key Architecture & Envelope Encryption
AWS KMS manages symmetric (AES-256-GCM) and asymmetric cryptographic keys. KMS keys never leave AWS KMS unencrypted security modules (HSMs).
KMS Key Types
AWS Owned Keys : Created and managed by AWS services for internal service use. Free of charge, but cannot be viewed, audited, or managed directly by customers.
AWS Managed Keys : Created automatically by AWS on your behalf (e.g., aws/s3 , aws/redshift , aws/glue ). Free key storage, but API calls incur standard KMS charges. Automatically rotated every year; customers cannot change that schedule.
Customer managed KMS keys : Created, owned, and managed by you. Support custom key policies, explicit cross-account sharing, on-demand rotation, configurable automatic rotation periods for eligible symmetric keys, and deletion scheduling. Pricing varies by Region and key type.
Custom and external key stores : AWS CloudHSM custom key stores use customer-controlled HSMs in AWS, while external key stores connect KMS to supported key material outside AWS.
The Envelope Encryption Mechanics
Encrypting multi-gigabyte or terabyte dataset files directly via the KMS API ( kms:Encrypt ) is impossible because KMS APIs have a strict payload limit of 4 KB . Instead, AWS uses Envelope Encryption : [ KMS key ]
- |
- v (kms:GenerateDataKey)
- [ Plaintext Data Key (DEK) ] <---> Encrypts Large File Data locally
- |
- +--> Encrypted by KMS --> [ Encrypted DEK ] (Stored in file metadata)
The Encryption Process:
The application/service calls KMS API kms:GenerateDataKey supplying the KMS key ARN.
KMS returns two components: a Plaintext Data Encryption Key (DEK) and an Encrypted DEK (encrypted by the KMS key).
The application uses the Plaintext DEK in memory to encrypt the dataset locally (using AES-256).
The application erases the Plaintext DEK from memory .
The application saves the ciphertext file along with the Encrypted DEK stored in the file header or object metadata.
The Decryption Process:
The application reads the file and extracts the Encrypted DEK from metadata.
The application passes the Encrypted DEK to KMS via kms:Decrypt .
KMS decrypts the DEK using the KMS key and returns the Plaintext DEK to the application.
The application decrypts the dataset in memory and wipes the Plaintext DEK.
Amazon S3 Encryption Options & Bucket Keys
Amazon S3 supports multiple server-side and client-side encryption options:
Encryption Type Key Management KMS API Charges? Key Audit in CloudTrail?
- SSE-S3 (Default) Managed fully by Amazon S3 (AES-256) No No
- SSE-KMS Managed via AWS KMS (AWS Managed or CMK) Yes (per object call) Yes (Detailed logs)
- DSSE-KMS Dual-layer envelope encryption via KMS Yes (2x calls) Yes
- SSE-C Customer provides raw key in HTTP headers No No
- Client-Side Encrypted prior to upload using AWS Encryption SDK Yes (if KMS used) Yes
- Optimizing Costs & Performance: Amazon S3 Bucket Keys
When running massive EMR, Glue, or Athena queries that read/write millions of S3 objects encrypted with SSE-KMS, default SSE-KMS behavior executes a kms:GenerateDataKey or kms:Decrypt call for every single S3 object . This rapidly leads to:
KMS request cost : per-request rates vary by Region and key type.
KMS throttling : exceeding the applicable request quota can cause ThrottlingException ; monitor Service Quotas and request metrics for the selected Region.
Solution: S3 Bucket Keys ( s3:BucketKeyEnabled = true ) When S3 Bucket Keys are enabled on a bucket, S3 requests a short-lived bucket-level Data Key from KMS. S3 uses this key to derive DEKs for new objects written to the bucket. This can reduce KMS API calls by up to 99% , cutting request costs and substantially reducing KMS throttling pressure. It does not remove every KMS call or guarantee that throttling is impossible, so retain retries and quota monitoring.
KMS Key Policies vs IAM Policies
Unlike most AWS services where an IAM policy is sufficient to grant access, KMS operates under a dual-authorization model:
CRITICAL EXAM PRINCIPLE : A customer managed KMS key cannot be used unless its KMS Key Policy explicitly allows the principal or explicitly delegates access control to the account's IAM policies!
To allow identity-based IAM policies to manage access to a KMS key, the Key Policy MUST contain the default root delegation statement: {
- "Sid": "EnableIAMUserPermissions",
- "Effect": "Allow",
- "Principal": {
- "AWS": "arn:aws:iam::123456789012:root"
- },
- "Action": "kms:*",
- "Resource": "*"
- }
- Enforcing Encryption via S3 Bucket Policies
To guarantee regulatory compliance, data engineers must enforce two common bucket-policy controls when policy requires them:
Enforce Encryption in Transit (HTTPS / TLS) : Block unencrypted HTTP connections ( aws:SecureTransport: false ).
Enforce Specific KMS Encryption Key : Reject uploads ( s3:PutObject ) that do not use SSE-KMS with an approved CMK. {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "EnforceTLSInTransit",
- "Effect": "Deny",
- "Principal": "*",
- "Action": "s3:*",
- "Resource": [
- "arn:aws:s3:::finance-data-lake-prod",
- "arn:aws:s3:::finance-data-lake-prod/*"
- ],
- "Condition": {
- "Bool": {
- "aws:SecureTransport": "false"
- }
- }
- },
- {
- "Sid": "EnforceApprovedKMSEncryption",
- "Effect": "Deny",
- "Principal": "*",
- "Action": "s3:PutObject",
- "Resource": "arn:aws:s3:::finance-data-lake-prod/*",
- "Condition": {
- "StringNotEquals": {
- "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-east-1:123456789012:key/abc-123-def-456"
- }
- }
- }
- ]
- }
- Summary Checklist for DEA-C01 Exam
KMS Payload Limit : 4 KB (requires Envelope Encryption for data files).
S3 Bucket Keys : Reduces KMS API requests by ~99% for SSE-KMS data lakes.
Enforce TLS : Set Effect: Deny when aws:SecureTransport is false .
Cross-Account KMS : Must update BOTH the KMS Key Policy in the key account and the IAM Policy in the calling account.