Study guide
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization
Key Takeaways Explain This Topic AI
S3 storage classes trade access latency, Availability Zone design, minimum storage duration, minimum billable size, and retrieval charges; use the current Regional pricing page rather than memorizing one Region's rates.
S3 Intelligent-Tiering has three automatic low-latency tiers and two optional asynchronous archive tiers; objects smaller than 128 KB remain in Frequent Access and are not charged monitoring fees.
Since September 2024, new S3 Lifecycle configurations do not transition objects smaller than 128 KB by default to any storage class; an explicit object-size filter can override that default where economics and target-class rules permit.
S3 Lifecycle rule actions can handle noncurrent object versions, expired object delete markers, incomplete multipart uploads (aborting after N days to prevent hidden storage costs), and S3 Storage Lens provides organization-wide visibility into cost optimization opportunities.
Last updated: October 2026
- 5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization
- Fundamentals of Amazon S3 Storage Hierarchy
Amazon Simple Storage Service (Amazon S3) is the primary object storage service for building scalable, high-durability data lakes on AWS. S3 offers 99.999999999% (11 9s) of data durability across all standard storage classes by redundantly storing objects across a minimum of three Availability Zones (AZs) within an AWS Region (with the exception of single-AZ classes). However, data access patterns, retrieval latency requirements, and cost profiles vary significantly throughout the lifecycle of data lake assets.
To optimize storage expenditure without sacrificing operational performance, AWS provides a hierarchy of S3 storage classes designed for specific access patterns:
Storage Class Target Access Pattern Min Storage Duration Min Billable Size First-Byte Latency Retrieval Fee Availability SLA
- S3 Standard Active, frequently accessed data None None Milliseconds None 99.99%
- S3 Express One Zone High-performance, single-AZ directory buckets None None Single-digit ms None 99.9%
S3 Intelligent-Tiering Unknown or changing access patterns None Objects <128 KB stay in Frequent Access Milliseconds for automatic tiers None 99.9%
- S3 Standard-IA Infrequently accessed data, immediate access 30 days 128 KB Milliseconds Per GB 99.9%
- S3 One Zone-IA Infrequently accessed non-critical data 30 days 128 KB Milliseconds Per GB 99.5%
- S3 Glacier Instant Retrieval Archive data accessed 1–2 times per year 90 days 128 KB Milliseconds Per GB 99.9%
S3 Glacier Flexible Archive Archive data with flexible retrieval options 90 days 40 KB 1 min – 12 hrs Per GB 99.9%
- S3 Glacier Deep Archive Long-term archival, compliance & backup 180 days 40 KB 12 hrs – 48 hrs Per GB 99.9%
- S3 Intelligent-Tiering Deep Dive
For data lakes with unpredictable, dynamic, or unknown access patterns, S3 Intelligent-Tiering delivers automatic cost savings by moving objects between granular access tiers based on monitoring access frequency. Unlike traditional storage classes, Intelligent-Tiering incurs no retrieval fees .
S3 Intelligent-Tiering operates across five distinct access tiers:
Frequent Access Tier (Automated) : Default tier for ingested objects. Pricing matches S3 Standard.
Infrequent Access Tier (Automated) : Objects not accessed for 30 consecutive days automatically shift to this tier, yielding ~40% cost savings compared to Standard.
Archive Instant Access Tier (Automated) : Objects not accessed for 90 consecutive days shift to this tier, yielding ~68% cost savings with millisecond retrieval latency.
Archive Access Tier (Opt-in) : Objects can enter this optional tier after a configured period beginning at 90 days. Retrieval is asynchronous, with expedited, standard, and bulk times depending on the retrieval choice.
Deep Archive Access Tier (Opt-in) : Objects can enter this optional tier after a configured period beginning at 180 days. Retrieval is asynchronous and can take hours.
Operational Constraints & Guardrails
Monitoring Fee : A per-object monitoring and automation fee applies; its amount varies by Region and current pricing.
Minimum Object Size : Objects smaller than 128 KB are retained in the Frequent Access tier and are not monitored or moved to lower cost tiers.
Minimum Billing Duration : Intelligent-Tiering itself has no minimum storage-duration charge; optional archive access and early deletion behavior must still be checked against current class rules.
S3 Lifecycle Policies & Transition Mechanics
An S3 Lifecycle Configuration is a set of rules containing actions that S3 applies to a group of objects. Lifecycle configurations enable automated data transition to lower-cost storage classes or automated deletion (expiration) as data ages.
Supported Transition Pathways
Lifecycle rules move objects only to supported colder classes; restoring an archived object is a separate operation and does not change its storage class. A rule can often transition directly from S3 Standard to an eligible colder class without visiting every intermediate class, subject to the source class, object age, minimum-duration charges, and current transition constraints.
Lifecycle Rule Elements & Filter Parameters
Rules are defined in XML/JSON format using the following key elements:
Prefix / Tag Filters : Limits rule execution to objects with matching key prefixes (e.g., logs/year=2025/ ) or specific metadata tags ( environment=production ).
Object Size Filters : Configures ObjectSizeGreaterThan and ObjectSizeLessThan parameters (e.g., preventing objects under 128 KB from transitioning to Standard-IA).
Transitions : Specifies Days or Date after object creation when transition to a target storage class occurs (e.g., transition to Standard-IA after 30 days, then to Glacier Deep Archive after 90 days).
Expiration : Specifies when objects are permanently deleted or when expired object delete markers are purged.
Managing Versioned Buckets
In buckets with S3 Versioning enabled, lifecycle rules manage current and noncurrent versions independently:
NoncurrentVersionTransitions : Moves previous object versions to cost-effective storage classes after a specified number of days.
NoncurrentVersionExpiration : Permanently deletes noncurrent versions after a retention window expires.
ExpiredObjectDeleteMarkers : Automatically removes delete markers that no longer have noncurrent object versions behind them.
AbortIncompleteMultipartUpload : Automatically purges uncommitted upload parts after a set duration (e.g., 7 days). Failing to configure this action is a leading cause of unexpected storage costs.
- S3 Cost Optimization Tooling & Best Practices
- S3 Storage Lens
S3 Storage Lens provides organization-wide visibility into object storage usage, cost optimization opportunities, and data protection metrics. Storage Lens offers interactive dashboards in the AWS Management Console with over 35 metrics aggregated at the organization, account, region, bucket, or prefix level. Key optimization metrics include:
Percentage of unencrypted objects.
Incomplete multipart upload storage volume.
Noncurrent version storage accumulation.
Percentage of eligible data candidates for S3 Intelligent-Tiering or Glacier.
- Code Example: S3 Lifecycle Configuration in JSON {
- "Rules": [
- {
- "ID": "DataLakeLifecycleAndCleanupRule",
- "Status": "Enabled",
- "Filter": {
- "And": {
- "Prefix": "raw-telemetry/",
- "ObjectSizeGreaterThan": 131072
- }
- },
- "Transitions": [
- {
- "Days": 30,
- "StorageClass": "STANDARD_IA"
- },
- {
- "Days": 90,
- "StorageClass": "GLACIER_IR"
- },
- {
- "Days": 180,
- "StorageClass": "DEEP_ARCHIVE"
- }
- ],
- "NoncurrentVersionTransitions": [
- {
- "NoncurrentDays": 14,
- "StorageClass": "GLACIER_IR"
- }
- ],
- "NoncurrentVersionExpiration": {
- "NoncurrentDays": 60
- },
- "AbortIncompleteMultipartUpload": {
- "DaysAfterInitiation": 7
- }
- }
- ]
- }