Study guide
2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion
Key Takeaways Explain This Topic AI
S3 Batch Operations performs large-scale batch actions across billions of S3 objects, driven by an S3 Inventory report or CSV manifest file with an IAM service role.
A single PutObject request supports at most 5 GB, so larger objects require multipart upload; the current maximum S3 object size is 50 TB in supported commercial Regions.
Amazon S3 provides at least 3,500 PUT/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix and scales beyond those rates; gradual ramp-up and multiple well-distributed prefixes can reduce 503 Slow Down responses during scaling.
S3 Transfer Acceleration utilizes AWS CloudFront's globally distributed edge locations to route uploads over optimized AWS private network backbone, dramatically reducing latency for geographically distant clients.
AWS CLI bulk data transfers require configuring max_concurrent_requests, multipart_chunksize, and max_queue_size in the ~/.aws/config file to fully saturate available network bandwidth.
Last updated: August 2026
2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion
Quick Answer: Operating data lakes at scale requires specialized bulk ingestion and management primitives. S3 Batch Operations can execute one batch action across billions of objects defined by a manifest. For ingestion, single files over 5 GB must use S3 Multipart Upload . S3 provides at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix and scales beyond those baselines; clients should retry 503 Slow Down responses and add independent prefixes when a workload needs more parallel request capacity.
Amazon S3 Batch Operations Architecture
When managing millions or billions of objects in Amazon S3, standard API loops (e.g., iterating PutObjectTagging in code) are inefficient, error-prone, and rate-limited. S3 Batch Operations manages server-side parallel execution across massive object sets. +-----------------------------------------------------------------------------------+
- | S3 Batch Operations Workflow |
- | |
- | +-------------------+ +-----------------------+ +-----------------+ |
- | | Manifest File | ----> | S3 Batch Operations | ----> | Targeted Action | |
- | | (S3 Inventory / | | Job Engine | | - S3 Copy | |
- | | CSV Object List) | | (IAM Role Execution) | | - SSE-KMS Encrypt| |
- | +-------------------+ +-----------------------+ | - Lambda Invoke | |
- | | - Glacier Restore| |
- | +-----------------+ |
- +-----------------------------------------------------------------------------------+
- 1. Job Input (The Manifest)
S3 Batch Operations requires a manifest listing the target objects. Supported formats:
S3 Inventory Report: Generated automatically by S3 on a daily/weekly basis in CSV or Parquet format containing object keys, versions, size, and metadata.
Custom CSV File: A user-created CSV file formatted as Bucket, Key or Bucket, Key, VersionId .
2. Supported Native Operations
S3 Copy: Copy objects across buckets, accounts, Regions, or update storage classes. Copying an object onto a destination (including the same key) can apply SSE-KMS to an unencrypted object.
Update Object Encryption: Change supported encrypted objects from SSE-S3 to SSE-KMS, select a different customer managed KMS key, or enable S3 Bucket Keys without a copy. UpdateObjectEncryption does not support unencrypted, DSSE-KMS, or SSE-C objects; use a Batch Copy job when the source is unencrypted.
S3 Object Tagging: Replace or add key-value tags to billions of objects.
S3 Access Control Lists (ACLs): Modify object permissions.
S3 Glacier Restore: Initiate bulk restoration requests for archived objects.
S3 Object Lock: Apply or modify Legal Holds and Retention Periods for compliance.
AWS Lambda Invocation: Execute custom serverless logic per object (e.g., image resizing, validation, virus scanning, schema conversion).
3. Execution & Completion Reporting
Once configured with an IAM Service Role, S3 Batch Operations processes objects in parallel. Upon job completion, it writes a Completion Report manifest containing detailed success/failure records for every target key.
High-Throughput Ingestion Protocols: Multipart Uploads
When ingesting large datasets or giant files into Amazon S3, object limits dictate the ingestion protocol:
- Constraint / Limit Exact Value / Metric
- Minimum Object Size 0 Bytes
- Maximum Single PUT Upload 5 GB
- Maximum Total Object Size 50 TB
- Recommended Multipart Threshold Objects > 100 MB
- Mandatory Multipart Threshold Objects > 5 GB
- Part Count Range 1 to 10,000 Parts
- Part Size Range 5 MB to 5 GB (Last part can be < 5 MB)
- Multipart Upload Mechanics [Large 50 GB File]
- |--> Part 1 (500 MB) --> Parallel Upload thread 1 --|
- |--> Part 2 (500 MB) --> Parallel Upload thread 2 --|--> S3 Assembles Object
- |--> ... |--> (etag = hash-partcount)
- |--> Part 100 (500MB) --> Parallel Upload thread 10 -|
Key Advantages for Data Engineers:
Parallel Uploads: Saturates available network bandwidth by uploading multiple parts concurrently.
Fault Tolerance: If a single 500 MB part fails mid-upload due to network jitter, only that specific part is retried—not the entire 50 GB file.
Pause & Resume: Uploads can be initiated, paused, and completed over extended timeframes.
Amazon S3 Performance Scaling & Prefix Design Patterns
Amazon S3 automatically scales to support massive request rates, but application performance depends on prefix layout.
Key Metric: S3 supports 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per S3 Prefix .
What Defines an S3 Prefix?
An S3 prefix is any string between the bucket name and the object file name. s3://my-data-lake-bucket/raw/telemetry/2026/08/13/device_987.json
- |<-------- Bucket ------->|<----------- Prefix ----------->|<- Key ->|
- Sudden Single-Prefix Request Ramps
Sequential and date-based object names are supported; applications do not need to randomize ordinary S3 key names for routine performance. A newly created or abruptly heated prefix can still return temporary HTTP 503 Slow Down responses while S3 scales. Clients should use retries with exponential backoff and measure whether the workload actually exceeds the published per-prefix baseline.
Optional Parallel Prefixes for Exceptional Hot Workloads
When an ingestion burst needs more request parallelism than one prefix provides, distribute writes across multiple independent prefixes, for example with a bounded hash bucket. Preserve date components for query pruning, and record the bucket logic so readers can enumerate every location: s3://my-data-lake-bucket/logs/a1f8-2026/08/13/sensor_1.json --> Partition 1 (3,500 PUT/s)
- s3://my-data-lake-bucket/logs/b9c2-2026/08/13/sensor_2.json --> Partition 2 (3,500 PUT/s)
- s3://my-data-lake-bucket/logs/e3d4-2026/08/13/sensor_3.json --> Partition 3 (3,500 PUT/s)
- S3 Transfer Acceleration & Event-Driven Pipelines
- S3 Transfer Acceleration (S3TA)
Enables fast, secure transfers of files over long distances between client devices and an S3 bucket by routing traffic through AWS CloudFront Edge Locations .
Uses optimized network paths, TCP window size tuning, and AWS private backbone routing.
Use Case: Global mobile apps or remote office facilities uploading gigabytes of data to a centralized S3 bucket in us-east-1 across continents.
S3 Event Notifications vs. Amazon EventBridge
When objects land in S3, downstream processing jobs (Glue, Lambda, EMR) must be triggered automatically.
- Feature S3 Event Notifications Amazon EventBridge Integration
- Targets SQS, SNS, AWS Lambda directly 20+ AWS targets, Event Buses, Custom APIs
- Event Filtering Basic prefix/suffix matching (e.g. .parquet ) Advanced JSON payload pattern matching
- Delivery Guarantee At-least-once (can send duplicate events) At-least-once with Archive & Replay features
- Setup Bucket property configuration Enable S3 EventBridge integration on bucket
- AWS CLI Bulk Transfer Tuning Parameters
To optimize bulk data ingest using AWS CLI commands ( aws s3 sync or aws s3 cp ), tune the underlying AWS SDK parameters in ~/.aws/config : [default]
- s3 =
- max_concurrent_requests = 50
- multipart_chunksize = 64MB
- multipart_threshold = 64MB
- max_queue_size = 10000
max_concurrent_requests : Increases thread pool size (default is 10) to upload/download multiple parts in parallel.
multipart_chunksize : Configures chunk size for multipart uploads. Increasing chunk size reduces part management overhead for gigabyte files.