Study guide
4.5 Containers, Lambda & Distributed Processing Performance
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
4.5 Containers, Lambda & Distributed Processing Performance
Key Takeaways Explain This Topic AI
Choose Lambda for short event-driven work and ECS or EKS for containerized workloads that need longer execution, custom runtimes, daemon processes, or explicit CPU and memory control.
Lambda performance depends on memory-linked CPU, concurrency, initialization, batching, ephemeral storage, and downstream capacity; reserved concurrency can protect both the function and its dependencies.
Lambda can mount EFS for shared persistent files, while configurable /tmp ephemeral storage is local to one execution environment and must not be treated as a durable shared store.
Distributed performance is bounded by partition balance, shuffles, serialization, network, and stragglers; increasing workers cannot repair a single hot key or driver-side collect operation.
Last updated: August 2026
4.5 Containers, Lambda & Distributed Processing Performance
The correct compute service follows workload shape. Lambda is an event-driven function runtime. Amazon ECS and Amazon EKS run containerized applications with longer lifetimes and more control. EMR and Glue provide distributed data frameworks. Selecting compute by language alone misses the operational constraints the exam tests.
Start with volume, velocity, and variety
Characterize the input before selecting compute. Volume is the amount retained or processed per run; it drives storage, scan bytes, partition count, and whether work must be distributed. Velocity is the arrival and change rate; it drives shard throughput, batching windows, backpressure, checkpoints, and recovery lag. Variety is the number and variability of formats, schemas, and source systems; it drives deserialization, cataloging, schema evolution, normalization, and quarantine design.
For example, a nightly 20 TB Parquet backfill has high volume but low arrival velocity and fits a distributed batch engine. A continuous 2 MB/s JSON feed has lower daily volume but stricter velocity and schema-variation concerns, so a stream with checkpoints and a raw replay copy is more important. Quantify all three instead of equating “big data” only with byte count.
Lambda fit and limits
Lambda fits record handlers, lightweight transformations, schedulers, and control-plane automation. It has a maximum execution duration, so a multi-hour backfill belongs in Glue, EMR Serverless, AWS Batch, or a container workflow. Invocation payload limits and source integration limits also matter, especially now that Kinesis can accept intermittently large records that some downstream integrations cannot consume at the same size.
Increasing Lambda memory also increases available CPU. Benchmark duration and total cost instead of assuming the smallest memory is cheapest. Important controls include:
Reserved concurrency: Guarantees and caps concurrency for one function, protecting a database or vendor API from uncontrolled scale-out.
Provisioned concurrency: Keeps initialized environments ready when predictable low startup latency justifies the cost.
Event-source batch size and window: Amortize invocation overhead, but larger batches increase retry scope unless partial batch response is used.
Parallelization factor: Increases concurrent batches per Kinesis shard while preserving ordering for the same partition key.
Timeout and memory: Set from measurements and leave enough time for cleanup or a failure destination.
Initialize SDK clients and static resources outside the handler so a warm environment can reuse them. Reuse database connections carefully through RDS Proxy, and cache secrets for a bounded interval with the Parameters and Secrets extension.
Lambda storage choices
The /tmp directory is configurable ephemeral storage attached to one execution environment. It is useful for decompression, local sort spill, or model files reused by warm invocations, but it is not shared or durable.
Lambda can mount an Amazon EFS access point when functions need a shared POSIX file system larger than ephemeral storage. The function must have VPC network connectivity, mount-target reachability, security-group rules, and file permissions. EFS adds network latency and does not turn a 15-minute function into a batch engine. Store durable objects in S3 when file-system semantics are unnecessary.
ECS and EKS
Use containers when a task needs a custom operating-system package, a long-running consumer, more predictable CPU or memory, local daemon behavior, or a runtime model outside Lambda's fit.
Choice Strong fit Operational responsibility
ECS with Fargate AWS-native container scheduling without managing EC2 hosts Task definitions, networking, scaling, images, logs
ECS on EC2 Specialized instances, GPUs, or host-level cost control Above plus EC2 capacity and patching
EKS Kubernetes APIs, ecosystem, and portability are requirements Cluster and add-on lifecycle, pod scheduling, networking, policies
Right-size both requests and limits in Kubernetes. An inflated request wastes nodes; a low memory limit causes termination. For a parallel file workload, partition the work into independent messages and use service autoscaling from queue depth or custom lag metrics.
Distributed-computing mechanics
Distributed systems divide data into partitions processed by workers. Speedup stops when work is not divisible or coordination dominates. Common bottlenecks are:
Shuffle: Grouping or joining by key moves records across the network.
Skew: One hot key gives one task much more data than peers.
Serialization: Large row objects or inefficient formats consume CPU and network.
Driver concentration: Collecting a dataset or huge metadata list on one driver defeats distribution.
Small files: Scheduling and object-request overhead dominates useful work.
Data structures and algorithms in pipelines
A hash map gives expected O(1) key lookup for in-memory deduplication or joins, but memory grows with unique keys; an unbounded stream therefore needs windows, state expiry, or an external store. A queue or deque models ordered work and bounded buffering. A heap maintains a top-k set or priority order in O(log k) per update without sorting every record. A tree keeps ordered keys for range traversal; database B-trees make point and range access efficient but add write and index-maintenance cost. A graph models relationships and pipeline dependencies; breadth-first or depth-first traversal grows with the vertices and edges explored, so constrain path depth and filters.
Algorithm choice affects the distributed plan. Hash partitioning co-locates equal keys but can create a hot partition. Sort-merge operations support ordered/range work but add O(n log n) local sorting and shuffle. A DAG topological order is appropriate for dependencies only when there is no cycle. State the access pattern, input scale, memory bound, and failure behavior before choosing the structure.
Performance workflow
Measure input bytes, output bytes, CPU, memory, shuffle bytes, spill, task duration, and skew before scaling. Push filters and projections to the source, select columnar formats, broadcast only genuinely small dimensions, and compact files. Then increase parallelism until the source, network, or destination becomes the bottleneck.
For resilient processing, every unit of work needs an identifier and bounded retry. A container or Lambda retry can execute twice, so commit outputs transactionally or to a unique staging path before publishing them. Compute elasticity without idempotency produces faster duplicates.