Study guide
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures
Key Takeaways Explain This Topic AI
Diagnose Glue and Spark memory failures by identifying driver versus executor pressure, then reduce input early, remove driver-side collect operations, address skew, tune partitions, or select a suitable worker type.
YARN container memory failures require inspecting executor heap, memory overhead, Python or native memory, container logs, and the settings for the deployed Spark and EMR release rather than assuming one universal default.
Lambda has a 15-minute invocation limit and up to 10,240 MB of memory; stream or chunk bounded input and move long-running distributed batch work to Glue, EMR, ECS, or another suitable service.
Glue job bookmarks track source progress; backfills and reprocessing need an explicit reset, rewind, pause, or isolated output plan so bookmark state does not cause omissions or duplicates.
EMR Spark Event Logs exported to Amazon S3 enable post-execution performance profiling via the Spark History Server, helping data engineers diagnose execution skew, GC pauses, and YARN resource starvation.
Last updated: August 2026
Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures
Data engineers designing automated analytics pipelines on AWS must rapidly diagnose and resolve runtime execution failures across serverless and managed compute engines. This section provides an authoritative technical breakdown of failure modes, diagnostic logging workflows, and memory tuning strategies for AWS Glue , Amazon EMR , and AWS Lambda .
AWS Glue Job Execution & Memory Troubleshooting
AWS Glue PySpark jobs execute on managed Apache Spark clusters. Failures in Glue typically stem from memory pressure on either the Spark driver or Spark executor nodes, or misconfigured state management via Glue Job Bookmarks.
1. Driver OutOfMemoryError (OOM)
The Spark driver node orchestrates execution, builds query plans, and collects metadata. A driver crash ( java.lang.OutOfMemoryError: Java heap space ) occurs when data or metadata returned to the driver exceeds its JVM heap size.
Root Causes:
Calling .collect() or .toPandas() on large Spark DataFrames, pulling gigabytes of distributed data onto the single driver node.
Listing millions of S3 objects during schema inference or partition discovery without pushdown filters.
Using un-partitioned Glue DynamicFrames with massive single-file outputs.
Remediation Strategies:
Eliminate Driver Collection: Replace .collect() with distributed S3 writes ( DataFrame.write.parquet() ) or use .take(n) for debugging.
Apply Pushdown Predicates: Filter S3 partitions at the Glue Data Catalog level before reading objects into memory: datasource = glueContext.create_dynamic_frame.from_catalog(
- database="analytics_db",
- table_name="raw_logs",
- push_down_predicate="year == '2026' and month == '08'"
- )
Scale Worker Types: Upgrade Glue Worker Types to allocate larger driver heaps:
- G.1X : 1 DPU (4 vCPUs, 16 GB RAM)
- G.2X : 2 DPUs (8 vCPUs, 32 GB RAM)
- G.4X : 4 DPUs (16 vCPUs, 64 GB RAM)
- G.8X : 8 DPUs (32 vCPUs, 128 GB RAM)
Select memory-optimized or larger workers only after metrics identify driver or executor pressure; verify current disk and worker availability in the target Region.
2. Executor OutOfMemoryError (OOM)
Executor nodes process distributed partition data chunks. Executor OOM errors occur when individual partitions exceed worker memory or when operations cause severe data skew.
Remediation:
Repartitioning Data: Use .repartition(N) or .coalesce(N) to equalize partition sizes across executors and eliminate skewed partitions.
Group Size & Factor Tuning: Adjust Glue DynamicFrame S3 reading parameters: groupFiles="inGroup" and groupSize="134217728" (128 MB) to bundle small files into uniform partitions.
3. Glue Job Bookmarks State Management
Glue Job Bookmarks track processed S3 objects by maintaining state across job runs.
Common Failure Scenarios:
Changed Source Context: Reusing a transformation_ctx after changing its S3 source path can apply old bookmark state and skip files. S3 bookmarks otherwise use object last-modified time, so a new or modified object can be selected.
Duplicate Processing: Calling job.init() without job.commit() prevents successful bookmark advancement and can lead to duplicate processing on later runs.
Resolution: Configure job parameters ( --job-bookmark-option set to job-bookmark-enable , job-bookmark-disable , or job-bookmark-pause ). When re-processing updated historical partitions, pause or reset the bookmark state via the AWS Management Console or AWS CLI ( aws glue reset-job-bookmark --job-name <name> ).
AWS EMR Cluster & YARN Execution Troubleshooting
Amazon EMR clusters run Apache Spark, Hive, and Presto managed by YARN (Yet Another Resource Negotiator). Troubleshooting EMR requires inspecting YARN resource allocations and container lifecycle limits.
1. YARN Container Memory Exceeded Failures
A frequent EMR failure mode is YARN forcibly terminating an executor container: Container killed by YARN for exceeding memory limits. 10.5 GB of 10.0 GB physical memory used.
Root Cause: Spark executors require physical memory beyond the JVM heap for off-heap execution, PySpark Python process communications, and memory-mapped file operations.
Remediation via Memory Overhead Tuning: Increase spark.executor.memoryOverhead (default is max(384MB, 0.10 * spark.executor.memory) ). When using heavy PySpark UDFs, NumPy, or Apache Arrow, set memory overhead explicitly: spark-submit \
- -conf spark.executor.memory=8g \
- -conf spark.executor.memoryOverhead=2048m \
- -conf spark.driver.memory=8g \
- -conf spark.driver.memoryOverhead=2048m \
- etl_script.py
- 2. Spot Instance Terminations & Resiliency
EMR core and task nodes running on EC2 Spot Instances can be reclaimed by AWS with a 2-minute notification, resulting in process termination ( Exit Code 137 / 143 ).
Resiliency Architecture:
Use EMR Instance Fleets configured with the capacity-optimized allocation strategy across multiple EC2 instance families (e.g., r5.2xlarge , r5a.2xlarge , r6g.2xlarge ).
Use the EMR release-specific YARN graceful-decommission settings for managed scale-down. Newer releases can keep shuffle data available while a node drains, but a Spot interruption can still outlast the available notice, so retain retries and durable checkpoints.
3. Diagnostic Logging & Spark History Server
To diagnose executor losses, memory leaks, and GC pauses:
Enable Spark Event Logging in spark-defaults.conf : spark.eventLog.enabled true spark.eventLog.dir s3://my-emr-logs-bucket/spark-events/
Launch a persistent Spark History Server on an EC2 instance or viewing container to inspect stage execution timelines, DAG visualizations, and garbage collection (G1GC) metric graphs.
AWS Lambda ETL Bottlenecks & Failure Remediation
AWS Lambda is widely used for event-driven micro-ETL (e.g., processing S3 object creation triggers). However, strict service quotas require deliberate architectural planning.
Key Service Quotas & Mitigations
Execution Timeout (15 minutes): If an S3 file transformation exceeds 900 seconds, Lambda terminates the execution.
Fix: Read bounded chunks or byte ranges when processing can be safely partitioned, or offload long-running distributed work to Glue, EMR, or containers. S3 Select is closed to new customers.
Memory Quota (10,240 MB) & /tmp Ephemeral Storage: Default /tmp space is 512 MB, configurable up to 10,240 MB (10 GB).
Fix: If intermediate files exceed memory, configure Lambda /tmp storage to 10 GB or mount an Amazon EFS file system for shared persistence.
Concurrency Limits & Throttling (429 TooManyRequestsException ): Unreserved concurrency spikes can exhaust the account baseline limit (1,000 per region).
Fix: Assign Reserved Concurrency to critical ETL Lambdas to guarantee capacity, or configure Provisioned Concurrency to eliminate cold-start latency.
Execution Failure Troubleshooting Matrix
Service Failure Symptom Underlying Root Cause Key Diagnostic Metric / Log Marker Recommended Resolution
AWS Glue Driver OutOfMemoryError Memory exhaustion via .collect() or listing millions of small S3 objects CloudWatch Logs: java.lang.OutOfMemoryError: Java heap space Scale to G.2X / G.4X workers; use push_down_predicate ; replace .collect()
AWS Glue Skipped or duplicated S3 records Misconfigured or uncommitted Glue Job Bookmarks Job state output: Job run skipped files Reset bookmark via AWS CLI; ensure job.commit() is called in PySpark script
Amazon EMR Container killed by YARN Off-heap memory (PySpark / Arrow) exceeding JVM memory overhead YARN NodeManager log: Exceeding memory limits Increase spark.executor.memoryOverhead to 2048m or higher
Amazon EMR Executor Lost / Exit Code 137 Spot Instance reclamation by EC2 capacity manager EMR Cluster Metrics: SpotInstanceInterrupted Use Instance Fleets with capacity-optimized strategy and YARN Graceful Decommissioning
AWS Lambda Task timed out after 900.00s S3 file transformation exceeding 15-minute execution cap CloudWatch Metrics: Duration reaching Timeout value Process bounded byte ranges or orchestrate Glue/EMR/container work