Study guide
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem
Key Takeaways Explain This Topic AI
Amazon EMR architecture separates compute and storage by decoupling EC2 instance nodes (Primary, Core, Task) from persistent data stored on Amazon S3 via the EMR File System (EMRFS).
Primary nodes manage cluster orchestration (YARN ResourceManager and HDFS NameNode), Core nodes execute tasks and store HDFS data, and Task nodes provide compute without storing HDFS data.
EMRFS lets Amazon EMR read and write durable Amazon S3 data directly, with encryption and consistent S3 semantics; S3 Select is a legacy feature available only to existing S3 Select customers.
The EMR runtime for Apache Spark is an API-compatible, performance-optimized Spark distribution; improvement depends on EMR release, workload, file layout, and configuration.
Bootstrap Actions execute custom shell scripts as root on all cluster nodes before Hadoop/Spark daemons start, enabling custom library installation and security hardening.
Last updated: August 2026
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem
Amazon EMR (formerly Elastic MapReduce) is the industry-standard managed big data platform on AWS for processing petabyte-scale datasets using open-source frameworks such as Apache Spark , Apache Hive , Presto/Trino , Apache HBase , and Apache Flink . By decoupling compute resources from data storage, EMR enables data engineering teams to scale compute clusters dynamically while maintaining cost-effective, persistent data lakes on Amazon S3 .
Understanding the internal architecture of EMR clusters, node roles, storage abstractions, framework integrations, and cluster lifecycle management is a core requirement for the AWS Certified Data Engineer - Associate (DEA-C01) exam.
Amazon EMR Cluster Node Architecture
An Amazon EMR cluster consists of a collection of Amazon EC2 instances grouped into virtual node types. Each node type serves a specific architectural function within the Hadoop YARN (Yet Another Resource Negotiator) cluster topology.
EMR Node Classification & Responsibilities
Node Type Daemon Processes Primary Responsibilities HDFS Storage? Scalability & Fault Tolerance
Primary Node (Master) YARN ResourceManager, HDFS NameNode, Spark Master, Ganglia Coordinates cluster resource allocation, tracks job execution status, manages HDFS metadata, and dispatches tasks to worker nodes. Yes (Metadata / Small System Files) Single point of failure by default. High Availability (HA) deployments provision 3 Primary Nodes in the cluster's one selected subnet and Availability Zone.
Core Node YARN NodeManager, HDFS DataNode Executes data processing tasks (Spark executors, MapReduce mappers/reducers) AND hosts persistent HDFS data partitions across attached EBS volumes. Yes (Runs HDFS DataNode) Can be scaled out dynamically. Scaling down requires HDFS block decommissioning; improper termination risks data loss if HDFS replication factor is violated.
Task Node YARN NodeManager Strictly executes compute tasks (Spark executors, Presto workers). Holds zero HDFS data and runs no HDFS daemons. No (Pure Compute) 100% Stateless . Safe to scale up or down aggressively, or launch as EC2 Spot Instances, without risking HDFS data corruption or cluster failure.
Exam Tip: If an exam scenario asks how to cut costs for an EMR cluster processing data stored on Amazon S3 without incurring risk of HDFS data loss or HDFS block rebalancing overhead, the optimal architecture is to maintain a minimal set of On-Demand Core nodes and scale compute capacity using Task Nodes on EC2 Spot Instances .
Storage Layer Architecture: HDFS vs. EMRFS vs. Amazon S3
Amazon EMR supports two distinct storage abstractions for big data processing workloads: local Hadoop Distributed File System (HDFS) and the EMR File System (EMRFS) .
- Architectural Comparison: HDFS vs. EMRFS +-----------------------------------+
- | Amazon EMR Cluster |
- | |
- | +--------------+ +------------+ |
- | | Primary Node | | Core Nodes | |
- | +--------------+ +------------+ |
- | | |
- +-------------------------|---------+
- | EMRFS
- v
- +-------------------------+
- | Amazon S3 |
- | (Persistent Lake) |
- +-------------------------+
Architectural Attribute HDFS (Hadoop Distributed File System) EMRFS (EMR File System)
Storage Media Local Instance Store (NVMe) or EBS volumes attached to Core nodes. Amazon S3 Object Storage ( s3:// URI prefix).
Lifecycle Dependency Tied directly to the EMR cluster lifecycle; terminating the cluster deletes all HDFS data. Completely decoupled from cluster lifecycle; data persists independently in Amazon S3.
Cost Profile High (Requires keeping EC2/EBS provisioned continuously to preserve data). Ultra-low (Standard S3 pricing, S3 Intelligent-Tiering, S3 Glacier lifecycle rules).
Performance Profile Cluster-local storage suited to temporary data and high-throughput sequential access; latency and random-I/O behavior depend on the attached storage. Durable object storage optimized for scalable reads and writes through EMRFS.
Data Protection HDFS Block Replication (Default 3x replication factor). 99.999999999% (11 9s) durability, S3 Versioning, Cross-Region Replication (CRR).
Advanced EMRFS Capabilities
Current S3 processing boundary : S3 Select is closed to new customers. Prefer columnar formats, partition pruning, Spark predicate pushdown, and byte-range reads; existing S3 Select customers can continue using it.
KMS Encryption : EMRFS integrates natively with AWS Key Management Service (AWS KMS) to enforce transparent client-side or server-side encryption ( SSE-KMS / CSE-KMS ) for all data written to S3.
Consistent View (Historical) : Modern Amazon S3 provides strong read-after-write consistency for all applications out of the box, rendering legacy EMRFS DynamoDB consistency checks obsolete.
Big Data Open-Source Ecosystem Frameworks on EMR
Amazon EMR packages optimized builds of popular Apache ecosystem tools. AWS engineers continuously optimize these open-source engines specifically for AWS infrastructure:
Key Ecosystem Engines
Apache Spark : Distributed, in-memory processing engine for large-scale data processing, batch ETL, streaming, and machine learning. The EMR Runtime for Apache Spark is up to 3.5x faster than standard open-source Spark, reducing execution times and AWS costs.
Apache Hive : Data warehouse software that facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. EMR Hive can use the AWS Glue Data Catalog as a centralized, external Hive Metastore.
Presto / Trino : High-performance, distributed SQL query engine designed for fast interactive analytics against heterogeneous data sources (S3, Redshift, MySQL, Cassandra) without data movement.
Apache Flink : Open-source framework for stateful stream processing over unbounded data streams, supporting event-time processing and exactly-once semantics.
Apache Hudi / Apache Iceberg / Delta Lake : Open table formats supported on EMR that bring ACID transactions, time travel, and upsert/delete primitives to Amazon S3 data lakes.
Cluster Lifecycle: Bootstrap Actions vs. EMR Steps
EMR clusters can be provisioned as Long-Running Clusters (kept active for continuous interactive queries) or Transient Clusters (spin up, execute a specific workload, and terminate immediately upon completion).
1. Bootstrap Actions
Bootstrap actions are custom shell scripts executed on all cluster nodes (Primary, Core, and Task) before Hadoop, Spark, or other application daemons are installed and initialized. They run with root privileges.
Common Use Cases for Bootstrap Actions:
Installing custom Python libraries ( pip install pandas scikit-learn ) or system packages ( yum install -y libffi-devel ).
Modifying Linux kernel configuration settings or mounting external storage drives.
Injecting custom security certificates or configuring proxy settings.
2. EMR Steps
Steps are discrete, sequential units of work submitted to an active or transient cluster. A step can execute a Spark application ( spark-submit ), a Hive script ( hive -f script.q ), or a custom JAR file.
EMR Step Action-on-Failure Options:
TERMINATE_CLUSTER : Automatically shuts down the cluster if the step fails (ideal for transient production pipelines).
CANCEL_AND_WAIT : Cancels all pending steps in the queue and keeps the cluster running for debugging.
CONTINUE : Ignores the failure and immediately executes the next step in the queue.
Programmatic Provisioning: AWS CLI EMR Cluster Definition
The following AWS CLI command demonstrates how to launch a production-grade EMR cluster configured with Glue Catalog integration, custom Bootstrap Actions, and a transient Spark step: aws emr create-cluster \
- -name "Production-Sales-Aggregation-Cluster" \
- -release-label emr-6.13.0 \
- -applications Name=Spark Name=Hive \
- -use-default-roles \
- -ec2-attributes KeyName=my-ec2-key,SubnetId=subnet-0a1b2c3d4e \
- -instance-groups \
- InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m5.xlarge \
- InstanceGroupType=CORE,InstanceCount=2,InstanceType=r5.xlarge \
- InstanceGroupType=TASK,InstanceCount=4,InstanceType=c5.2xlarge \
- -bootstrap-actions Path=s3://my-company-emr-bootstrap/install-deps.sh,Name="Install Python Dependencies" \
--steps Type=Spark,Name="Execute Daily Aggregation",ActionOnFailure=TERMINATE_CLUSTER,Args=[--deploy-mode,cluster,--master,yarn,s3://my-company-emr-code/spark_aggregate.py] \
- -configurations '[
- {
- "Classification": "hive-site",
- "Properties": {
"hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveMetastoreFactory"
- }
- },
- {
- "Classification": "spark-hive-site",
- "Properties": {
"hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveMetastoreFactory"
- }
- }
- ]' \
- -auto-terminate