Study guide
4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization)
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization)
Key Takeaways Explain This Topic AI
EMR Serverless automatically provisions, scales, and manages compute infrastructure for Apache Spark and Hive jobs without requiring cluster sizing, OS tuning, or EC2 management.
EMR Instance Fleets can consider many compatible EC2 types and a list of candidate subnets; EMR chooses capacity in one subnet and Availability Zone for a cluster rather than spanning one cluster across AZs.
Replaceable task nodes are often good Spot candidates, while Primary and stateful Core nodes commonly use On-Demand capacity; choose the mix from the workload's recovery design and data durability needs.
The capacity-optimized Spot allocation strategy selects pools with the most available capacity to reduce interruption risk, but checkpointing and retries are still required.
EMR Managed Scaling automatically adds or removes On-Demand and Spot instances based on YARN memory and container queues, outperforming legacy metric-based scaling rules.
Last updated: August 2026
4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization)
Modern enterprise data engineering demands a flexible approach to compute capacity management. AWS provides two primary deployment models for big data processing: Amazon EMR on EC2 (giving data teams granular control over instance types, auto-scaling policies, and network topologies) and Amazon EMR Serverless (a fully serverless option that eliminates cluster management entirely).
Selecting the right deployment model, balancing On-Demand and EC2 Spot Instances, and configuring advanced Spot allocation policies are fundamental skills tested on the DEA-C01 exam.
EMR Deployment Models: EMR on EC2 vs. EMR Serverless vs. EMR on EKS
AWS offers three distinct operational models for running open-source big data frameworks:
Architectural & Operational Feature Comparison
Operational Feature EMR on EC2 EMR Serverless EMR on EKS
Management Overhead High (Configure EC2 instance types, EBS sizes, AMI updates, YARN tuning). Zero (Serverless; AWS manages compute, OS, and framework infrastructure). Medium (Requires managing Kubernetes clusters, pod specs, and EKS node groups).
Cluster Startup Time 3 to 10 minutes (EC2 instance provisioning + bootstrap scripts). Seconds (Pre-initialized capacity) or ~1-2 minutes cold start. Seconds (If Kubernetes pod capacity is pre-allocated).
Scaling Granularity Instance-level (Adding/removing EC2 nodes via EMR Managed Scaling). Fine-grained worker level (Per-vCPU / RAM allocation per task execution). Pod-level (Kubernetes Horizontal Pod Autoscaler).
Cost Model EMR Surcharge + EC2/EBS pricing per second (Min 1 min). Billed per vCPU, RAM (GB), and Storage (GB) per second used by job workers. EMR Surcharge + EKS EC2/Fargate worker cost.
Ideal Workload Profile Long-running clusters, customized C++ binaries, legacy Hadoop HDFS applications. Ad-hoc queries, transient batch ETL, variable workload spikes, serverless pipelines. Containerized microservice ecosystems sharing unified Kubernetes clusters.
EMR Serverless Deep Dive Architecture
EMR Serverless introduces the concept of an EMR Serverless Application . An application is assigned a framework (e.g., Apache Spark or Apache Hive) and a release version.
Core Serverless Configuration Parameters
Initial Capacity (Pre-Initialized Capacity) : Data engineers can pre-allocate worker resources to respond instantly to incoming jobs, eliminating cold-start latency for latency-sensitive SLAs.
Maximum Capacity ( maximumCapacity ) : Establishes upper boundary limits on total vCPU, Memory (GB), and Disk (GB) resources that an application can consume concurrently across all executing jobs. This prevents runaway billing during recursive code bugs or unexpected data spikes.
- Worker Sizing : Individual worker sizes can be configured specifically for driver and executor tasks: {
- "monitoringConfiguration": {
- "s3MonitoringConfiguration": {
- "logUri": "s3://my-company-emr-logs/serverless/"
- }
- },
- "maximumCapacity": {
- "cpu": "1000 vCPU",
- "memory": "4000 GB",
- "disk": "20000 GB"
- }
- }
- EMR on EC2 Capacity Strategy: Uniform Instance Groups vs. Instance Fleets
When deploying EMR on EC2 , data engineers must choose between two node allocation structures:
1. Uniform Instance Groups
Allows selecting exactly one EC2 instance type and pricing model (On-Demand or Spot) per node group (Primary, Core, Task).
Highly restrictive for Spot Instance usage. If the chosen single instance type experiences a Spot capacity shortage in a specific Availability Zone, the cluster cannot provision worker capacity and job execution halts.
2. Instance Fleets (Recommended Best Practice)
Allows specifying multiple EC2 instance types and subnet options. At launch, EMR selects one subnet, so every node in a cluster remains in one Availability Zone.
Uses a target capacity model based on weighted capacity units (e.g., an m5.xlarge provides 4 units, while an m5.2xlarge provides 8 units).
Significantly increases Spot fulfillment success rates by allowing EMR to automatically pick available Spot pools from a broad list of instance types.
On-Demand vs. Spot Optimization & Resiliency
EC2 Spot Instances offer discounts of up to 90% compared with On-Demand prices by using spare AWS compute capacity. AWS can reclaim Spot capacity and ordinarily provides a two-minute interruption notice before stopping or terminating an instance.
- Node Allocation Strategy Matrix EMR Cluster Node Strategy
- ├── Primary Node ---> 100% On-Demand (Never Spot)
- ├── Core Nodes ---> 100% On-Demand (Preserves HDFS Stability)
- └── Task Nodes ---> Spot candidates (Capacity-Optimized; checkpoint and retry)
- Spot Allocation Strategies for Instance Fleets
capacity-optimized (Recommended for resilient EMR Spot fleets) : Provisions from pools with the most available capacity. It reduces interruption risk relative to price-only selection, but it does not eliminate interruptions; jobs still need checkpoints, idempotency, and retries.
price-capacity-optimized : Balances capacity depth with lower prices, selecting pools that are both resilient to interruptions and cost-effective.
lowest-price (Legacy / Avoid) : Strictly selects the cheapest Spot pool. This often results in high interruption frequencies when multiple users compete for the same low-cost pool.
Graceful Node Decommissioning
Graceful node decommissioning stops placing new work on a node selected for scale-down and gives supported engines time to drain work. Newer EMR releases can preserve Spark shuffle availability during managed scale-down when the documented YARN and external-shuffle-service settings are enabled. A Spot interruption notice is short, however, so decommissioning does not guarantee that tasks or local shuffle data survive; checkpoints, durable source data, and retries remain necessary.
EMR Managed Scaling
Unlike traditional EC2 Auto Scaling based on static CloudWatch CPU metrics, EMR Managed Scaling continuously samples cluster YARN metrics every 5 to 10 seconds. It automatically resizes Core and Task node fleets to optimize resource utilization.
Key Managed Scaling Controls
Sets explicit bounds: Minimum Capacity Units and Maximum Capacity Units .
Separates Maximum On-Demand Capacity Units from total capacity, forcing all dynamic scale-out units to be provisioned exclusively as low-cost EC2 Spot Instances.