Study guide
10.2 Designing High Availability Architectures Across AWS Data Services
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
10.2 Designing High Availability Architectures Across AWS Data Services
Key Takeaways Explain This Topic AI
Amazon Aurora Multi-AZ architecture decouples compute from storage, maintaining 6 physical copies of data across 3 Availability Zones with quorum writes (4 of 6) and quorum reads (3 of 6).
Amazon EMR can use three primary nodes to remove a primary-node single point of failure, but every cluster remains in one subnet and one Availability Zone, so this feature does not survive an AZ outage.
Amazon MSK achieves fault tolerance by spreading Kafka brokers across 3 AZs, requiring topic replication factors of 3 and min.insync.replicas settings of 2 for zero-data-loss durability.
Stateful stream processors recover from checkpoints and replay retained input; end-to-end exactly-once outcomes also require compatible source semantics and idempotent or transactional sinks.
OpenSearch Service Multi-AZ with Standby distributes data copies and standby capacity across three Availability Zones to speed failover; durability still depends on replicas, snapshots, and failure scope.
Last updated: August 2026
10.2 Designing High Availability Architectures Across AWS Data Services
High Availability (HA) in AWS data engineering refers to the capability of an analytics platform to remain continuously operational and accessible during hardware faults, software failures, or whole Availability Zone (AZ) outages. Unlike Disaster Recovery—which focuses on cross-Region recovery after catastrophic events—HA focuses on intra-Region fault tolerance, automatic health monitoring, and self-healing infrastructure.
1. High Availability Storage and Database Topologies
Achieving high availability at the data tier requires decoupling compute nodes from persistent storage while distributing state across redundant physical facilities (AZs).
Amazon Aurora Decoupled Quorum Architecture
Amazon Aurora achieves unmatched database availability through its custom distributed storage volume:
6-Way Replication Across 3 AZs: Aurora automatically replicates every 10 GB storage chunk six times across three AZs within a Region.
Quorum Mechanics:
Write Quorum (4 of 6): A write operation is acknowledged as successful as soon as 4 out of the 6 storage nodes confirm the write. A complete loss of one entire AZ (2 copies) does not impair write availability.
Read Quorum (3 of 6): Read operations require agreement from 3 nodes, allowing rapid reconstruction of corrupted blocks.
Storage Auto-Healing: Storage disk failures are repaired transparently in the background without affecting database compute availability.
Read Replica Auto-Failover: If the primary writer instance fails, Aurora automatically promotes one of up to 15 Read Replicas to primary writer status in under 120 seconds (often within 30 seconds). Aurora Shared Distributed Storage Volume
- +-----------------------------------------------------------------------+
- | AZ 1: [ Data Copy 1 ] [ Data Copy 2 ] |
- | AZ 2: [ Data Copy 3 ] [ Data Copy 4 ] <-- Write Quorum: 4 of 6 |
- | AZ 3: [ Data Copy 5 ] [ Data Copy 6 ] <-- Read Quorum: 3 of 6 |
- +-----------------------------------------------------------------------+
- Amazon RDS Multi-AZ Deployment (Instance Level)
For traditional relational engines (PostgreSQL, MySQL, SQL Server), standard Amazon RDS Multi-AZ uses synchronous block-level storage replication:
Provisions a primary database instance in AZ-A and a standby instance in AZ-B.
Synchronously replicates storage updates at the EBS volume layer.
In the event of primary hardware failure, AZ outage, or database crash, RDS automatically updates DNS records to point to the standby instance. Application connection strings remain unchanged.
2. High Availability in Distributed Analytics Engines
Distributed compute frameworks like Apache Spark and Hadoop on Amazon EMR require special configuration to prevent primary node failures from aborting long-running ETL jobs.
Amazon EMR Multi-Master High Availability
By default, a standard EMR cluster has one primary node, creating a single point of failure (SPOF). Choosing an EMR cluster with multiple primary nodes provisions 3 primary nodes within the cluster's single subnet and Availability Zone:
HDFS NameNode HA: Active NameNode running on Primary Node 1 with Standby NameNodes on Nodes 2 and 3, coordinated via Apache ZooKeeper and Quorum Journal Managers.
YARN ResourceManager HA: Active ResourceManager with automatic failover to standbys.
Availability boundary: Instance fleets can consider subnets in multiple AZs when the cluster is launched, but EMR selects one subnet and provisions every node in that one AZ. Three primary nodes tolerate a primary-node failure, not an AZ failure. Use reproducible data in S3 and orchestration that can launch a replacement cluster in another AZ for broader recovery.
AWS Glue and Serverless Compute Fault Tolerance
AWS Glue ETL jobs are serverless and inherently resilient:
Glue abstracts worker infrastructure, but architecture should not assume a documented worker-AZ topology or that every worker loss is transparent. Keep durable inputs in S3, configure retries, and make outputs idempotent so a failed attempt can be rerun safely.
3. Fault-Tolerant Streaming Data Pipelines
Real-time ingestion pipelines must ingest continuous streams without data loss during infrastructure failures.
Amazon MSK (Managed Streaming for Apache Kafka) HA Best Practices
To ensure multi-AZ fault tolerance in Amazon MSK:
Broker Distribution: Provision MSK clusters with brokers evenly distributed across 3 Availability Zones .
Topic Replication Factor: Set default.replication.factor = 3 so every Kafka topic partition has a primary leader broker in one AZ and follower replicas in two other AZs.
In-Sync Replicas Configuration: Set min.insync.replicas = 2 and require producers to use acks = all (or acks = -1 ). This guarantees that a write is only acknowledged after it has been durably committed to at least 2 brokers across distinct AZs.
Amazon Kinesis Data Streams Durability
Amazon Kinesis Data Streams natively provides multi-AZ high availability:
Data records are synchronously replicated across 3 Availability Zones before write acknowledgment.
Enhanced Fan-Out (EFO): Dedicated 2 MB/sec throughput per consumer per shard using HTTP/2 push delivery, shielding downstream consumers from reading contention during node recovery.
Retention periods can be extended up to 365 days, providing a safety buffer to replay data after downstream pipeline crashes.
Stateful Streaming Checkpointing
Stateful stream processing using Managed Service for Apache Flink or Glue Streaming preserves state during worker node failure:
Managed Service for Apache Flink coordinates checkpoints and service-managed snapshots; applications should not assume a user-visible immutable S3 checkpoint path.
Embedded RocksDB can back large application state, depending on runtime and application configuration.
After worker failure, Flink restores state and source positions from a completed checkpoint. End-to-end exactly-once results additionally require replayable sources plus sinks and application logic that support transactional or idempotent recovery.
- 4. Resilient Search and Query Infrastructure
- Amazon OpenSearch Service Multi-AZ with Standby
For enterprise operational analytics, Amazon OpenSearch Service offers Multi-AZ with Standby :
Node Allocation: Distributes active data nodes across 2 AZs and standby nodes in a 3rd AZ.
Enforced Shard Distribution: Primary shards and replica shards are strictly isolated in separate AZs.
Automated Failover: If an active node or AZ fails, OpenSearch promotes a standby node in under 1 minute without dropping indexing requests or incurring write performance degradation.