Study guide
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage
Key Takeaways Explain This Topic AI
Amazon Redshift uses a Massively Parallel Processing (MPP) shared-nothing architecture consisting of a Leader Node for query planning and compilation, and multiple Compute Nodes executing parallel slice operations.
RA3 node instances decouple compute from storage by leveraging Redshift Managed Storage (RMS), which automatically scales storage up to 16 PB per cluster using high-performance Amazon S3 backed by local NVMe SSD caching.
Automatic WLM adjusts memory and concurrency for query needs, while short query acceleration uses a dedicated space to keep eligible short queries from waiting behind long-running work.
Concurrency Scaling adds transient capacity for eligible read and write queries when enabled on WLM queues; eligibility and usage credits must still be monitored.
Redshift Serverless scales compute measured in RPUs and meters compute separately from managed storage; base capacity, maximum capacity, and usage limits are cost controls rather than fixed-bill guarantees.
Last updated: August 2026
- 6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage
- Data Warehousing & MPP Architecture Fundamentals
In modern data engineering architectures, enterprise data warehouses serve as the central repository for analytical processing (OLAP). Unlike transactional databases (OLTP) optimized for row-by-row CRUD operations and low-latency writes, analytical data warehouses are optimized for complex aggregations, multi-table joins, and scanning billions of records across narrow column subsets.
Amazon Redshift is AWS's flagship fully managed cloud data warehouse. It utilizes a Massively Parallel Processing (MPP) , shared-nothing architecture. In an MPP database, multiple compute nodes work in parallel to execute single query tasks across partitioned datasets. Every compute node has dedicated CPU, RAM, and storage slices, ensuring that computational capacity scales linearly as cluster size increases.
Redshift Cluster Anatomy: Leader Node & Compute Nodes
An Amazon Redshift cluster consists of two distinct node tiers: a single Leader Node and one or more Compute Nodes . Client App (JDBC/ODBC) ---> [ Leader Node ] ---> Compute Node 1 (Slice 0, Slice 1)
- --> Compute Node 2 (Slice 2, Slice 3)
- 1. The Leader Node
The Leader Node acts as the SQL entry point and orchestrator for the entire cluster. Client applications connect exclusively to the Leader Node via standard PostgreSQL-compliant JDBC or ODBC drivers (typically on port 5439 ).
Key functions of the Leader Node include:
Query Parsing & Optimization : Accepts SQL queries, parses execution trees, and optimizes execution plans.
Code Compilation : Compiles optimized query plans into C++ execution code segments and distributes compiled binary code to Compute Nodes.
Aggregating Partial Results : Receives intermediate query results returned by compute nodes, performs final aggregations or sorting, and returns the final result set to the client application.
Catalog Management : Maintains database metadata, table schemas, view definitions, and security permissions.
Exam Note : The Leader Node does not store user table data and is not billed separately. Data storage and query execution occur strictly on Compute Nodes.
2. Compute Nodes & Node Slices
Compute Nodes execute compiled query code sent by the Leader Node and return intermediate results. Each Compute Node is divided into virtual processing units called Slices .
Slice Architecture : A slice is assigned a portion of the node's CPU, memory, and disk capacity. The number of slices per node depends on the node instance size (for example, ra3.xlplus has 2 slices, ra3.4xlarge has 4 slices, and ra3.16xlarge has 16 slices).
Parallel Execution : When data is loaded into Redshift, table rows are distributed across slices according to the table's Distribution Style. During query execution, all slices operate concurrently in parallel.
- Decoupled Compute & Storage: RA3 Nodes & Redshift Managed Storage (RMS)
- Legacy Node Types vs. RA3 Architecture
Historically, Redshift offered coupled compute-and-storage instance types:
Dense Storage (DS2) : Optimized for large data volumes using hard disk drives (HDDs), but limited compute performance.
Dense Compute (DC2) : Optimized for high performance using local NVMe SSDs, but storage was strictly bound to node count.
In coupled architectures, scaling storage required adding compute nodes (increasing costs unnecessarily), while scaling compute forced paying for unneeded storage capacity.
The RA3 Instance Family & RMS
RA3 instances revolutionize Redshift architecture by completely decoupling compute capacity from storage capacity through Redshift Managed Storage (RMS) .
- Feature Legacy Coupled Nodes (DC2/DS2) Next-Gen Decoupled Nodes (RA3)
- Storage Layer Fixed local disks (HDD/SSD) Redshift Managed Storage (RMS) on Amazon S3
- Storage Scaling Requires adding compute nodes Automatically scales up to 16 PB per cluster
- Caching Layer Local disk only High-speed local NVMe SSD caching
- Data Sharing Not supported across clusters Native Redshift Data Sharing across clusters
Automatic Optimization Fewer decoupled-storage capabilities Automatic table optimization, Automatic WLM, and short query acceleration
How Redshift Managed Storage (RMS) Works
RMS leverages Amazon S3 as the underlying persistent storage tier, offering 99.999999999% (11 9s) of data durability. However, to maintain sub-second query performance, RA3 compute nodes use high-performance local NVMe SSDs as an intelligent, tier-1 data cache.
Hot Data Handling : Frequently queried data blocks reside in local NVMe SSD cache.
Cold Data Offload : As data grows beyond local SSD capacity, colder data blocks spill over automatically to RMS on S3 without any administrative intervention.
High-Bandwidth Networking : RA3 nodes use AWS Nitro System high-bandwidth networking (up to 100 Gbps) to stream data blocks rapidly from RMS into compute slices during heavy scans.
Automatic WLM and Short Query Acceleration
Automatic workload management (WLM) assigns memory and adjusts concurrency from observed query resource needs. Heavy joins can run at lower concurrency while lighter scans and inserts run at higher concurrency. Queue priority expresses business importance, and query monitoring rules can log, hop, or cancel queries that cross defined boundaries.
Short query acceleration (SQA) works alongside WLM. Redshift uses machine learning to predict eligible short-running queries and runs them in a dedicated space so they do not wait behind long ETL scans. SQA is enabled by default in the default parameter group and in new parameter groups. It prioritizes short work; it does not make an inefficient large query inherently cheap. Long ETL scans ---> Automatic WLM queue and memory allocation
- Short eligible queries ---> SQA dedicated space ---> Earlier completion
- Workload Scaling & Elasticity Mechanisms
Redshift provides four major elasticity mechanisms to handle variable analytical workloads:
1. Elastic Resize
Elastic Resize allows you to add or remove nodes from an existing Redshift cluster, or change node types (e.g., from dc2.large to ra3.4xlarge ) within minutes. During an Elastic Resize, Redshift pauses active queries, updates slice configurations, re-maps storage pointers in RMS, and resumes database operations. Total downtime is typically under a few minutes.
2. Classic Resize
Classic Resize creates a completely new target cluster in the background, copies data from the source cluster snapshot, and updates DNS endpoints. Classic Resize takes hours or days for multi-terabyte datasets and is primarily used when resizing between incompatible node configurations where Elastic Resize is unsupported.
3. Concurrency Scaling
When hundreds of concurrent business intelligence (BI) users execute queries simultaneously, query queues can form in Redshift Workload Management (WLM). Concurrency Scaling addresses read query spikes by automatically provisioning transient cluster capacity within seconds.
Automatic Operations : As eligible queues build, Redshift routes supported read or write queries to concurrency scaling capacity automatically.
Eligibility Matters : Not every query or write pattern is eligible, so monitor queueing and the concurrency-scaling status rather than assuming all ingestion is offloaded.
Cost Efficiency : Clusters earn 1 hour of free Concurrency Scaling credits for every 24 hours the main cluster runs. Additional usage is billed on a per-second basis.
4. Redshift Serverless
Redshift Serverless removes the requirement to provision or manage data warehouse clusters. You configure base capacity and capacity controls in Redshift Processing Units (RPUs) . Redshift Serverless scales capacity for workload demand; compute is metered per second with a 60-second minimum, and managed storage and other applicable usage are billed separately.
Code Example: Managing Redshift Provisioned Clusters and Serverless via AWS SDK Boto3 import boto3
- # Initialize Redshift & Redshift Serverless clients
- redshift_client = boto3.client('redshift', region_name='us-east-1')
- serverless_client = boto3.client('redshift-serverless', region_name='us-east-1')
def scale_provisioned_cluster(cluster_identifier: str, target_nodes: int):
"""
Executes an Elastic Resize on a provisioned RA3 Redshift cluster.
"""
try:
- response = redshift_client.resize_cluster(
- ClusterIdentifier=cluster_identifier,
- ClusterType='multi-node',
- NodeType='ra3.4xlarge',
- NumberOfNodes=target_nodes,
- Classic=False
- )
- print(f"Elastic Resize initiated for cluster '{cluster_identifier}'. Target nodes: {target_nodes}")
- return response['Cluster']['ClusterStatus']
except Exception as e:
- print(f"Error initiating Elastic Resize: {str(e)}")
- raise e
def configure_serverless_workgroup(workgroup_name: str, base_rpu: int):
"""
Updates the base Redshift Processing Unit (RPU) capacity for a Serverless Workgroup.
"""
try:
- response = serverless_client.update_workgroup(
- workgroupName=workgroup_name,
- baseCapacity=base_rpu
- )
- print(f"Serverless Workgroup '{workgroup_name}' updated with base RPU capacity: {base_rpu}")
- return response['workgroup']['workgroupArn']
except Exception as e:
- print(f"Error updating Serverless Workgroup: {str(e)}")
- raise e