Study guide
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling
Key Takeaways Explain This Topic AI
Amazon DynamoDB uses an internal MD5-based hashing algorithm on the Partition Key (PK) to allocate data items uniformly across physical storage partitions, each supporting a maximum of 10 GB of storage, 1,000 WCUs, and 3,000 RCUs.
Single-Table Design leverages generic key attributes (PK and SK) and entity overloading to represent multi-entity relational hierarchies within a single DynamoDB table, eliminating high-latency table joins.
Global Secondary Indexes (GSIs) maintain an asynchronous copy of base table data with a candidate partition key and optional sort key, enabling flexible access patterns across partitions.
Local Secondary Indexes (LSIs) share the base table partition key but define an alternative sort key, must be created at table creation time, and share the 10 GB item collection size limit.
DynamoDB Streams capture item-level time-ordered change logs (INSERT, MODIFY, REMOVE) with 24-hour retention, enabling event-driven processing, Global Tables multi-region replication, and automated search index synchronization.
Last updated: August 2026
- 7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling
- DynamoDB Architecture & Physical Partitioning Fundamentals
Amazon DynamoDB is a fully managed, serverless Key-Value and Document NoSQL database engine designed to deliver consistent, single-digit millisecond latency at arbitrary operational scale. Unlike traditional relational database management systems (RDBMS) that run on fixed server instances, DynamoDB distributes data across an abstracted fleet of physical storage nodes managed entirely by AWS.
Physical Partition Allocation & Key Hashing
When a table is created, DynamoDB allocates physical storage nodes called partitions . Data placement across partitions is governed by an internal hashing algorithm:
Hash Function Execution : DynamoDB takes the value of the Partition Key (PK) (also called the Hash Attribute) and passes it through an internal hash function whose implementation is managed by DynamoDB.
Partition Hash Mapping : The resulting hash output determines the exact physical storage partition where the item will reside.
Partition Hard Limits : Each physical partition operates under strict resource limits:
Maximum Storage : 10 GB of total data items per partition.
Maximum Provisioned Write Capacity : 1,000 Write Capacity Units (WCUs) per partition.
Maximum Provisioned Read Capacity : 3,000 Read Capacity Units (RCUs) per partition. Item (PK: "USER#8842") ---> [ Internal Hash Function ] ---> Hash Value ---> Managed Physical Partition
When a table's total data volume exceeds 10 GB or total provisioned throughput exceeds 1,000 WCUs / 3,000 RCUs, DynamoDB automatically splits physical partitions. Partition splitting divides the key range in half, allocating new physical storage partitions without requiring downtime or application modification.
Hot Partitions & Partition Key Anti-Patterns
Because throughput limits apply at the physical partition level, an unbalanced distribution of access requests leads to a hot partition . If a single partition receives traffic exceeding 1,000 WCUs or 3,000 RCUs, DynamoDB throttles incoming requests and returns an HTTP 400 ProvisionedThroughputExceededException error, even if total table-level capacity is underutilized.
Common Partition Key Anti-Patterns:
Low Cardinality Keys : Choosing attributes with few distinct values, such as Status ( ACTIVE vs INACTIVE ) or Gender ( M vs F ). Millions of items map to the exact same partition.
Monotonically Increasing Keys : Using timestamps or sequential IDs ( 2026-08-13-0001 , 2026-08-13-0002 ) as partition keys. All new write operations hit the exact same physical partition handling the current time window.
Remediation Strategies:
High Cardinality Attributes : Select keys with millions of distinct values, such as UUID , user_id , or device_mac_address .
Write Sharding (Random Salting) : Append a calculated random suffix ( 0 through N-1 ) or calculated hash modulo to partition keys (e.g., ORDER#2026-08-13#2 ). This splits write bursts across multiple partitions. Reads then scatter-gather across all shards.
Throughput Capacity Planning: Provisioned vs. On-Demand
DynamoDB offers two billing and capacity management modes:
Capacity Feature Provisioned Mode On-Demand Mode
Workload Profile Predictable, steady traffic with gradual scaling Unpredictable, spikey, or unknown application traffic
- Capacity Specification Explicit RCU and WCU settings (with Auto Scaling) Automatic request-based scaling
- Cost Model Billed per provisioned RCU/WCU per hour Billed per million read/write request units
Throttling Risk Occurs if traffic exceeds provisioned limits No throttling up to double the previous traffic peak
RCU and WCU Calculation Rules for the Exam
Calculating throughput requirements is a frequent exam requirement. Remember these exact sizing formulas:
Read Capacity Units (RCUs)
Item Size Basis : Rounded up to the nearest 4 KB increment.
Eventually Consistent Read (Default): 1 RCU provides 2 reads per second up to 4 KB.
Strongly Consistent Read : 1 RCU provides 1 read per second up to 4 KB.
Transactional Read : 2 RCUs provide 1 transactional read per second up to 4 KB. RCUs required = ( ⌈ Item Size in KB 4 KB ⌉ ) × Reads per second × Consistency Factor \text{RCUs required} = \left( \lceil \frac{\text{Item Size in KB}}{4\text{ KB}} \rceil \right) \times \text{Reads per second} \times \text{Consistency Factor} RCUs required = ( ⌈ 4 KB Item Size in KB ⌉ ) × Reads per second × Consistency Factor
Where Consistency Factor = 1.0 for Strongly Consistent, 0.5 for Eventually Consistent, and 2.0 for Transactional.
Write Capacity Units (WCUs)
Item Size Basis : Rounded up to the nearest 1 KB increment.
Standard Write : 1 WCU provides 1 write per second up to 1 KB.
Transactional Write : 2 WCUs provide 1 transactional write per second up to 1 KB. WCUs required = ( ⌈ Item Size in KB 1 KB ⌉ ) × Writes per second × Write Type Factor \text{WCUs required} = \left( \lceil \frac{\text{Item Size in KB}}{1\text{ KB}} \rceil \right) \times \text{Writes per second} \times \text{Write Type Factor} WCUs required = ( ⌈ 1 KB Item Size in KB ⌉ ) × Writes per second × Write Type Factor
Where Write Type Factor = 1.0 for Standard Write, and 2.0 for Transactional Write.
Single-Table Design & Entity Overloading
In relational databases, normalization divides entities into discrete tables ( Customers , Orders , OrderItems ) linked by foreign keys. Querying normalized data requires high-latency table JOIN operations. DynamoDB does not support JOIN operations.
In DynamoDB NoSQL architecture, Single-Table Design consolidates multiple relational entities into a single DynamoDB table. Access patterns dictate the table schema, rather than entity relationships.
Overloading Primary Keys (PK & SK)
Single-table schemas use generic attribute names like PK (Partition Key) and SK (Sort Key) to store disparate record types:
- PK (Partition Key) SK (Sort Key) Type Entity Attributes
- CUST#1001 METADATA Customer Name: "Jane Doe" , Email: "jane@example.com"
- CUST#1001 ORDER#2026-08-13#9901 Order Total: 149.50 , Status: "SHIPPED"
- CUST#1001 ORDER#2026-08-13#9902 Order Total: 89.00 , Status: "PENDING"
- ORDER#9901 ITEM#SKU-772 OrderItem Qty: 2 , UnitPrice: 49.75
- Item Collections
An Item Collection consists of all items in a table (or index) that share the exact same Partition Key ( PK ) value across different Sort Keys ( SK ).
Using a single Query API call with key condition expressions (such as PK = :cust_id AND SK BEGINS_WITH("ORDER#") ), an application can fetch matching order items in a single HTTP network round-trip ; omit the sort-key condition when the result must also include the profile item with single-digit millisecond latency.
Secondary Indexes: Global Secondary Indexes (GSIs) vs. Local Secondary Indexes (LSIs)
When access patterns require querying attributes other than the base table's primary key, DynamoDB secondary indexes create alternative projection views of table data.
- Architectural Aspect Local Secondary Index (LSI) Global Secondary Index (GSI)
- Partition Key (PK) Must be IDENTICAL to base table PK Can be DIFFERENT from base table PK
- Sort Key (SK) Must be DIFFERENT from base table SK Optional (can be any scalar attribute)
- Creation Timing ONLY at table creation time Any time (table creation or post-creation)
- Deletion CANNOT be deleted after creation Can be added or deleted at any time
Throughput Allocation Consumes base table provisioned RCUs/WCUs Has dedicated provisioned or on-demand RCUs/WCUs
Consistency Support Supports Eventually OR Strongly Consistent reads Supports ONLY Eventually Consistent reads
Item Collection Limit Max 10 GB total per item collection across base & LSI Unlimited storage across partitions
Write Throttling Impact Shared with base table If GSI is throttled, base table writes are throttled!
Critical Exam Warning: GSI Write Backpressure
Because DynamoDB asynchronously replicates base table writes to GSIs, if a GSI runs out of provisioned Write Capacity Units (WCUs), the GSI rejects updates. To preserve data consistency between base table and index, DynamoDB will throttle writes on the base table itself , throwing a ProvisionedThroughputExceededException on base table write operations! Always ensure GSI write capacity matches or exceeds base table write velocity.
Index Projections
When creating secondary indexes, you choose which attributes to project into the index:
KEYS_ONLY : Only base table PK, SK, and index keys are projected. Lowest storage cost.
INCLUDE : Keys plus specifically selected non-key attributes are projected.
ALL : Every attribute from the base table is projected into the index. Highest storage cost, but avoids base table fetches.
- Advanced Data Engineering Features: DAX, Streams & Global Tables
- DynamoDB Accelerator (DAX)
DAX is a fully managed, highly available in-memory write-through cache designed specifically for DynamoDB. It reduces read latencies from single-digit milliseconds to microseconds ( < 1 ms ).
Architecture : Runs inside a Private Subnet in your VPC. Client SDKs replace standard DynamoDB endpoints with DAX cluster endpoints.
Cache Mechanisms : Item cache (caches individual GetItem calls) and Query cache (caches Query and Scan result sets).
DynamoDB Streams & Change Data Capture (CDC)
DynamoDB Streams captures an item-level, time-ordered sequence of write operations ( INSERT , MODIFY , REMOVE ) on a table, retaining records for 24 hours .
Stream View Types:
KEYS_ONLY : Only key attributes of modified items.
NEW_IMAGE : Entire item as it appears after modification.
OLD_IMAGE : Entire item as it appeared before modification.
NEW_AND_OLD_IMAGES : Both new and old item representations.
Common stream consumption pattern: DynamoDB Stream trigger → \rightarrow → AWS Lambda → \rightarrow → Amazon OpenSearch Service for full-text search indexing or S3 Data Lake landing.
DynamoDB Global Tables
Global Tables provides multi-region, multi-active replication for globally distributed applications. Leveraging underlying DynamoDB Streams, writes to any regional table replica are asynchronously replicated across selected AWS regions with typical latency under one second.
Time to Live for Lifecycle Expiration
DynamoDB Time to Live (TTL) expires items from a table using one designated attribute containing a Unix epoch timestamp in seconds as a Number. An expired timestamp makes an item eligible for background deletion; deletion is not immediate and AWS generally completes it within a few days. Applications should filter expired items if they must stop returning them at the deadline.
TTL deletion does not consume write capacity in the Region where the expiration occurs. It appears in DynamoDB Streams as a service deletion, which can trigger cleanup of derived data. With Global Tables, the initial expiration does not consume local write units, but each replicated delete consumes replicated write capacity in replica Regions. TTL is appropriate for sessions, idempotency records, or other retention windows; use an explicit conditional/application delete when the business requires exact-time removal.
- Code Example: Python Boto3 Single-Table Querying and GSI Creation import boto3
- from boto3.dynamodb.conditions import Key
- # Initialize DynamoDB resource
- dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
- table = dynamodb.Table('ECommerceSingleTable')
def query_customer_orders(customer_id: str, start_date: str):
- """
- Queries single-table schema for a specific customer's orders
placed after start_date using KeyConditionExpression.
"""
try:
- response = table.query(
- KeyConditionExpression=Key('PK').eq(f"CUST#{customer_id}") &
- Key('SK').begins_with(f"ORDER#{start_date}"),
- ConsistentRead=True # Strongly consistent read on base table
- )
- items = response.get('Items', [])
- print(f"Retrieved {len(items)} order records for customer {customer_id}")
- return items
except Exception as e:
- print(f"Error querying DynamoDB table: {str(e)}")
- raise e
def query_gsi_by_order_status(status: str):
"""
Queries Global Secondary Index (GSI1) to fetch all orders by status across customers.
"""
try:
- response = table.query(
- IndexName='GSI1-Status-Date-Index',
- KeyConditionExpression=Key('GSI1_PK').eq(f"STATUS#{status}"),
- ConsistentRead=False # GSIs only support eventually consistent reads
- )
- return response.get('Items', [])
except Exception as e:
- print(f"GSI Query Failed: {str(e)}")
- raise e