Study guide
1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda
Key Takeaways Explain This Topic AI
DynamoDB Streams captures time-ordered sequences of item-level modifications (INSERT, MODIFY, REMOVE) with 24-hour strictly ordered retention per partition key, enabling asynchronous Change Data Capture (CDC).
Kinesis Data Streams for DynamoDB extends retention up to 365 days and supports additional fan-out consumers and Amazon Data Firehose delivery without consuming DynamoDB table read capacity units.
AWS Lambda Event Source Mapping (ESM) reads streams using polling threads, supporting batch sizes (1 to 10,000 records), batch windowing (up to 300s), parallelization factor (up to 10 concurrent Lambda invocations per shard), and bisect-on-function-error.
Idempotent processing using unique message identifiers or DynamoDB conditional writes prevents duplicate side-effects caused by Lambda retries during stream processing failures.
Last updated: August 2026
- 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda
- Change Data Capture (CDC) Architecture
In event-driven cloud architectures, capturing database modifications as a real-time event stream is known as Change Data Capture (CDC) . CDC enables downstream decoupled architectures—such as invalidating caches, updating search indexes in OpenSearch, updating data warehouse tables, or triggering serverless workflows—without performing expensive periodic database polling.
AWS provides native CDC capabilities for Amazon DynamoDB through two streaming integrations: DynamoDB Streams and Amazon Kinesis Data Streams for DynamoDB .
DynamoDB Streams Fundamentals
DynamoDB Streams captures a time-ordered sequence of item-level modifications (INSERT, MODIFY, REMOVE) occurring within a DynamoDB table. The log entries are written asynchronously to the stream within milliseconds of the database update.
Key Characteristics of DynamoDB Streams
Ordering & Scope : Record modification events are strictly ordered per item primary key . Events for distinct items may arrive in arbitrary order.
Shard Lifecycle : Stream records are organized into shards, which represent a log of table modifications. Shards auto-split and merge in tandem with underlying DynamoDB physical partition scaling.
24-Hour Fixed Retention : Stream records are retained for exactly 24 hours . Data older than 24 hours is automatically trimmed and permanently lost. Retention period is non-configurable.
Zero Impact on Read Capacity : Polling or reading records from DynamoDB Streams does NOT consume table Read Capacity Units (RCUs).
Stream View Types
When enabling DynamoDB Streams, you must specify the Stream View Type , which defines what information is captured in each stream record:
- Stream View Type Content Included in Event Payload Use Case
- KEYS_ONLY Only the key attributes of the modified item Triggering lightweight cache invalidation
- NEW_IMAGE The entire item as it appears after modification Forwarding new state to downstream analytical sinks
- OLD_IMAGE The entire item as it appeared before modification Auditing deleted records or state cleanups
NEW_AND_OLD_IMAGES Both the new and old item state images Calculating state diffs, operational metrics, or audit logs
Kinesis Data Streams Integration for DynamoDB
For advanced analytics pipelines, AWS allows streaming DynamoDB table changes directly into Amazon Kinesis Data Streams instead of standard DynamoDB Streams.
Architectural Comparison: DynamoDB Streams vs. Kinesis Data Streams
- Feature / Limit Native DynamoDB Streams Kinesis Data Streams for DynamoDB
- Data Retention Fixed 24 hours (non-extendable) Up to 365 days (8,760 hours)
- Max Concurrent Readers 2 readers per shard max Up to 20 readers per shard with Enhanced Fan-Out (EFO)
- Delivery Sinks Custom consumers (AWS Lambda, KCL) Direct delivery to S3/Redshift via Amazon Data Firehose
Pricing Model No charge to enable the stream or call its read API; downstream consumers can have separate charges Pay for Kinesis capacity and payload processing
Multi-Region Replication Used internally by DynamoDB Global Tables Regional stream; cross-account or cross-Region delivery requires an explicit architecture
Decision Guidance: Choose native DynamoDB Streams for lightweight, low-cost Lambda triggers requiring <24h retention. Choose Kinesis Data Streams for DynamoDB when requiring retention >24h, fan-out to >2 consumers, or direct ingestion into Amazon S3 via Firehose.
AWS Lambda Event Source Mapping (ESM)
AWS Lambda integrates with streaming sources (DynamoDB Streams, Kinesis, MSK) via an internal infrastructure component called the Event Source Mapping (ESM) . The ESM runs a polling loop on behalf of your serverless architecture, fetching batches of records from stream shards and invoking your Lambda function synchronously.
Tuning ESM Batching & Concurrency
Batch Size ( BatchSize ) : Configures the maximum number of records (1 to 10,000) passed to a single Lambda invocation.
Maximum Batching Window ( MaximumBatchingWindowInSeconds ) : Sets the maximum time (0 to 300 seconds) Lambda waits to gather records before invoking the function. If BatchSize is not met, Lambda invokes the function as soon as the window expires.
Parallelization Factor ( ParallelizationFactor ) : By default, Lambda processes each stream shard with 1 concurrent invocation to guarantee ordered processing. Setting ParallelizationFactor (from 1 to 10) divides a single shard into concurrent sub-streams grouped by partition key, allowing up to 10 concurrent Lambda invocations per shard while preserving strict partition key ordering!
Resilient Error Handling & Dead Letter Destinations
When a Lambda function throws an unhandled exception while processing a stream batch, the ESM halts shard processing and retries the entire batch repeatedly until records expire (24 hours). This causes a blocked shard (poison pill block).
To prevent shard blocking, configure advanced ESM error controls:
BisectBatchOnFunctionError : When set to True , if a batch invocation fails, Lambda automatically splits the failing record batch into two smaller sub-batches and retries them separately. It recursively bisects until the exact failing record is isolated.
MaximumRetryAttempts & MaximumRecordAgeInSeconds : Limits how many times Lambda retries a failing record batch or drops records older than a set threshold.
On-Failure Destination ( DestinationOnFailure ) : Automatically routes metadata about dropped or failed records directly to an Amazon SQS queue or Amazon SNS topic for offline inspection and replay.
Idempotent Stream Processing Patterns
Because stream processing guarantees at-least-once delivery , network retries or Lambda function re-invocations can deliver the same stream event multiple times. Lambda functions MUST be designed to be idempotent (processing the same event multiple times produces the exact same side-effect).
Idempotency Techniques
Deterministic Unique Keys : Use the DynamoDB Streams eventID or business transaction ID as an idempotency key.
DynamoDB Conditional Writes : Maintain a dedicated tracking table in DynamoDB. Before processing, insert the eventID using a conditional expression: attribute_not_exists(event_id) If the insert succeeds, process the payload. If a ConditionalCheckFailedException is raised, skip processing as it is a duplicate event.
- Code Example: Idempotent Python Lambda Handler for DynamoDB CDC import json
- import os
- import time
- import boto3
- from botocore.exceptions import ClientError
- dynamodb = boto3.resource('dynamodb')
- PROCESSED_EVENTS_TABLE = os.environ.get('PROCESSED_EVENTS_TABLE', 'ProcessedStreamEvents')
- table = dynamodb.Table(PROCESSED_EVENTS_TABLE)
def lambda_handler(event, context):
- records = event.get('Records', [])
- print(f"Processing batch of {len(records)} DynamoDB stream records.")
for record in records:
- event_id = record['eventID']
- event_name = record['eventName'] # INSERT, MODIFY, REMOVE
# Attempt idempotent registration
try:
- table.put_item(
- Item={'event_id': event_id, 'ttl': int(time.time()) + 86400},
- ConditionExpression='attribute_not_exists(event_id)'
- )
except ClientError as e:
if e.response['Error']['Code'] == 'ConditionalCheckFailedException':
- print(f"Duplicate event detected: {event_id}. Skipping execution.")
- continue
- raise e
- # Parse CDC Image Payload
- ddb_data = record.get('dynamodb', {})
- new_image = ddb_data.get('NewImage', {})
- old_image = ddb_data.get('OldImage', {})
- print(f"Executing CDC event {event_name} for ID {event_id}")
- # Perform downstream integration logic here (e.g. forward to S3/OpenSearch)