Study guide
1.4 Streaming Distribution, Replay & Processing State
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
1.4 Streaming Distribution, Replay & Processing State
Key Takeaways Explain This Topic AI
Fan-out sends one stream to independent consumers, while fan-in combines multiple sources; queues, streams, and delivery services preserve different ordering and replay guarantees.
Replay requires retained source records, a deliberate starting position, idempotent sinks, and checkpoint control; retrying a delivery service is not the same as replaying a durable stream.
Stateless transforms can scale horizontally from each record alone, whereas stateful operations such as windows, joins, and deduplication require checkpoints and event-time policies.
Backpressure must be absorbed with scaling, buffering, rate control, or load shedding; unbounded retries merely move congestion and can amplify a failure.
Last updated: August 2026
1.4 Streaming Distribution, Replay & Processing State
Service comparisons are easier when you first model the flow semantics . A source produces records, a transport orders and retains them, consumers maintain progress, and sinks apply side effects. Most exam failures come from selecting a service with the right name but the wrong delivery or recovery behavior.
Fan-in and fan-out
Fan-in combines many producers or upstream streams into one processing path. Thousands of devices can write to a Kinesis stream, or multiple EventBridge rules can target one queue. The risk is a hot partition or a consumer that cannot match the combined rate.
Fan-out gives multiple independent consumers the same logical events. Kinesis enhanced fan-out gives registered consumers dedicated per-shard read throughput. Kafka consumer groups behave differently: consumers in the same group divide partitions, while separate groups each receive the topic independently. Amazon SNS can publish one message to several SQS queues, giving each downstream application its own backlog and retry policy.
Requirement Useful pattern Key limitation
Several apps independently replay the same ordered events Kinesis or MSK with separate consumer checkpoints Retention, partitioning, and consumer lag must be managed
One work item processed by one of many workers SQS competing consumers Standard queues do not provide a replayable ordered log
Managed delivery to S3 or Redshift Amazon Data Firehose Delivery buffering is not a general-purpose consumer log
One event routed to several isolated workloads SNS to separate SQS queues or EventBridge targets Each target needs its own failure handling
Ordering scope
Ordering is almost never global. Kinesis orders records within a shard; Kafka orders within a topic partition; an SQS FIFO queue orders within a message group. Increasing parallelism by adding shards, partitions, or message groups increases throughput but divides the ordering boundary. Choose a partition key that preserves the business sequence—such as account ID—while spreading unrelated entities. A single constant key preserves broad ordering but creates a hot partition.
Replay is not retry
A retry repeats a failed delivery or invocation. A replay deliberately reads retained historical records again from an earlier offset, sequence, or timestamp. Kinesis can start consumers at trim horizon, latest, or a timestamp, subject to retention. Kafka consumer groups can reset offsets while the records remain in the topic. DynamoDB Streams retains records for only 24 hours, so it is a narrow recovery window.
Safe replay requires four controls:
Retain the source long enough to cover detection and repair.
Save checkpoints separately for each consumer application.
Make the sink idempotent with a source event ID, conditional write, merge key, or processed-event table.
Decide whether replay writes to the production sink, a shadow destination, or a corrected partition.
Without idempotency, replay can double-charge a card, duplicate a warehouse row, or send the same notification twice. Exactly-once claims usually describe a bounded service integration; end-to-end correctness still depends on the sink transaction.
Stateful and stateless processing
A stateless transform needs only the current record: parse JSON, redact one field, or convert a timestamp. It scales horizontally because any worker can process any record.
A stateful operation remembers prior events: a five-minute rolling count, stream-table join, session window, fraud sequence, or deduplication set. Managed Service for Apache Flink checkpoints operator state and source positions so a restarted job can recover consistently. Event time and watermarks control how long the system waits for late events; an aggressive watermark improves latency but can exclude late data.
State grows with key cardinality and retention. Use state time-to-live where valid, monitor checkpoint duration, and avoid a key that concentrates most state on one task.
Backpressure and rate limits
Backpressure appears when arrival rate exceeds processing rate. Iterator age, Kafka consumer lag, queue depth, and Firehose delivery freshness expose the symptom. Remedies include scaling consumers, increasing partitions, batching calls, compacting small records, or reducing work per event. Producers calling DynamoDB, RDS, or external APIs must respect target limits with bounded exponential backoff and jitter.
An unbounded retry loop is not resilience: it holds capacity, duplicates traffic, and increases the target's load. After bounded retries, move the failed unit to a durable failure destination with enough context to repair and replay it. Alert on backlog age , not only count, because a small backlog of very old records can be more serious than a brief burst.
Exam decision sequence
Ask: What is the ordering boundary? Does every consumer need every record? How long must records be replayable? Who owns the checkpoint? Is the operation stateful? Can the sink tolerate duplicates? Those answers select the architecture more reliably than latency alone.