Study guide
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning
Key Takeaways Explain This Topic AI
Kinesis Data Streams (KDS) provides real-time, sub-second continuous ingestion requiring custom consumer logic (KCL/Lambda), whereas Amazon Data Firehose delivers near-real-time buffered batches directly to data lakes (S3, Redshift, OpenSearch) without consumer management.
Provisioned shard throughput remains 1 MB/s or 1,000 records/sec for sustained writes and 2 MB/s for shared reads; a stream can retain its 1 MiB default record limit or be configured for intermittent records up to 10 MiB.
Partition key selection determines hash key distribution using MD5 hashes across the 128-bit hash key space; poor partition key selection leads to hot shards and ProvisionedThroughputExceededException.
Amazon Data Firehose can transform records with AWS Lambda and convert JSON to Apache Parquet or ORC before delivery; buffering ranges and defaults depend on the destination (Amazon S3 supports 1–128 MiB and 0–900 second hints).
Last updated: August 2026
- 1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning
- Streaming Ingestion Fundamentals on AWS
In enterprise data engineering, streaming ingestion pipelines process high-velocity data generated continuously by thousands of distributed sources—such as IoT sensors, application log forwarders, clickstream trackers, and financial transaction feeds. AWS provides two foundational services for streaming data ingestion: Amazon Kinesis Data Streams (KDS) and Amazon Data Firehose (formerly Kinesis Data Firehose).
While both services ingest streaming payloads, they serve fundamentally different architectural paradigms: KDS is a low-latency, stateful event store built for real-time custom processing, whereas Amazon Data Firehose is a fully managed, serverless stream delivery service designed to load streaming data directly into storage sinks, analytical data stores, and third-party destinations.
- Amazon Kinesis Data Streams (KDS) Architecture
- Capacity Modes: Provisioned vs. On-Demand
Kinesis Data Streams operates in one of two capacity modes:
Provisioned Mode : You explicitly specify the number of shards allocated to the stream. Each shard provides a fixed unit of read and write capacity. You are billed hourly per shard regardless of data throughput.
On-Demand Mode : AWS automatically manages and scales stream capacity in response to traffic. Current choices include On-demand Standard and the account-level On-demand Advantage mode, which supports warm throughput for forecast peaks. Regional stream quotas vary and reach well beyond the older 200 MB/s default, so design from the current quota page instead of memorizing one obsolete stream-wide ceiling.
Shard Limits & Throughput Dynamics
A shard is the base throughput unit of a Kinesis Data Stream. Understanding exact shard throughput limits is critical for the AWS Data Engineer Associate exam:
- Dimension Provisioned Shard Capacity Limit
- Write Throughput (Ingress) 1 MB/s or 1,000 records/sec (whichever threshold is hit first)
- Standard Read Throughput (Egress) 2 MB/s total, shared across all standard consumers polling via GetRecords
Enhanced Fan-Out Read Throughput 2 MB/s per consumer , dedicated per shard via HTTP/2 push ( SubscribeToShard )
Maximum Record Size 1 MiB by default; configurable up to 10 MiB for intermittent large records (sustained shard rates do not increase)
Data Retention Period Default: 24 hours . Configurable up to 365 days (8,760 hours)
If a producer attempts to write data that exceeds 1 MB/s or 1,000 records/sec on a single shard, KDS returns a ProvisionedThroughputExceededException . Producers must handle this using exponential backoff with randomized jitter, or scale shard capacity.
Partitioning & MD5 Hash Distribution
When a producer writes a record using the PutRecord or PutRecords API, it must supply a string-based Partition Key . KDS applies an MD5 hash algorithm to the partition key, mapping it to a 128-bit integer space ranging from 0 to 2^128 - 1 ( 340,282,366,920,938,463,463,374,607,431,768,211,455 ). Each shard in a stream is assigned a contiguous range of hash keys. Partition Key ("user_8492") ---> MD5 Hash ---> 128-bit Hash Key Range ---> Assigned Shard ID
Hot Shard Mitigation Strategy: If partition keys are poorly chosen (e.g., using a constant value, region code, or timestamp rounded to the hour), a disproportionate percentage of hash values will map to a single shard. This causes a hot shard , triggering ProvisionedThroughputExceededException errors while adjacent shards remain underutilized.
Best Practice : Use high-cardinality attributes as partition keys (e.g., UUID, transaction ID, device serial number).
Explicit Hash Key Override : Producers can override partition key hashing by supplying an ExplicitHashKey parameter to directly target specific hash ranges during resharding operations ( SplitShard or MergeShards ).
Consumer Architecture: Standard vs. Enhanced Fan-Out (EFO)
Standard Consumers : Use short or long polling via HTTP GET requests ( GetRecords ). All standard consumers attached to a shard share the 2 MB/s egress throughput budget. If 5 consumers read from the same standard shard, each receives an average throughput of ~400 KB/s, increasing read latency (typically 200 ms to 1,000 ms).
Enhanced Fan-Out (EFO) Consumers : Introduced for sub-second, low-latency topologies. EFO uses HTTP/2 streaming ( SubscribeToShard ) to push data from KDS to consumers with typical latency under 70 ms. Each EFO consumer gets a dedicated 2 MB/s egress bandwidth pipe per shard without competing with other registered consumers.
- Amazon Data Firehose Architecture
- Fully Managed Delivery & Buffering Mechanics
Amazon Data Firehose is a serverless ingest service that eliminates custom consumer code, shard provisioning, and scaling management. Data Firehose buffers incoming streaming data before writing it to downstream destinations.
Buffering behavior is governed by two configurable parameters:
Buffer Size : A destination-specific hint. Amazon S3 and Redshift destinations accept 1–128 MiB (default 5 MiB ).
Buffer Interval : Also destination-specific. Amazon S3 and Redshift accept 0–900 seconds (default 300 seconds ). Zero buffering is unavailable when dynamic partitioning is enabled, and S3 backup uses a nonzero interval.
Firehose flushes buffered records to the destination sink as soon as either condition is satisfied first. For example, if Buffer Size is set to 64 MB and Buffer Interval is set to 300 seconds, Firehose will flush data immediately upon accumulating 64 MB of records, or every 5 minutes if data volume is low.
Destination Sinks & Native Integrations
Firehose natively delivers data to standard AWS and third-party endpoints:
Amazon S3 : Raw or transformed object delivery.
Amazon Redshift : Automatically buffers data into S3 and executes an intermediate COPY command into Redshift staging tables.
Amazon OpenSearch Service : Indexing log and metric streams.
Third-Party Analytics Sinks : Direct HTTPS delivery to Snowflake, Datadog, Splunk, and New Relic.
Inline Transformations & Format Conversion
Firehose provides two powerful serverless data transformation capabilities prior to delivery:
AWS Lambda Transformation : Firehose invokes an AWS Lambda function to execute custom inline ETL (e.g., stripping PII, decoding base64 payloads, filtering events). Lambda returns processed records back to Firehose ( Ok , Dropped , or ProcessingFailed ).
Serverless Format Conversion : Firehose can convert incoming raw JSON payloads directly into columnar storage formats— Apache Parquet or Apache ORC —before writing to Amazon S3. Firehose references table schemas defined in the AWS Glue Data Catalog to perform this conversion automatically without spinning up EMR or Glue ETL jobs, significantly improving Athena query performance and lowering storage costs.
Dynamic Partitioning & S3 Prefix Formatting
Firehose supports Dynamic Partitioning , allowing records to be written to custom S3 bucket prefixes evaluated dynamically from record attributes or timestamp parameters. For example, log events containing a customer_id and event_type can be dynamically partitioned in S3: s3://telemetry-data-lake/customer_id=!{partitionKeyFromQuery:customer_id}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/
Architectural Decision Matrix: KDS vs. Data Firehose
Capability / Feature Amazon Kinesis Data Streams (KDS) Amazon Data Firehose
Primary Purpose Real-time event streaming & custom stateful processing Managed stream ingestion & loading into data sinks
Latency Sub-second (70ms with EFO; 200ms standard) Destination-specific buffering hints; S3 and Redshift allow 0–900 seconds, while dynamic partitioning adds buffering
- Storage Retention 24 hours to 365 days (replayable stream) No persistent storage (transient buffering only)
- Consumer Model Pull/Push to custom apps (KCL, Lambda, Flink) Managed push to S3, Redshift, OpenSearch, Splunk
- Format Conversion Manual (handled in consumer code) Serverless JSON to Parquet/ORC via Glue Catalog
- Capacity Management Shard management (Provisioned or On-Demand) Completely serverless, automatic scaling
- Code Example: Kinesis Data Streams Boto3 Producer with Retry Logic import json
- import random
- import time
- import boto3
- from botocore.exceptions import ClientError
- kinesis_client = boto3.client('kinesis', region_name='us-east-1')
- STREAM_NAME = 'telemetry-ingestion-stream'
def send_telemetry_event(device_id: str, payload: dict):
- # High-cardinality partition key avoids hot shards
- partition_key = f"device#{device_id}"
- data_bytes = json.dumps(payload).encode('utf-8')
- max_retries = 5
- base_delay = 0.1 # 100ms
for attempt in range(max_retries):
try:
- response = kinesis_client.put_record(
- StreamName=STREAM_NAME,
- Data=data_bytes,
- PartitionKey=partition_key
- )
print(f"Record ingested successfully! ShardId: {response['ShardId']}, SequenceNo: {response['SequenceNumber']}")
return response
except ClientError as e:
error_code = e.response['Error']['Code']
if error_code == 'ProvisionedThroughputExceededException':
- # Exponential backoff with full jitter
- sleep_time = random.uniform(0, base_delay * (2 ** attempt))
- print(f"Throughput exceeded. Retrying in {sleep_time:.3f} seconds (Attempt {attempt + 1})...")
- time.sleep(sleep_time)
else:
- raise e
- raise RuntimeError("Failed to ingest record after maximum retries due to throughput limits.")