Study guide
Streaming Analytics with Amazon Kinesis and Amazon MSK
AWS Certified Solutions Architect - Professional SAP-C02
Track your progress as you read
Finished sections get a check mark.
1 Introduction and SAP-C02 Exam Facts
Current SAP-C02 Exam Facts Exam Format, Scoring, and Retakes Candidate Profile and Four Exam Domains
2 Multi-Account Governance
Multi-Account Strategy, Account Topologies, and OU Hierarchies Service Control Policies, Inheritance, and Permission Guardrails Control Tower, Landing Zones, Central Logging, and Resource Sharing
3 Hybrid and Inter-VPC Connectivity
Transit Gateway, VPC Connectivity, and PrivateLink Direct Connect, VPN, and Hybrid Redundancy Hybrid DNS with Route 53 Resolver Segmentation, Traffic Monitoring, and Service Endpoints
4 Identity Federation and Cross-Account Access
IAM Identity Center, External IdPs, and Workforce Federation Cross-Account Roles, STS, and Resource Policies Permissions Boundaries, Delegated Administration, and Least Privilege at Scale
5 Data Protection, Encryption, and Compliance
KMS, Multi-Region Keys, Envelope Encryption, and Certificates Encryption in Transit, Security Groups, NACLs, and VPC Endpoints Centralized Audit, Security Hub, GuardDuty, Inspector, and Access Analyzer
6 Deployment, Compute, Containers, and Serverless
Infrastructure as Code, CI/CD, Change Management, and Rollback EC2 Families, Auto Scaling, and Managed Compute ECS, EKS, Fargate, and ECR at Enterprise Scale Lambda, Step Functions, API Gateway, and Configuration Management
7 Storage, Caching, and Edge Performance
S3 Storage Classes, Object Lock, Replication, and Access Paths EBS, EFS, FSx Family, and Storage Gateway CloudFront Behaviors, Origin Access, Edge Compute, and Global Accelerator ElastiCache, MemoryDB, DAX, and Buffering versus Cache versus Replicas
8 Purpose-Built and Global Databases
Aurora Storage, Replicas, and Global Database DynamoDB Keys, Global Tables, and DAX Purpose-Built Database Selection
9 Integration, Messaging, and Streaming
API Gateway Architectures Decoupled Messaging with Amazon SQS and Amazon SNS Event-Driven Design with Amazon EventBridge Streaming Analytics with Amazon Kinesis and Amazon MSK
10 Resilience and Disaster Recovery
Disaster Recovery Patterns Route 53 Routing and Application Recovery Controller Backup, Replication, and Service Quotas High Availability, Failover, Chaos, and Self-Healing
11 Operational Excellence and Automation
CloudWatch Centralized Telemetry Distributed Tracing with AWS X-Ray Systems Manager Automation Deployment Strategies and Auto-Remediation
12 Cost Optimization and Visibility
Cost Explorer, Budgets, CUR, and Allocation Tags Savings Plans, Reserved Instances, Coverage, and Sharing Spot, Mixed Auto Scaling, Compute Optimizer, and Storage Lens Data Transfer Modeling: NAT, Endpoints, CloudFront, and Hybrid
13 Improve Security and Performance
Secrets, Least Privilege Audits, and Traceability Config Rules, Patching, Backup, and Regulatory Controls Performance Tuning for Existing Workloads
14 Migration Assessment and Wave Planning
Seven Migration Strategies (7 Rs) and Workload Selection Application Discovery, Migration Hub, and Migration Evaluator Wave Planning, Prioritization, and Assessment Governance
15 Migration Tools and Data Transfer
AWS Application Migration Service (MGN) AWS DMS, SCT, and Change Data Capture Large-Scale Data Transfer and Landing-Zone Prerequisites
16 Target Architecture, Modernization, and Cutover
Selecting Compute, Storage, and Database Platforms for Migrated Workloads Container Modernization: ECS, EKS, Fargate, ECR, and Strangler Fig Serverless, Decoupling, Purpose-Built Databases, and Integration Services Cutover, Rollback, and Unscored Emerging AI Topics
Streaming Analytics with Amazon Kinesis and Amazon MSK
Key Takeaways Explain This Topic AI
Amazon Kinesis Data Streams is a replayable, shard-based stream with independent consumers; Amazon Data Firehose is a delivery service that buffers and lands data without a consumer GetRecords API.
Amazon MSK is managed Apache Kafka: partitions, consumer groups, and the Kafka ecosystem. Choose it when Kafka compatibility matters; choose Kinesis when AWS-native consumers (Lambda, Firehose, Flink) and a simpler shard model matter.
A Kinesis shard provisions 1 MB/s or 1,000 records/s write and 2 MB/s read (classic consumers share that read cap); Kafka partitions are the parallel ordered unit for MSK.
Amazon Managed Service for Apache Flink runs stateful stream jobs (windows, joins) against Kinesis or MSK; it is not a drop-in replacement for an SQS worker.
Use SQS for discrete work items that competing consumers delete; use a stream for clickstream and other ordered, multi-consumer, replay-within-retention workloads. Lambda MaximumBatchingWindowInSeconds is 0–300 seconds.
Last updated: September 2026
Why streams are not queues
SAP-C02 Task 2.5 asks you to design large-scale access patterns and to apply buffering as a performance pattern. A buffer might be Amazon SQS (Task 2.4 decoupling) or a stream . The Professional mistake is using a queue for clickstream that five teams must read, or using a stream for order capture that should be deleted after one successful worker. Independent SAP-C02 study material by ExamFinch treats Amazon Kinesis Data Streams , Amazon Data Firehose (formerly Kinesis Data Firehose), Amazon Managed Streaming for Apache Kafka (Amazon MSK) , and Amazon Managed Service for Apache Flink as a decision set.
HarborCart's mobile app emits millions of navigation events per hour. Product, personalization, and data-science teams all want the same feed. Warehouse pick jobs remain on SQS FIFO. Those are different problems.
Kinesis Data Streams versus Data Firehose versus MSK
Amazon Kinesis Data Streams stores records in shards . Producers PutRecord / PutRecords . Consumers read with GetRecords (shared throughput) or enhanced fan-out SubscribeToShard (dedicated throughput). Records remain for a retention period : default 24 hours , extendable to 8,760 hours (365 days) . Retention above 24 hours is billed extra. Multiple consumer applications can read the same shard independently; a crash recovers by checkpointing within retention. That is why clickstream belongs here.
Provisioned mode: you choose shard count. On-demand mode: AWS scales with traffic; you can configure warm throughput when you forecast a peak. AWS documents that on-demand aggregate read capacity scales with write throughput and recommends enhanced fan-out when more than one consumer application must read in parallel without sharing the 2 MB/s classic read budget.
Amazon Data Firehose is a delivery stream . Producers (or a Kinesis data stream, or an MSK topic) send records; Firehose buffers by size (MB) and interval (seconds) , optionally invokes Lambda to transform, optionally converts to Parquet/ORC, and loads Amazon S3 , Amazon Redshift , Amazon OpenSearch Service , Splunk , Snowflake , HTTP endpoints, and similar destinations. There is no consumer application API to replay Firehose the way you replay a Kinesis shard. If the only need is "land clickstream in the data lake," Firehose alone is enough. If product and fraud must react in seconds and data-science still wants S3, use Data Streams plus a Firehose consumer (or Flink) rather than Firehose alone.
Amazon MSK runs Apache Kafka . Topics split into partitions . A record key hashes to a partition; order is per partition . Consumer groups share work: each partition is consumed by one member of the group, while separate groups independently reread the topic (within Kafka retention). Use MSK when HarborCart already has Kafka clients, Kafka Connect , compacted topics, transactions, or a requirement to stay Kafka-portable. MSK Serverless reduces broker sizing; provisioned clusters expose Kafka configuration, storage, and throughput knobs. MSK Replicator and MSK Connect appear in hybrid and pipeline stems. Lambda, Firehose, EventBridge Pipes, and Flink all integrate with MSK.
- Need Kinesis Data Streams Amazon Data Firehose Amazon MSK
- Multiple independent real-time consumers Yes No GetRecords consumers Yes (consumer groups)
- Replay within retention Yes (24 h–365 d) Not a stream replay API Yes (Kafka retention)
- Managed load to S3/OpenSearch/Splunk As a consumer of Streams Primary job Firehose integration or Connect
- AWS Lambda event source mapping Yes Not as a stream ESM Yes (MSK ESM)
- Kafka protocol / Connect / compact topics No No Yes
- Simplest AWS-native clickstream Often yes Yes if lake-only If Kafka is already the standard
- Shards versus partitions
A Kinesis shard is a capacity unit. AWS documents each shard as supporting up to 1,000 records per second for writes and 1 MB/s write (including partition keys), five read transactions per second, and 2 MB/s total read for classic consumers. Scale by resharding (split/merge) or by on-demand. The partition key chooses the shard; a hot key (all events with key web ) concentrates on one shard and throttles ( ProvisionedThroughputExceeded ). HarborCart uses sessionId or shopperId as the partition key so clickstream spreads while one shopper stays ordered.
A Kafka partition on MSK is the ordered log segment. Throughput depends on broker sizing, partition count, replica I/O, and consumer lag—not on a single published "1 MB/s per partition" Kinesis-style number. Parallelism of a consumer group cannot exceed the partition count. Compacted topics and exact Kafka semantics are why MSK is not "Kinesis with a different name."
Enhanced fan-out on Kinesis gives each registered consumer up to 2 MB/s per shard dedicated, with AWS documenting up to 20 such consumers on a stream. Classic GetRecords consumers share the shard's 2 MB/s. If product Lambda and a Flink job both trail a busy stream, enhanced fan-out (or separate on-demand headroom) is the Task 2.5 answer.
Managed Service for Apache Flink
Amazon Managed Service for Apache Flink (the current name for the managed Flink service; older stems may still say Kinesis Data Analytics) runs Apache Flink applications. Use it for stateful streaming: event-time windows, joins between clickstream and order events, sessionization, and continuously updated aggregates. Sources include Kinesis Data Streams and Amazon MSK . Studio notebooks (Apache Zeppelin) help interactive analysis, then you deploy a durable application.
Flink is not an SQS replacement. If HarborCart needs "exactly one worker captures payment," that remains FIFO SQS. If HarborCart needs "unique shoppers in a 5-minute window per campaign," that is Flink (or a carefully designed Lambda with tumbling windows—Lambda's Kinesis ESM supports tumbling windows , but complex joins belong in Flink).
Queue versus stream
- Question Queue (SQS) Stream (Kinesis / MSK)
- Who should process a message? Usually one competing consumer Many independent consumers
- After success? Delete the message Advance a checkpoint ; others still read
Replay last week? Only if it is still within retention and not deleted (max 14 days) Yes, within stream retention (Kinesis up to 365 days)
Ordering FIFO per message group Per shard (partition key) or per Kafka partition
Typical HarborCart use Order capture, emails via SNS+SQS, warehouse jobs Clickstream, inventory position ticks, fraud features
Do not put clickstream on SQS expecting three analytics teams to share it: the first worker deletes the message. Do not put payment capture on Kinesis expecting the record to vanish after one consumer; every consumer sees it, and you still need idempotency.
Lambda batch windows and parallelization
Lambda event source mappings poll streams and queues and invoke the function with a batch .
AWS documents MaximumBatchingWindowInSeconds from 0 to 300 (one-second steps). For Kinesis , DynamoDB streams , and SQS , the default window is 0 seconds (invoke as soon as records are available). For Amazon MSK , self-managed Kafka, Amazon MQ, and DocumentDB, the default window is 500 ms . A window trades latency for fewer, fuller invocations —Task 2.5 buffering.
Batch size is the maximum records per invoke (Kinesis up to 10,000 ). Each Kinesis batch is from a single shard . ParallelizationFactor (1 default to 10 ) lets Lambda process multiple concurrent batches per shard while still preserving order per partition key . When IteratorAge is high and keys are diverse, raising parallelization (and shard count) is the scale lever. A single hot partition key still serializes.
On failure, stream mappings can bisect the batch, retry, and send to an on-failure destination . SQS mappings should use ReportBatchItemFailures so one poison message does not recycle the entire batch. FIFO SQS plus Lambda will not concurrently process two batches of the same message group; that preserves order and caps scale per group.
HarborCart's clickstream Lambda uses a 2-second batching window, batch size 500, parallelization factor 4, and enhanced fan-out. A Flink job on the same stream computes 5-minute unique shoppers. Firehose (as a stream consumer) writes Parquet to S3 for Athena. Order capture never uses this path.
HarborCart clickstream design
Mobile and web producers put events to Kinesis Data Streams on-demand with partition key shopperId .
Retention 7 days so a bad deploy can replay (additional cost above 24 hours).
Lambda (enhanced fan-out) updates a hot personalization cache.
Managed Service for Apache Flink computes session windows.
Firehose delivers to S3.
If a Kafka-based data-science platform must consume with existing Connectors, add MSK as a parallel ingest or replicate—do not pretend Kinesis speaks Kafka.
Warehouse jobs stay on SQS FIFO ; clickstream never shares that queue.
Traps
Firehose when the stem needs multiple real-time consumers or replay.
SQS for multi-team clickstream.
Kinesis for single-worker payment capture that should be deleted.
One hot partition key.
Classic consumers starving each other when enhanced fan-out is the requirement.
A 24-hour Lambda batch window (the maximum is 300 seconds ).
Assuming MSK and Kinesis have the same shard math.
Using Flink for a three-step order workflow that belongs in SQS plus Step Functions.