Study guide
5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication
Google Cloud Certified Professional Data Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Security, IAM, and Data Protection
1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
2 Chapter 2: Reliability and Compliance Architecture
2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack
3 Chapter 3: Cross-Cloud and Federated Governance
3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance
4 Chapter 4: Data Ingestion and Pipeline Planning
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion
5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub
5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow
6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows
7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics
7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management
8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion
8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage
9 Chapter 9: Storage Selection and Cloud Storage Architecture
9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools
10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL
10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery
11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore
11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections
12 Chapter 12: Enterprise Data Warehousing with BigQuery
12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions
13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery
13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling
14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML
14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles
15 Chapter 15: Operations, Orchestration, and Data Governance
15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services
5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication
Key Takeaways Explain This Topic AI
Pub/Sub provides FIFO message ordering within a partition key when enable_message_ordering is enabled on the subscription and publishers attach an identical non-empty orderingKey string attribute to published messages.
Ordering in Pub/Sub is scoped strictly to the ordering key; messages with different ordering keys (or no ordering key) are routed across forwarders independently and processed concurrently without ordering constraints or throughput limits.
When an ordered message fails acknowledgement or times out, Pub/Sub halts delivery of all subsequent messages sharing that same ordering key (Head-of-Line blocking) until the failed message is acknowledged, successfully redelivered, or routed to a Dead-Letter Topic.
Pub/Sub Exactly-Once Delivery (EOD) eliminates duplicate deliveries caused by network timeouts during acknowledgement transmission or subscriber transient restarts, but introduces moderate latency and throughput overhead and cannot prevent duplicate publishing at the source.
Enterprise streaming architectures must implement downstream idempotency patterns—such as deterministic insertId in BigQuery Storage Write API, primary key upserts in Cloud Spanner, or Beam Deduplicate.byKey() transforms in Dataflow—to guarantee end-to-end data integrity.
Last updated: September 2026
5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication
Exam Focus: The Google Cloud Professional Data Engineer exam frequently presents scenarios involving in-order stream processing , handling poison-pill messages that cause Head-of-Line (HoL) blocking , evaluating the architectural boundaries of Exactly-Once Delivery (EOD) , and selecting the correct downstream idempotency pattern across BigQuery, Cloud Spanner, and Cloud Dataflow.
Distributed messaging systems must balance throughput, latency, and consistency. By default, Cloud Pub/Sub operates as a highly available, out-of-order, at-least-once messaging service capable of processing hundreds of millions of messages per second across global regions. However, mission-critical business processes—such as financial ledger balance updates, order lifecycle transitions, and database Change Data Capture (CDC) replication—mandate strict chronological ordering and deduplication. Meeting these demands requires understanding Pub/Sub's ordering keys, exactly-once delivery subscription semantics, and downstream sink deduplication patterns.
1. Architecture of Pub/Sub Ordering Keys
Standard Cloud Pub/Sub does not enforce First-In, First-Out (FIFO) delivery across an entire topic. Forcing total global ordering across a topic would channel all messages through a single coordinator, introducing a severe performance bottleneck that restricts throughput to a few thousand messages per second.
To achieve massive horizontal scalability while guaranteeing strict sequential delivery where required, Cloud Pub/Sub implements Ordering Keys :
The Ordering Key ( orderingKey ): A string attribute assigned to a message by the publisher (e.g., account_id , device_uuid , customer_id ).
Consistent Hash Partitioning: Pub/Sub hashes the orderingKey and assigns all messages sharing that exact key to a specific forwarder server queue in the storage layer.
Subscription-Level Activation: Message ordering is enforced only if the subscription is explicitly created with --enable-message-ordering : gcloud pubsub subscriptions create ordered-orders-sub \
- -topic=orders-topic \
- -enable-message-ordering
Scoped Ordering Isolation: Ordering is guaranteed strictly per ordering key . If publisher P P P sends messages with orderingKey = "User_A" , Pub/Sub delivers all messages for User_A in the exact chronological order they were received by the service. Meanwhile, messages published with orderingKey = "User_B" are processed concurrently on separate forwarders, completely decoupled from User_A . [ Publisher ] ──(Publishes with orderingKey)──>
- │
- (Hash-Based Routing)
- ▼
- ┌─────────────────────────────────────── Pub/Sub Topic ───────────────────────────────────────┐
- │ │
- │ [ Key: Account_101 ] ──> [ M1 ] ──> [ M2 ] ──> [ M3 ] ──> Delivered in strict order (FIFO)│
- │ │
- │ [ Key: Account_102 ] ──> [ M1 ] ──> [ M2 ] ──────────────> Delivered in strict order (FIFO)│
- │ │
- │ [ Key: None (Unordered) ] ───────────────────────────────> Delivered concurrently / OoO │
- └────────────────────────────────────────────────────────────────────────────────────────────┘
- Publisher-Side Sequencing Requirements
For ordering to work end-to-end, publishers must adhere to strict sequencing rules:
Sequential Publish Acks: The publisher application must publish message N + 1 N+1 N + 1 only after receiving a successful publish acknowledgement for message N N N , or it must use the Google Cloud client libraries that implement internal ordered publish buffering.
Client-Side Ordered Batching: When message ordering is enabled in official client libraries (Java, Python, Go), the publisher client buffers messages locally by ordering key. If publishing message M 1 M_1 M 1 fails due to a transient network error, the client library automatically pauses publishing subsequent messages for that ordering key until M 1 M_1 M 1 succeeds or the failure is explicitly handled. Calling resumePublish(orderingKey) is required if an unrecoverable publish error occurs.
2. Head-of-Line (HoL) Blocking and Poison-Pill Failure Handling
While ordering keys guarantee FIFO delivery, they introduce a significant operational vulnerability known as Head-of-Line (HoL) Blocking .
The Mechanics of Head-of-Line Blocking
Suppose a publisher sends messages M 1 , M 2 , M 3 , … , M 5000 M_1, M_2, M_3, \dots, M_{5000} M 1 , M 2 , M 3 , … , M 5000 with orderingKey = "Store_42" :
Pub/Sub delivers M 1 M_1 M 1 to an available worker instance.
The worker encounters a fatal bug (e.g., M 1 M_1 M 1 contains an unparseable payload or malformed barcode string) that causes the consumer process to crash or throw an unhandled exception before acknowledging M 1 M_1 M 1 .
Because ordering is strictly enforced, Pub/Sub refuses to deliver M 2 , M 3 , … , M 5000 M_2, M_3, \dots, M_{5000} M 2 , M 3 , … , M 5000 to any consumer until M 1 M_1 M 1 is successfully acknowledged.
When M 1 M_1 M 1 's acknowledgement deadline ( ackDeadlineSeconds ) expires, Pub/Sub redelivers M 1 M_1 M 1 to another worker.
The new worker also crashes on M 1 M_1 M 1 , repeating the cycle indefinitely.
Result: The entire stream of events for Store_42 is completely frozen, accumulating backlog lag. Crucially, all other ordering keys ( Store_43 , Store_44 ) continue processing normally without interruption. [ Ordered Queue: Store_42 ]
- [ M1: POISON PILL (Fails Ack) ] <=== BLOCKS ENTIRE QUEUE (Head-of-Line Blocking)
- [ M2: Valid Message ] (Held in storage; cannot be delivered)
- [ M3: Valid Message ] (Held in storage; cannot be delivered)
- [ M4: Valid Message ] (Held in storage; cannot be delivered)
- [ Ordered Queue: Store_43 ]
- [ M1 ] ──> [ M2 ] ──> [ M3 ] ===> Delivered smoothly in parallel
- Architectural Strategies to Eliminate Head-of-Line Blocking
Data engineers must architect resilience mechanisms to detect and unblock poisoned ordering queues:
Dead-Letter Topics (DLQ) with Retry Thresholds:
Attach a Dead-Letter Topic to the ordered subscription with maxDeliveryAttempts configured (e.g., 5 attempts).
When M 1 M_1 M 1 fails 5 times, Pub/Sub automatically forwards M 1 M_1 M 1 to the designated Dead-Letter Topic and marks M 1 M_1 M 1 acknowledged on the primary subscription.
Once M 1 M_1 M 1 is cleared, Pub/Sub immediately begins delivering M 2 , M 3 , … M_2, M_3, \dots M 2 , M 3 , … , completely resolving the Head-of-Line block automatically.
Consumer-Side Error Isolation (Try/Catch/Acknowledge):
The consumer application wraps payload parsing in robust error handling.
If a message fails schema validation or business logic parsing, the worker writes the raw message along with the stack trace to an external error sink (e.g., BigQuery quarantine table or Cloud Storage error bucket) and immediately acknowledges the message to Pub/Sub.
This prevents the lease from expiring and allows subsequent messages on that ordering key to proceed without delay.
3. Cloud Pub/Sub Exactly-Once Delivery (EOD)
Standard Cloud Pub/Sub subscriptions provide at-least-once delivery . In at-least-once messaging, duplicates arise naturally from distributed networking realities:
Transport ACK Loss: A worker finishes processing message M 1 M_1 M 1 and transmits an ack RPC back to Google Cloud. A transient network router flap drops the ACK packet. Pub/Sub's ackDeadlineSeconds expires, and Pub/Sub redelivers M 1 M_1 M 1 to another worker.
Worker Crashes: A worker processes M 1 M_1 M 1 , writes the result to a database, and abruptly loses power or suffers an Out-Of-Memory (OOM) kill before transmitting the ACK.
In production systems without ordering or special controls, duplicate delivery rates typically range from 0.1% to 1.0% of total volume.
The Exactly-Once Delivery (EOD) Feature
Google Cloud provides native Exactly-Once Delivery at the subscription tier: gcloud pubsub subscriptions create exactly-once-sub \
- -topic=orders-topic \
- -enable-exactly-once-delivery
- How EOD Operates Under the Hood
Distributed Acknowledgement Consensus: When EOD is enabled, Pub/Sub tracks message acknowledgement state across regional consensus logs.
Guaranteed Lease Protection: If an acknowledgement is received and confirmed by Pub/Sub, the service guarantees that the message will never be redelivered to any subscriber on that subscription.
Synchronous Ack Response ( AcknowledgeResult ): In standard subscriptions, ack calls are asynchronous fire-and-forget operations. In EOD subscriptions, the client library waits for a response from the Pub/Sub backend:
SUCCESS : The message was successfully acknowledged; no redelivery will occur.
EXPIRED_ACK_DEADLINE : The acknowledgement arrived after the ack deadline expired, and the message may have already been redelivered to another worker. The client can discard local state or perform rollback.
INVALID_ACK_ID : The message ack ID is invalid or was already acknowledged.
The Critical Architectural Boundary: Publisher Duplicates vs. Delivery Duplicates
Exam Trap — What EOD Does NOT Guarantee: Pub/Sub Exactly-Once Delivery operates strictly within the subscription boundary . It guarantees that a specific Pub/Sub message (identified by a unique message_id ) will be delivered to subscribers exactly once.
It CANNOT prevent publisher duplicate publications. If an upstream payment gateway attempts to publish a transaction, experiences an HTTP 504 gateway timeout, and automatically retries the publish call, Pub/Sub receives two distinct publish requests . Pub/Sub assigns each request a different, unique message_id .
Because they have different message_id s, Pub/Sub treats them as two completely independent messages and delivers both to the EOD subscription. Therefore, end-to-end exactly-once processing always requires downstream deduplication at the storage sink .
4. At-Least-Once vs. Exactly-Once Delivery: Comparative Trade-Off Matrix
Architectural Dimension Standard At-Least-Once Subscription Exactly-Once Delivery (EOD) Subscription
Delivery Guarantee Every message delivered >= 1 time; duplicates occur on network retry Every message delivered exactly 1 time per subscription
Acknowledgement Overhead Asynchronous fire-and-forget; negligible latency Synchronous consensus check; higher ack round-trip latency
Throughput Ceiling Virtually unlimited; horizontally scales to gigabytes/sec Slightly constrained by regional ack consensus coordination
Publisher Retry Handling Cannot detect publisher-side retries; delivers all published messages Cannot detect publisher-side retries; delivers all published messages
Impact on Ordering Keys Messages may be redelivered out of sequence if acks time out Enhances ordering reliability by preventing redundant deliveries
Cost Profile Standard Pub/Sub pricing Additional throughput charge per GB ingested under EOD
Ideal Architectural Fit High-throughput clickstream, IoT telemetry, log ingestion Financial transactions, inventory counts, audit-critical event streams
5. Downstream Deduplication & Idempotency Patterns
Because messaging systems cannot prevent upstream publisher retries, enterprise architectures must enforce idempotency at the storage layer. An operation is idempotent if executing it multiple times produces the exact same final system state as executing it once ( f ( f ( x ) ) = f ( x ) f(f(x)) = f(x) f ( f ( x )) = f ( x ) ). [ Upstream Publisher ] ──(Network Retry)──> [ Pub/Sub: Creates M1 and M2 (Same Payload) ]
- │
- (Delivered to Sink)
- ▼
- ┌────────────────────────────────────── Downstream Idempotent Sinks ──────────────────────────────────────┐
- │ │
- │ 1. BigQuery Storage Write API: Uses deterministic stream offset / row ID (7-day window dedup) │
- │ 2. Cloud Spanner / Bigtable: Uses natural business primary key with INSERT_OR_UPDATE upsert mutations │
- │ 3. Cloud Dataflow: Uses Beam Deduplicate.byKey() stateful transform over sliding time windows │
- └────────────────────────────────────────────────────────────────────────────────────────────────────────┘
- Pattern 1: BigQuery Deduplication Strategies
Storage Write API Default Stream Deduplication:
The BigQuery Storage Write API supports built-in deduplication within a stream.
When appending records, client applications specify a deterministic record hash or business key in the row's insertId or stream offset. BigQuery tracks these IDs within a rolling 7-day window and automatically discards duplicate rows.
SQL Deduplication via Window Functions:
In append-only analytical architectures, all incoming records (including duplicates) are inserted into a raw landing table.
Downstream views or scheduled queries filter duplicates using QUALIFY ROW_NUMBER() : SELECT transaction_id, customer_id, amount, event_timestamp
- FROM `my_project.raw_lake.transactions`
- WHERE DATE(_PARTITIONTIME) = CURRENT_DATE()
- QUALIFY ROW_NUMBER() OVER (
- PARTITION BY transaction_id
- ORDER BY event_timestamp DESC
- ) = 1;
- Pattern 2: Cloud Spanner and Cloud Bigtable Natural Key Upserts
Cloud Spanner: Relational databases enforce uniqueness through primary key constraints. Instead of issuing standard INSERT statements that fail on duplicate keys, the ingestion application applies an INSERT_OR_UPDATE mutation using a deterministic natural business key (e.g., SHA256(customer_id + transaction_timestamp) ). If a duplicate message is processed, Spanner overwrites the existing row with identical values without corrupting ledger totals.
Cloud Bigtable: Bigtable tables use lexicographically sorted row keys. By designing row keys that combine the business entity ID and event timestamp (e.g., device_id#2026-09-15T12:00:00Z ), duplicate message deliveries write to the exact same cell coordinate. In Bigtable, rewriting a cell with the same row key, column family, column qualifier, and timestamp simply updates the existing cell, achieving zero-overhead idempotency.
Pattern 3: Cloud Dataflow / Apache Beam Stateful Deduplication
In continuous streaming pipelines processing events before writing to downstream storage, Cloud Dataflow provides the Deduplicate.byKey() transform: PCollection<TransactionEvent> deduplicatedEvents = rawEvents
- .apply("KeyByTransactionId", WithKeys.of(TransactionEvent::getTransactionId))
- .apply("DeduplicateTransactions",
- Deduplicate.<String, TransactionEvent>byKey()
- .withDuration(Duration.standardMinutes(10)));
Stateful Processing Mechanics: Dataflow maintains a distributed state store (backed by persistent SSD shuffle disks) tracking transaction keys seen within a configurable time window (e.g., 10 minutes).
When an event arrives, Dataflow checks its state cache. If the key exists in state, the record is immediately dropped as a duplicate; if not, the key is recorded in state and the event passes downstream.
6. Realistic Exam Scenarios & Architecture Pitfalls
Scenario / Challenge Common Architecture Anti-Pattern Correct Google Cloud Architecture
High-Throughput Global Ordering ; A global logistics platform ingests 200,000 package tracking events/sec. Developers set a single static orderingKey = "GLOBAL" on all messages to ensure global chronological order. Channeling all topic messages through one static ordering key. Anti-Pattern: Assigning a single ordering key bottlenecks all 200,000 msg/sec to a single forwarder queue, causing catastrophic latency and queue overflow. Solution: Set orderingKey = package_id . Events for each individual package are delivered in strict FIFO order, while millions of different packages scale linearly across thousands of forwarders.
Poison-Pill Pipeline Freezes ; An ordered subscription processing financial transactions halts completely because a single message contains corrupted JSON, blocking all subsequent customer transactions. Manually purging the subscription or redeploying the consumer application. Configure a Dead-Letter Topic with maxDeliveryAttempts = 5 . After 5 failed attempts, Pub/Sub automatically diverts the malformed message to the DLQ and acknowledges it on the main subscription, immediately unblocking the Head-of-Line queue.
Eliminating Upstream Ingestion Duplicates ; An IoT gateway retries HTTP publish calls during poor cellular connectivity, resulting in duplicate sensor readings in BigQuery despite enabling Exactly-Once Delivery on the subscription. Assuming Pub/Sub Exactly-Once Delivery is broken and switching to a self-managed Kafka cluster. Understand that Pub/Sub EOD only prevents delivery duplicates from Pub/Sub to subscribers; it cannot detect publisher retries that produce new message_id s. Implement downstream idempotency in BigQuery using the Storage Write API deduplication or a deduplicating view using QUALIFY ROW_NUMBER() = 1 .