ExamFinch.com My Study

Study guide

4.2 Pub/Sub Advanced Patterns: Dead-Lettering, Ordering, and Schemas

Practice this exam

Google Cloud Professional Data Engineer Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

0.1 About the Google Cloud Professional Data Engineer Exam 0.2 Study Strategy & Cloud Data Engineering Methodology

2 Chapter 1: Designing for Security and Compliance

1.1 IAM Roles, Service Accounts, and Least Privilege for Data 1.2 Data Encryption, Key Management, and Privacy with Cloud DLP 1.3 Compliance, Data Sovereignty, and Geographic Placement 1.4 Multi-Environment Architecture: Development, Staging, and Production

3 Chapter 2: Designing for Reliability and Resilience

2.1 High Availability, Disaster Recovery, and Failover Patterns 2.2 Fault Tolerance and ACID Guarantees in Distributed Systems 2.3 Data Validation, Cleansing, and Quality Assurance with Dataform 2.4 AI-Assisted Data Preparation and LLM Query Generation

4 Chapter 3: Architecture Patterns and Flexibility

3.1 Modern Data Architectures: Data Lake, Warehouse, Lakehouse, and Mesh 3.2 Data Migration Strategies and Hybrid/Multi-Cloud Integration 3.3 Database Migration Service and Heterogeneous Database Migrations

5 Chapter 4: Ingestion Architectures: Pub/Sub and Streaming Sources

4.1 Cloud Pub/Sub Core Architecture and Ingestion 4.2 Pub/Sub Advanced Patterns: Dead-Lettering, Ordering, and Schemas 4.3 Change Data Capture (CDC) with Datastream and Batch Ingestion

6 Chapter 5: Stream and Batch Processing with Apache Beam and Cloud Dataflow

5.1 Apache Beam Core Concepts and Pipeline Design 5.2 Advanced Windowing, Watermarks, and Triggers 5.3 Dataflow Pipeline Optimization, Scaling, and Templates

7 Chapter 6: Data Transformation with Dataproc, Data Fusion, and Workflow Orchestration

6.1 Apache Spark and Hadoop with Cloud Dataproc 6.2 Visual Data Pipelines with Cloud Data Fusion 6.3 Workflow Orchestration with Cloud Composer 6.4 AI Data Enrichment inside Data Pipelines

8 Chapter 7: Storage Service Selection and Cloud Storage Architecture

7.1 Google Cloud Storage Service Selection Decision Matrix 7.2 Cloud Storage Architecture, Classes, and Lifecycle Management

9 Chapter 8: Deep Dive into Google BigQuery Storage

8.1 BigQuery Storage Architecture and Data Organization 8.2 Partitioning and Clustering for Performance and Cost 8.3 BigLake and Hybrid Querying across Clouds 8.4 Data Warehouse Modeling: Star Schemas, Denormalization, and Slowly Changing Dimensions

10 Chapter 9: Operational and NoSQL Databases: Bigtable and Cloud Spanner

9.1 Cloud Bigtable Architecture and Row Key Design 9.2 Cloud Bigtable Schema Management and Performance Tuning 9.3 Cloud Spanner: Globally Scalable Relational Database

11 Chapter 10: BigQuery Query Performance and Optimization

10.1 BigQuery SQL Tuning and Performance Best Practices 10.2 BigQuery BI Engine and Materialized Views 10.3 Data Sharing and Analytics Hub 10.4 Preparing Data for Visualization and BI Tool Connectivity

12 Chapter 11: Machine Learning Readiness and Data Governance

11.1 Machine Learning with BigQuery ML 11.2 Feature Engineering and Data Preparation for AI/ML 11.3 Data Governance, Cataloging, and Discovery with Dataplex 11.4 Unstructured Data, Embeddings, and Retrieval-Augmented Generation

13 Chapter 12: Site Reliability Engineering and Monitoring for Data Systems

12.1 SRE Principles for Data: SLIs, SLOs, and SLAs 12.2 Observability: Cloud Monitoring, Cloud Logging, and Alerting 12.3 Troubleshooting Data Pipelines and Workload Failures

14 Chapter 13: Workload Automation, CI/CD, and Cost Optimization

13.1 CI/CD and Version Control for Data Pipelines 13.2 BigQuery Cost Management and Capacity Planning 13.3 Holistic Cost Optimization across GCP Data Services 13.4 Query Workload Management: Interactive vs. Batch Jobs, Queues, and Quotas

4.2 Pub/Sub Advanced Patterns: Dead-Lettering, Ordering, and Schemas

Key Takeaways Explain This Topic AI

Dead-letter topics isolate poison-pill messages that exceed maxDeliveryAttempts, clearing head-of-line blocking and routing malformed events to auxiliary storage for debugging.

The Google-managed Cloud Pub/Sub service agent requires explicit roles/pubsub.publisher permissions on the dead-letter topic and roles/pubsub.subscriber on the source subscription for forwarding to succeed.

Exponential backoff retry policies dynamically increase redelivery intervals with randomized jitter, preventing the thundering herd problem against struggling downstream databases.

Subscriber FlowControlSettings (maxOutstandingMessages and maxOutstandingBytes) prevent worker heap exhaustion and container OOMKilled crashes during unexpected traffic surges.

Pub/Sub Lite was deprecated on June 17, 2024 and turned down on March 18, 2026; Kafka-compatible workloads now belong on Google Cloud Managed Service for Apache Kafka, and Pub/Sub charges $40 per TiB of throughput after the first 10 GiB free each month.

Last updated: September 2026

4.2 Pub/Sub Advanced Patterns: Dead-Lettering, Ordering, and Schemas

Important

High-scoring Professional Data Engineer exam candidates must master resilience engineering in streaming ingestion. This section focuses heavily on debugging stuck pipelines using Dead-Letter Topics (including critical service agent IAM permissions), tuning subscriber flow control to avoid Out-Of-Memory (OOM) failures, validating payloads via Avro/Protobuf schemas, and choosing correctly between Pub/Sub and Managed Service for Apache Kafka now that Pub/Sub Lite has been turned down.

Building resilient, production-ready ingestion pipelines requires architectures that anticipate failure. In real-world distributed systems, malformed payloads will arrive, downstream relational databases will experience transient connection pool saturation, and streaming workers will run out of memory if inundated with unthrottled message spikes. Google Cloud Pub/Sub incorporates sophisticated operational controls to ensure streaming platforms remain healthy, fault-tolerant, and cost-efficient under hostile operating conditions.

Poison Pills and Dead-Letter Topics (DLQs)

In streaming data processing, a poison pill is a message that cannot be processed successfully due to corrupted byte payloads, malformed JSON, missing mandatory schema fields, or unhandled runtime software exceptions in the subscriber application code.

When a subscriber worker encounters a poison pill, the processing thread fails and issues a negative acknowledgement ( NACK ) or allows the acknowledgement deadline to lapse. Under default at-least-once delivery, Pub/Sub redelivers the poison pill immediately. The worker crashes again, creating an infinite redelivery loop that consumes compute quota, inflates Cloud Logging costs, and introduces head-of-line blocking that starves valid messages waiting in the subscription backlog. +-----------------------------+

  • | Primary Pub/Sub Topic |
  • +-----------------------------+
  • |
  • v
  • +------------------------------------------------------------------------------------------------+
  • | Primary Subscription (maxDeliveryAttempts = 5) |
  • | |
  • | [Msg A (Valid)] ---> Delivered ---> Worker Processes Successfully ---> ACK |
  • | [Msg B (Poison)] ---> Delivered ---> Worker Crashes / NACKs (Attempt 1..5) |
  • | | |
  • | | (Attempts > 5) |
  • | v |
  • | Automatically ACKed on Source |
  • +------------------------------------------------------------------------------------------------+
  • |
  • v
  • +-----------------------------+
  • | Dead-Letter Topic (DLQ) |
  • +-----------------------------+
  • |
  • +---------------------------+---------------------------+
  • | |
  • v v
  • +-----------------------------------------+ +-----------------------------------------+
  • | Dead-Letter Subscription (Inspection) | | Cloud Storage Archival / Alerting |
  • | (Data Engineers Debug & Fix Schema) | | (Cloud Function Notifies On-Call SRE) |
  • +-----------------------------------------+ +-----------------------------------------+
  • Dead-Letter Topic Mechanics

To break infinite crash loops, data engineers configure a Dead-Letter Policy on the subscription:

deadLetterTopic : The fully qualified resource name of a dedicated Pub/Sub topic designated to receive failed messages ( projects/my-proj/topics/orders-dlq ).

maxDeliveryAttempts : An integer threshold between 5 and 100 (typically set to 5).

Pub/Sub tracks the delivery attempt count within the message metadata field deliveryAttempt . Each time a message is dispatched to a subscriber and subsequently NACK'd or timed out, deliveryAttempt increments. When the attempt count exceeds maxDeliveryAttempts :

Pub/Sub forwards the exact original message—preserving payload data, user attributes, and original publish timestamp—to the configured dead-letter topic.

Pub/Sub appends a custom system attribute ( CloudPubSubDeadLetterSourceDeliveryCount ) indicating the total delivery attempts made.

Pub/Sub automatically acknowledges the poison message on the original source subscription, clearing head-of-line blocking and restoring healthy pipeline throughput.

Mandatory IAM Permissions for Dead-Lettering (Critical Exam Trap)

A frequent source of pipeline misconfiguration on enterprise projects and exam questions relates to IAM permissions. The dead-letter forwarding mechanism is executed asynchronously by the Cloud Pub/Sub Google-managed Service Agent , not by the client application.

For dead-letter routing to operate, the Cloud Pub/Sub service agent ( service-<PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com ) must be granted two explicit IAM roles:

roles/pubsub.publisher on the Dead-Letter Topic (allowing the service agent to write failed messages).

roles/pubsub.subscriber on the Source Subscription (allowing the service agent to acknowledge and dequeue the forwarded messages).

If either permission is omitted, dead-letter routing fails silently. Pub/Sub cannot forward the message or clear it from the source subscription, and the poison pill continues redelivering in an infinite crash loop.

Dead-Letter Operational Workflows

Once messages arrive in the dead-letter topic, engineering teams establish triage workflows:

Inspection Subscription: Data engineers attach a pull subscription to inspect malformed payloads, identify upstream serialization bugs, and patch schema definitions.

Cloud Storage Archival: An automated export push subscription dumps failed records into a Cloud Storage bucket for historical auditing and compliance.

Automated Re-drive Tooling: After deploying a software patch to the consumer service, an automated utility reads records from the dead-letter topic and re-publishes them back to the primary topic for reprocessing.

Retry Policies and Exponential Backoff

When consumer services fail due to transient downstream outages—such as Cloud SQL reaching maximum database connection limits, BigQuery encountering API rate limits, or an external payment gateway returning HTTP 503—immediately redelivering failed messages exacerbates the failure (known as the thundering herd problem ).

Pub/Sub subscriptions support two distinct Retry Policies :

Immediate Retry (Default): Failed or NACK'd messages are redelivered as soon as possible. This is appropriate when failures are isolated to transient network packet drops, but hazardous during systemic database brownouts.

Exponential Backoff Retry: Pub/Sub applies a randomized truncated exponential backoff delay before re-dispatching failed messages. You configure two boundary parameters:

Minimum Backoff ( minimumBackoff ): The minimum delay before redelivery (e.g., 10 seconds).

Maximum Backoff ( maximumBackoff ): The maximum cap on redelivery delay (e.g., 600 seconds).

Between these limits, Pub/Sub progressively doubles the redelivery interval for each successive failure with randomized jitter: Delay = min ⁡ ( MaximumBackoff , MinimumBackoff × 2 attempt + jitter ) \text{Delay} = \min(\text{MaximumBackoff}, \text{MinimumBackoff} \times 2^{\text{attempt}} + \text{jitter}) Delay = min ( MaximumBackoff , MinimumBackoff × 2 attempt + jitter )

This delay smooths out incoming load spikes, giving struggling downstream relational databases or microservices time to clear connection pools and recover stability.

Subscriber Flow Control and Memory Management

In high-throughput pull architectures (such as Apache Beam / Dataflow pipelines, custom Java microservices, or Go workers), workers pull messages concurrently into memory. If an upstream burst publishes 500,000 large messages (e.g., 2 MB payloads each) in a few seconds, unconstrained workers pulling greedily will rapidly consume gigabytes of RAM, triggering fatal Java java.lang.OutOfMemoryError or Linux kernel Out-Of-Memory (OOM) killer terminations. +--------------------------------------------------------------------------------+

  • | Worker Node (e.g., 4 vCPU, 8 GB RAM) |
  • | |
  • | +------------------------------------------------------------------------+ |
  • | | Client Library FlowControlSettings | |
  • | | • maxOutstandingMessages = 1000 | |
  • | | • maxOutstandingBytes = 256 MB | |
  • | | • limitExceededBehavior = Block (Wait for in-flight tasks to ACK) | |
  • | +------------------------------------------------------------------------+ |
  • | | |
  • | [Throttled Ingestion Keeps Heap Healthy] |
  • | v |
  • | +------------------------------------------------------------------------+ |
  • | | Processing Worker Thread Pool (Active Tasks <= 1000, Memory <= 256 MB) | |
  • | +------------------------------------------------------------------------+ |
  • +--------------------------------------------------------------------------------+
  • FlowControlSettings Parameters

Google Cloud Pub/Sub client libraries provide client-side flow control via FlowControlSettings :

maxOutstandingMessages : Limits the maximum number of unacknowledged messages held concurrently in worker memory (e.g., capped at 1,000 messages). Once reached, the client pauses pulling new messages until active tasks are acknowledged.

maxOutstandingBytes : Limits the cumulative byte size of unacknowledged message payloads held in memory (e.g., capped at 256 MB). This is critical when message sizes fluctuate unpredictably.

LimitExceededBehavior : Dictates client behavior when thresholds are breached:

Block : The client suspends pulling from the gRPC stream until in-flight messages are acknowledged or timed out, preserving memory integrity.

Ignore : The client continues pulling, risking OOM crashes.

Sizing Formula for Worker Nodes

When provisioning streaming infrastructure, data engineers calculate memory allocations using the formula: Required Heap ≥ ( maxOutstandingBytes × 1.5 ) + Worker Runtime Overhead \text{Required Heap} \ge (\text{maxOutstandingBytes} \times 1.5) + \text{Worker Runtime Overhead} Required Heap ≥ ( maxOutstandingBytes × 1.5 ) + Worker Runtime Overhead

The 1.5 × 1.5\times 1.5 × multiplier accounts for Java object serialization overhead and garbage collection buffers. Setting maxOutstandingBytes appropriately ensures that even during massive upstream publishing surges, worker nodes operate safely within memory limits.

Pub/Sub Schemas: Data Contracts with Avro and Protobuf

In enterprise event-driven architectures, untyped raw JSON payloads introduce substantial operational risk: if an upstream producer changes a field name, omits a mandatory timestamp, or sends a string where an integer is expected, downstream BigQuery analytical tables or Dataflow transforms fail.

Pub/Sub Schemas enforce strict data contracts at the ingestion boundary. Schemas are defined as first-class Google Cloud resources associated directly with Pub/Sub topics.

Supported Schema Definitions

Apache Avro: A compact, binary format defined using JSON-based schema definitions ( .avsc ). Avro is widely used across Apache Beam, BigQuery, and Hadoop ecosystems.

Protocol Buffers (Protobuf): Google's high-performance, strongly typed serialization language ( .proto ). Protobuf generates compact binary encodings and strongly typed bindings across dozens of programming languages.

Encodings: Binary vs. JSON

When binding a schema to a topic, data engineers choose the payload encoding:

Binary Encoding: Maximum throughput, lowest wire size, and fastest serialization/deserialization. Payloads are stored as compact binary byte sequences.

JSON Encoding: The payload is formatted as plaintext JSON conforming strictly to the schema structure. Useful for human readability, browser clients, and REST debugging, though incurring larger payload sizes.

Publish-Time Schema Validation

When a topic is bound to a schema, the Pub/Sub service validates every published message synchronously before accepting it :

If the message payload conforms to the schema definition, Pub/Sub assigns a messageId and confirms the publish call with an OK status.

If the message violates the schema (missing mandatory fields, invalid data types, or unparseable bytes), Pub/Sub immediately rejects the message synchronously with an INVALID_ARGUMENT gRPC status error.

Preventing Data Lake Pollution: By rejecting corrupt records at the ingestion boundary, Pub/Sub guarantees that invalid data never enters subscriptions, BigQuery tables, or downstream stream pipelines.

Schema Revisions and Evolution

Pub/Sub supports Schema Revisions , enabling schema evolution over time while maintaining system compatibility:

BACKWARD Compatibility: Consumers running the new schema version can read messages written with the old schema version. This allows consumers to be upgraded before producers.

FORWARD Compatibility: Consumers running the old schema version can read messages written with the new schema version (e.g., new optional fields are ignored). This allows producers to be upgraded before consumers.

FULL Compatibility: Combines backward and forward compatibility, ensuring continuous bidirectional interoperability across rolling application deployments.

NONE : Disables compatibility checks between revisions (not recommended in production).

Pub/Sub vs. Managed Service for Apache Kafka (and the Retired Pub/Sub Lite)

Read this before you trust any older study material. For years the standard exam answer for "steady, ultra-high-volume, Kafka-compatible ingestion at lowest cost" was Pub/Sub Lite , a provisioned, partition-based sibling of Pub/Sub. Google deprecated Pub/Sub Lite on June 17, 2024 , closed it to new customers on September 24, 2024 , and turned it down on March 18, 2026 . The documented migration path is Google Cloud Managed Service for Apache Kafka or Pub/Sub . Any course, blog, or practice question that still recommends provisioning Lite partitions is describing a product you can no longer create.

The live decision is now between two products: +------------------------------------------+ +------------------------------------------+

  • | Pub/Sub (serverless & global) | | Managed Service for Apache Kafka |
  • | - Global topic; automatic sharding | | - Real Apache Kafka, KRaft mode |
  • | - No capacity to provision | | - Provision total vCPU + memory |
  • | - $40 per TiB throughput | | - Rack-aware 3-zone cluster |
  • | (first 10 GiB/month free) | | - Brokers auto-provisioned from vCPU |
  • | - DLQ, schemas, replay, exactly-once | | - Kafka API, ACLs, no code rewrite |
  • | - Push, pull, BigQuery & GCS exports | | - Private Service Connect access |
  • +------------------------------------------+ +------------------------------------------+
  • Pub/Sub: the Google-native default

Pub/Sub is a global, fully managed utility with no partitions or servers to manage. Ingress scales from zero to gigabytes per second, high availability is built in across zones, and dead-lettering, exactly-once delivery, schemas, replay via Seek, and direct BigQuery and Cloud Storage export subscriptions are all first-class.

Its pricing is consumption-based and worth memorizing precisely:

  • Throughput Type Price Free Allowance
  • Standard publish and subscribe (Message Delivery Basic) $40 per TiB First 10 GiB per month per billing account
  • BigQuery subscriptions $50 per TiB None
  • Cloud Storage subscriptions $50 per TiB None

Note the shape of that BigQuery subscription price: $50 per TiB covers both reading from the subscription and writing into BigQuery, with no additional BigQuery data ingestion charge . Comparing a direct BigQuery subscription against "Pub/Sub plus a Dataflow streaming job plus Storage Write API" on cost is a recurring exam framing, and the direct subscription usually wins when no transformation is required.

Managed Service for Apache Kafka: when the Kafka API is the requirement

This service runs the same open-source Apache Kafka you already operate, so existing producers and consumers migrate without application code changes — which is the single strongest signal for choosing it.

Architecturally:

You specify total vCPU count and memory , and the service provisions brokers from that vCPU count. Scaling means updating vCPU and memory, which can trigger automatic partition rebalancing.

Clusters are provisioned in a rack-aware three-zone configuration with replicas distributed across zones; topics default to at least three replicas with a minimum of two in-sync replicas. Single-zone and dual-zone clusters are not supported, and zone selection is automatic.

It runs Kafka in KRaft mode rather than ZooKeeper mode.

Clients reach the cluster through Private Service Connect , so a single cluster can be accessed securely from multiple VPCs, projects, and regions, with Cloud DNS supplying consistent bootstrap URLs.

Security layers TLS on every connection, SASL or mTLS authentication tied to IAM identities , authorization through both IAM role bindings and Kafka ACLs , and encryption at rest with Google-managed or customer-managed keys.

Google handles infrastructure and OS updates through rolling restarts, broker failure detection and replacement, and security patching. You remain responsible for topic retention policies (the main cost lever), cluster sizing, and ACL management.

The Decision Matrix

  • Signal in the Scenario Choose
  • Existing Kafka producers and consumers must migrate without code changes Managed Service for Apache Kafka
  • Existing Kafka ACL and partition-assignment semantics must be preserved Managed Service for Apache Kafka
  • Bursty or unpredictable volume; nobody wants to size a cluster Pub/Sub
  • Need dead-letter topics, schema enforcement, ordering keys, or Seek replay as managed features Pub/Sub
  • Need a direct, no-code path from the stream into BigQuery or Cloud Storage Pub/Sub export subscriptions
  • Global publishers writing to one logical topic Pub/Sub (global by design)
  • An older practice question says "use Pub/Sub Lite" Neither — Lite was turned down on March 18, 2026
  • Exam Traps and Antipatterns Summary

Antipattern / Trap Why It Fails Correct Exam Solution

Forgetting IAM permissions on Dead-Letter Topics Pub/Sub service agent cannot forward messages or acknowledge source subscription, causing infinite redelivery loops Grant roles/pubsub.publisher on DLQ topic and roles/pubsub.subscriber on source subscription to the Pub/Sub Service Agent

Setting maxDeliveryAttempts too low (e.g., 1 or 2) Transient network glitches or brief database locks cause valid messages to be prematurely routed to DLQ Set maxDeliveryAttempts between 5 and 10, combined with exponential backoff retry

Unbounded subscriber memory pulling in streaming clients Sudden burst of large messages triggers fatal Java heap OOM or Kubernetes OOMKilled terminations Configure client FlowControlSettings with maxOutstandingMessages , maxOutstandingBytes , and Block limit behavior

Answering "Pub/Sub Lite" for steady, ultra-high-volume Kafka-compatible ingestion Pub/Sub Lite was deprecated on June 17, 2024 and turned down on March 18, 2026; it can no longer be provisioned Managed Service for Apache Kafka for Kafka-API workloads, or Pub/Sub where the Kafka API is not required

Building a Dataflow job purely to move untransformed messages into BigQuery Adds a streaming pipeline, its workers, and its failure modes for a hop Pub/Sub can make natively Use a direct BigQuery subscription at $50 per TiB, which includes the BigQuery write with no separate ingestion charge

Relying on application-side JSON parsing for schema integrity Malformed payloads pass through Pub/Sub, crashing downstream Dataflow workers and polluting tables Bind Avro or Protobuf Pub/Sub Schemas directly to topics for synchronous publish-time rejection

Loading diagram... Pub/Sub Advanced Architecture: Publish-Time Schema Validation, Exponential Backoff, and Dead-Letter Routing