Study guide
5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
Google Cloud Certified Professional Data Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Security, IAM, and Data Protection
1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems 1.2 Data Encryption, Key Management (CMEK), and Cloud KMS 1.3 Sensitive Data Protection: Cloud DLP, De-identification, and Masking
2 Chapter 2: Reliability and Compliance Architecture
2.1 High Availability, Regional vs Multi-Regional Deployments, and Disaster Recovery 2.2 Data Residency, Sovereignty, and Regulatory Compliance (HIPAA, GDPR) 2.3 Modern Data Architecture Patterns: Data Mesh, Lakehouse, and Modern Data Stack
3 Chapter 3: Cross-Cloud and Federated Governance
3.1 BigLake Architecture: Unified Multicloud Storage and Fine-Grained Access Control 3.2 BigQuery Omni: Multicloud Analytics Across AWS S3, Azure ADLS, and Google Cloud 3.3 Analytics Hub and Federated Computational Governance
4 Chapter 4: Data Ingestion and Pipeline Planning
4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect) 4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection 4.3 Change Data Capture (CDC) with Datastream for Continuous Database Ingestion
5 Chapter 5: Streaming Ingestion with Cloud Pub/Sub
5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle 5.2 Pub/Sub Message Ordering, Exactly-Once Delivery, and Deduplication 5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
6 Chapter 6: Pipeline Processing with Apache Beam and Cloud Dataflow
6.1 Apache Beam Programming Model: Pipelines, PCollections, and Core PTransforms 6.2 Cloud Dataflow Architecture: Autoscaling, Worker Allocation, and Dataflow Prime 6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows
7 Chapter 7: Advanced Dataflow: Robustness and Exactly-Once Semantics
7.1 Watermarks, Triggers, Accumulating vs Discarding Panes, and Late Data Handling 7.2 Side Inputs, Broadcast Joins, and Stateful Stream Processing 7.3 Pipeline Robustness: Dead-Letter Queues, Error Handling, and Pipeline Lifecycle Management
8 Chapter 8: Managed Data Processing: Cloud Dataproc and Cloud Data Fusion
8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning 8.2 Dataproc Serverless and Autoscaling Policies for Batch Processing 8.3 Cloud Data Fusion: Visual ETL/ELT Pipelines, CDAP Plugins, and Data Lineage
9 Chapter 9: Storage Selection and Cloud Storage Architecture
9.1 GCP Storage Selection Decision Matrix: Relational, NoSQL, Object, and Analytical Stores 9.2 Cloud Storage Architecture: Storage Classes, Lifecycle Rules, and Object Versioning 9.3 Cloud Storage Security: Bucket Lock (WORM), Access Control, and Data Migration Tools
10 Chapter 10: Relational Cloud Databases: Cloud Spanner and Cloud SQL
10.1 Cloud Spanner Architecture: TrueTime, External Consistency, and Global Scalability 10.2 Cloud Spanner Schema Design: Primary Keys, Interleaved Tables, and Query Tuning 10.3 Cloud SQL: High Availability, Read Replicas, Maintenance, and Point-in-Time Recovery
11 Chapter 11: Scalable NoSQL Databases: Cloud Bigtable and Firestore
11.1 Cloud Bigtable Architecture: Storage Model, Tablet Splitting, and Cluster Management 11.2 Cloud Bigtable Row Key Design: Lexicographical Ordering, Salt Keys, and Anti-Patterns 11.3 Cloud Firestore: Document Model, Real-Time Sync, and ACID Collections
12 Chapter 12: Enterprise Data Warehousing with BigQuery
12.1 BigQuery Architecture: Capacitor Columnar Format, Jupiter Network, and Borg Slots 12.2 BigQuery Partitioning: Ingestion-Time, Date/Timestamp, and Integer Range Strategies 12.3 BigQuery Clustering: Cardinality, Multi-Column Ordering, and Performance Optimization 12.4 Data Warehouse Modeling: Normalization, Star Schemas, and Slowly Changing Dimensions
13 Chapter 13: Advanced Analytical Queries and Transformations in BigQuery
13.1 Advanced SQL in BigQuery: Window Analytic Functions, ARRAY/STRUCT, and PIVOT/UNPIVOT 13.2 Modern SQL Orchestration with Dataform: DAG Modeling, Testing, and Documentation 13.3 BI Engine In-Memory Acceleration and Enterprise Looker/LookML Semantic Modeling
14 Chapter 14: Machine Learning and Feature Engineering with BigQuery ML
14.1 BigQuery ML Architecture: In-Database Training, Model Topologies, and SQL Analytics 14.2 Feature Engineering: ML.TRANSFORM, Preprocessing Pipelines, and Vertex AI Feature Store 14.3 MLOps, Model Governance, Continuous Monitoring, and Deployment Lifecycles
15 Chapter 15: Operations, Orchestration, and Data Governance
15.1 Workflow Orchestration with Cloud Composer: Managed Airflow, DAG Design, and Dependencies 15.2 Unified Data Governance with Dataplex: Lakes, Zones, Data Quality, and Lineage 15.3 Pipeline Observability, FinOps, and Cost Optimization Across GCP Data Services
5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
Key Takeaways Explain This Topic AI
BigQuery direct subscriptions write streaming messages directly from Pub/Sub into BigQuery tables without intermediate compute (Dataflow, Cloud Functions), drastically lowering architectural complexity, operational overhead, and latency.
BigQuery direct subscriptions require the service account service-{project-number}@gcp-sa-pubsub.iam.gserviceaccount.com to possess roles/bigquery.dataEditor and support use_topic_schema, write_metadata, and drop_unknown_fields.
Cloud Storage direct subscriptions stream batched messages directly into GCS buckets as raw text or Avro files, governed by batching flush triggers: max_duration (up to 10 minutes), max_bytes (up to 10 GiB), or max_messages.
Dead-Letter Topics (DLQ) isolate poison pills and malformed payloads by automatically forwarding messages that exceed maxDeliveryAttempts (5 to 100 attempts) to a secondary topic, preventing infinite redelivery loops.
When direct subscriptions encounter fatal schema incompatibilities or data type mismatches, messages are routed directly to the configured Dead-Letter Topic rather than silently dropped or permanently blocking ingestion.
Last updated: September 2026
5.3 Pub/Sub BigQuery/Cloud Storage Direct Subscriptions and Dead-Letter Topics
Exam Focus: The Google Cloud Professional Data Engineer exam expects you to recognize when to replace complex streaming ETL pipelines (Cloud Dataflow or Cloud Functions) with zero-compute Direct Subscriptions to BigQuery and Cloud Storage. You must master schema mapping flags ( use_topic_schema , write_metadata , drop_unknown_fields ), understand Cloud Storage batching triggers ( max_duration , max_bytes ), configure Dead-Letter Topics (DLQ) , and assign the exact IAM permissions required by the Pub/Sub service agent.
Historically, landing real-time streaming data from Cloud Pub/Sub into analytical data stores required deploying intermediate compute infrastructure. Engineers had to provision, monitor, and scale Apache Beam pipelines on Cloud Dataflow or maintain dozens of event-driven Cloud Functions simply to deserialize JSON payloads and stream them into BigQuery or write them as files into Cloud Storage. With the introduction of Pub/Sub Direct Subscriptions , Google Cloud eliminated intermediate compute for straightforward ingestion, providing serverless, zero-maintenance streaming pipelines at wire speed.
1. BigQuery Direct Subscriptions: Serverless, Zero-Pipeline Ingestion
A BigQuery Direct Subscription writes messages directly from a Cloud Pub/Sub topic into an existing BigQuery table without requiring a Dataflow worker pool or serverless functions. [ Publishers ] ──> [ Pub/Sub Topic ] ──(Direct Subscription)──> [ BigQuery Table ]
- │
- (Under the Hood)
- ▼
- [ BigQuery Storage Write API ]
- (Zero VM Compute, Sub-Second Latency)
- How BigQuery Direct Subscriptions Operate
Under the hood, Cloud Pub/Sub integrates directly with the BigQuery Storage Write API using committed streams. Pub/Sub handles all connection lifecycle management, batching, auto-scaling, and backoff retries internally:
Cost Efficiency: Customers pay only for standard Pub/Sub delivery fees and BigQuery Storage Write API volume ingestion charges ($0.025 per GB, with the first 2 TB per month free). There are zero virtual machine, memory, or worker compute costs.
Latency: Direct subscriptions achieve sub-second delivery latency from message publication to table queryability.
Key Configuration Parameters
When provisioning a BigQuery direct subscription, data engineers configure three critical behaviors: gcloud pubsub subscriptions create bq-direct-sub \
- -topic=telemetry-topic \
- -bigquery-table=my-project:analytics_lake.sensor_telemetry \
- -use-topic-schema \
- -write-metadata \
- -drop-unknown-fields
use_topic_schema :
When enabled, Pub/Sub parses message payloads using the schema registered on the topic (Avro or Protocol Buffers) and maps fields directly to corresponding BigQuery columns with matching names and compatible data types.
If disabled (or if the topic has no schema), Pub/Sub writes the raw, unparsed message payload into a designated string or JSON column named data in the destination table.
write_metadata :
Automatically injects system metadata columns into the destination BigQuery table alongside message fields. The BigQuery table schema must define these exact column names:
subscription_name ( STRING ): Fully qualified resource name of the subscription.
message_id ( STRING ): Unique identifier assigned by Pub/Sub.
publish_time ( TIMESTAMP ): Time the message was committed to Pub/Sub storage.
attributes ( JSON or STRING ): Key-value dictionary of user-defined message attributes.
ordering_key ( STRING ): Present if message ordering was used.
Architecture Value: Essential for data lineage, auditing, partition clustering, and downstream deduplication queries.
drop_unknown_fields :
Dictates how Pub/Sub handles schema evolution when upstream producers send records containing fields not present in the BigQuery table schema.
If true : Pub/Sub automatically ignores and drops any unrecognized fields, successfully writing all matching fields into BigQuery.
If false : Pub/Sub rejects the message write operation because the record does not conform to the BigQuery table schema. The message remains unacknowledged and will either trigger delivery retries or be routed to a Dead-Letter Topic.
Mandatory IAM Permissions for BigQuery Direct Ingestion
Direct subscriptions do not run under user credentials; they execute under the Google-managed Cloud Pub/Sub Service Agent : service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com
To write data into BigQuery, the service agent must be granted the following IAM roles:
roles/bigquery.dataEditor on the destination BigQuery table or dataset (grants data plane write permissions via Storage Write API).
roles/bigquery.metadataViewer on the dataset or project (allows Pub/Sub to inspect table schema and verify column mapping).
2. Cloud Storage Direct Subscriptions: Direct Batch File Ingestion
A Cloud Storage Direct Subscription automatically streams and batches messages directly into Google Cloud Storage buckets without spinning up compute workers. This pattern is ideal for landing raw data lake records, creating cold compliance archives, and feeding batch analytical engines like Dataproc or BigLake.
Output Formats
Cloud Storage direct subscriptions support two file formats:
Text (Plaintext / Newline-Delimited JSON): Each message payload is written on a new line. Messages can be stored with or without trailing newlines.
Avro (Binary Container Format): Messages are packaged into structured Avro files containing schema definitions and metadata. Avro files support block-level compression (such as Snappy or Deflate) and are splittable across distributed queries in BigLake, Apache Spark, and Presto/Trino.
Batching Flush Triggers
Writing individual 1 KB files to Cloud Storage for every incoming message would cause extreme file fragmentation, overwhelming Cloud Storage metadata operations and creating the dreaded "small file problem" for downstream analytical engines. To prevent this, Cloud Storage direct subscriptions buffer incoming messages in memory and flush them to a new file whenever any one of three configurable triggers is satisfied: +----------------------------------------------------------------------------------+
- | CLOUD STORAGE DIRECT SUBSCRIPTION FLUSH TRIGGERS |
- +----------------------------------------------------------------------------------+
- | |
- | Incoming Streaming Messages ──> [ In-Memory Buffer ] |
- | │ |
- | ┌────────────────────────────────┼────────────────────────────────┐ |
- | ▼ ▼ ▼ |
- | Condition 1: Condition 2: Condition 3: |
- | max_duration reached max_bytes reached max_messages |
- | (e.g., 5 minutes) (e.g., 1 GiB) reached |
- | │ │ │ |
- | └────────────────────────────────┼────────────────────────────────┘ |
- | ▼ |
- | Flushes Buffer -> Commits New File to GCS |
- | gs://data-lake-raw/events/2026/09/15/file_uuid.avro |
- +----------------------------------------------------------------------------------+
max_duration : The maximum amount of time Pub/Sub buffers data before writing a file (configurable from 1 minute to 10 minutes , default: 5 minutes).
max_bytes : The maximum accumulated size of buffered messages before flushing (configurable from 1 MiB to 10 GiB , default: 1 GiB).
max_messages : The maximum message count allowed in a single file (configurable up to 100,000+ messages).
Customizing Object Naming and Partitioning
Direct subscriptions allow configuring dynamic directory structures and file extensions:
filename_prefix : Static directory prefix (e.g., telemetry/raw/ ).
filename_suffix : File extension (e.g., .avro , .json ).
filename_datetime_format : Formats directory hierarchies based on current UTC time (e.g., YYYY/MM/DD/hh/mm/ ). This enables automatic Hive-compatible directory partitioning ( gs://bucket/telemetry/raw/2026/09/15/12/events_abc.avro ) that external query engines can prune efficiently.
IAM Requirements for Cloud Storage Direct Subscriptions
The Pub/Sub Service Agent ( service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com ) must possess:
roles/storage.objectCreator (or roles/storage.admin ) on the destination bucket.
3. Architectural Decision Matrix: Direct Subscriptions vs. Dataflow vs. Cloud Functions
Architectural Attribute Pub/Sub Direct Subscriptions Cloud Dataflow (Apache Beam) Cloud Functions / Cloud Run
Compute Infrastructure Zero compute; fully managed by Google Managed VM worker pools ( n2-standard-4 ) Serverless container/event instances
Transformation Capabilities None; raw 1-to-1 payload mapping or simple schema binding Complex; windowing, joins, aggregations, data enrichment, masking Lightweight; stateless parsing, filtering, schema validation
Ingestion Latency Sub-second (BigQuery) / Minutes (GCS batch) Sub-second (Streaming Engine) Milliseconds to seconds
Cost Profile Lowest; standard Pub/Sub fees + storage API ingestion Moderate to high; hourly VM, memory, and Streaming Engine charges Low to moderate; per-invocation and vCPU-second billing
Multi-Sink Fan-Out Single destination per subscription Writes concurrently to BigQuery, Spanner, Bigtable, GCS in one DAG Writes to any downstream destination via client SDKs
Operational Overhead Zero; no worker scaling or pipeline monitoring Moderate; pipeline updates, draining, checkpoint tuning Low; cold-starts, concurrency limits, timeout tuning
Best Exam Fit Pure raw ingestion to BigQuery or GCS with zero transformations Real-time analytics, session windowing, multi-source joins, PII masking Simple webhook delivery, event-driven alerts, lightweight routing
4. Dead-Letter Topics (DLQ) Mechanics and Architecture
A Dead-Letter Topic (DLQ) is a secondary Pub/Sub topic designated to capture undeliverable or repeatedly failing messages (often called poison pills ).
Why Dead-Letter Topics Are Essential
In streaming pipelines, unparseable messages can cause catastrophic failures:
In Push Subscriptions , a malformed message causes the webhook endpoint to return HTTP 500, triggering infinite retries that consume CPU and swamp the service.
In Ordered Subscriptions , a failed message causes Head-of-Line blocking , freezing processing for that ordering key.
In BigQuery Direct Subscriptions , a message that violates data types (e.g., passing a string into an integer column) will fail write operations indefinitely. ┌──────────────────────────────────────── Primary Subscription ────────────────────────────────────────┐
- │ │
- │ Incoming Messages ──> [ Attempt 1: Failed ] ──> [ Attempt 2: Failed ] ... ──> [ Attempt 5: Failed ] │
- └──────────────────────────────────────────────────────────────────────────────────┬──────────────────┘
- │
- (Forward to DLQ & ACK Primary)
- ▼
- ┌─────────────────────────────────────── Dead-Letter Topic ───────────────────────────────────────────┐
- │ │
- │ [ Captured Poison-Pill Message ] ──> Preserves Payload + CloudPubSubDeadLetterSourceDeliveryAttempt │
- │ │ │
- │ ┌────────┴────────┐ │
- │ ▼ ▼ │
- │ [ Cloud Logging Alert ] [ Quarantine BQ Table ] │
- └─────────────────────────────────────────────────────────────────────────────────────────────────────┘
- How Dead-Letter Queuing Works
Threshold Configuration ( maxDeliveryAttempts ): The data engineer configures a Dead-Letter Topic and sets maxDeliveryAttempts (an integer between 5 and 100 , default: 5): gcloud pubsub subscriptions create main-orders-sub \
- -topic=orders-topic \
- -dead-letter-topic=orders-dlq-topic \
- -max-delivery-attempts=5
Delivery Attempt Counter: Pub/Sub increments an internal counter ( delivery_attempt ) every time it attempts delivery to a subscriber and the delivery fails (due to ack deadline expiration, explicit nack , or push endpoint error).
Forwarding to DLQ: When the delivery attempt count reaches maxDeliveryAttempts , Pub/Sub automatically:
Publishes the failed message to the configured Dead-Letter Topic .
Injects a system attribute: CloudPubSubDeadLetterSourceDeliveryAttempt indicating total failed delivery attempts.
Acknowledges the message on the primary subscription , removing it from the active delivery queue and immediately resolving any Head-of-Line block.
Mandatory IAM Permissions for Dead-Letter Topics
For Pub/Sub to divert failing messages, the Pub/Sub Service Agent ( service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com ) must possess two specific IAM permissions:
roles/pubsub.publisher on the Dead-Letter Topic (to publish the evicted message).
roles/pubsub.subscriber on the Primary Subscription (to acknowledge and evict the failing message).
Exam Trap — Silent DLQ Failure: If you configure a Dead-Letter Topic but fail to grant roles/pubsub.publisher and roles/pubsub.subscriber to the Pub/Sub Service Agent, Pub/Sub cannot forward the message . Instead, the message will continue to be redelivered to consumers indefinitely, exceeding maxDeliveryAttempts without ever routing to the DLQ.
5. DLQ Operations, Cloud Monitoring, and Poison-Pill Remediation
Isolating poison-pill messages in a Dead-Letter Topic is only the first step. Production architectures must monitor, alert on, and remediate failures:
Cloud Monitoring Alerting Policies:
Monitor the metric pubsub.googleapis.com/subscription/dead_letter_message_count .
Configure an alerting policy in Cloud Monitoring that triggers an urgent PagerDuty or Slack notification whenever the publish rate to the Dead-Letter Topic is greater than zero.
Monitor pubsub.googleapis.com/subscription/oldest_unacked_message_age to detect consumer lag across both primary and DLQ subscriptions.
Triage and Quarantine Sink:
Attach a BigQuery direct subscription to the Dead-Letter Topic, writing failed records into a quarantine_errors table.
Engineers inspect the quarantine_errors table to analyze payload discrepancies, schema incompatibilities, or frontend client bugs.
Re-Ingestion Workflow:
Once the application bug or schema definition is fixed, a recovery script or Dataflow job reads messages from the quarantine table or DLQ subscription, applies necessary transformations, and republishes the sanitized records back to the primary topic for reprocessing.
6. Realistic Exam Scenarios & Architecture Pitfalls
Scenario / Challenge Common Architecture Anti-Pattern Correct Google Cloud Architecture
Serverless Clickstream Ingestion ; A company needs to ingest 100,000 JSON clickstream events/sec into BigQuery for real-time dashboards. No complex transformations are required. Provisioning a 20-node Cloud Dataflow streaming cluster running an Apache Beam pipeline. Create a BigQuery Direct Subscription on the Pub/Sub topic targeting the BigQuery table. Enable write_metadata = true and grant roles/bigquery.dataEditor to the Pub/Sub service agent. Achieves zero-compute serverless ingestion at fractional cost.
Preventing File Fragmentation in Data Lakes ; A Cloud Storage direct subscription writes raw IoT records to a bucket. Analysts report that Spark queries take hours because millions of 5 KB files exist in the bucket. Writing a Cloud Function triggered on object creation to merge files together after upload. Adjust the Cloud Storage direct subscription batching settings: configure max_duration = 10 minutes and max_bytes = 1 GiB using the Avro format. Pub/Sub buffers records in memory and writes large, query-optimized 1 GiB Avro files, permanently resolving the small file problem.
Infinite Redelivery Loops on Poison Pills ; A subscription configured with a Dead-Letter Topic continues redelivering corrupted messages over 100 times without routing them to the DLQ. Increasing maxDeliveryAttempts to 100 and redeploying consumer worker pods. Verify IAM permissions. The Pub/Sub Service Agent lacks roles/pubsub.publisher on the Dead-Letter Topic or lacks roles/pubsub.subscriber on the primary subscription. Granting these roles enables Pub/Sub to forward the failed message and acknowledge it on the primary subscription.
Upstream Schema Evolution Breaking Ingestion ; A mobile app releases a new update adding an app_version string to its JSON payload. The BigQuery direct subscription immediately begins failing to write records because app_version is missing from the BigQuery table. Rewriting the ingestion pipeline in Dataflow with custom JSON parsing logic. Update the BigQuery direct subscription with --drop-unknown-fields . Pub/Sub drops the unexpected app_version field and writes all recognized fields cleanly into BigQuery, maintaining pipeline uptime while the data warehouse team adds the column.