Study guide

9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers

Google Cloud Associate Data Practitioner Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

About the Associate Data Practitioner Exam

2 Chapter 1: Data Processing Paradigms & Ingestion Strategies

1.1 ETL, ELT, and ETLT Transformation Paradigms 1.2 Data Transfer and Migration Tools 1.3 Data Quality, Cleaning, and Validation

3 Chapter 2: Data Formats & Storage Solution Selection

2.1 File Formats & Serialization Characteristics 2.2 Cloud Storage Buckets, Location Types & Storage Classes 2.3 Selecting Google Cloud Databases and Storage Services

4 Chapter 3: Loading Data & Modern Ingestion Architectures

3.1 Batch Loading Data into BigQuery 3.2 Streaming Ingestion & Deduplication Patterns 3.3 BigQuery External Tables, BigLake & Federated Queries 3.4 Schema Definition & Evolution for Loaded Data

5 Chapter 4: BigQuery SQL Analytics & Query Foundations

4.1 BigQuery SQL Syntax & Analytical Functions 4.2 Complex Data Types: Arrays, Structs, and JSON 4.3 Answering Business Questions with BigQuery: Worked Analyses

6 Chapter 5: BigQuery Performance Optimization & Cost Governance

5.1 Table Partitioning Strategies 5.2 Table Clustering & Data Skipping 5.3 Query Optimization, Execution Plans, and Cost Controls

7 Chapter 6: Interactive Analysis, Notebooks & Visual Presentation

6.1 Exploratory Analysis with Jupyter & Colab Enterprise 6.2 Looker Studio (Data Studio) Reporting & Dashboard Design 6.3 Enterprise BI with Looker & LookML Foundations 6.4 Building, Modifying & Sharing Looker Dashboards

8 Chapter 7: Machine Learning with BigQuery ML & Vertex AI

7.1 Planning an ML Project: BigQuery ML, AutoML, or Pretrained Models? 7.2 Creating, Training & Evaluating Models with BigQuery ML 7.3 Running Inference and Organizing Models in Model Registry 7.4 Using Pretrained Gemini Models from BigQuery with Remote Connections

9 Chapter 8: Pipeline Engines & Transformation Services

8.1 Transformation Engine Selection Matrix 8.2 Apache Beam & Cloud Dataflow Core Concepts 8.3 Stream Processing: Windowing, Watermarks, and Triggers

10 Chapter 9: Workflow Orchestration, Automation & Monitoring

9.1 SQL Pipelines with Dataform & Scheduled Queries 9.2 Workflow Orchestration with Cloud Composer & Workflows 9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers 9.4 Monitoring Pipelines: Dataflow Job UI, Cloud Logging & Cloud Monitoring

11 Chapter 10: Access Control, Governance & Data Sharing

10.1 IAM Principles & Role Hierarchies for Data Services 10.2 Cloud Storage Security & Access Controls 10.3 Fine-Grained Security: Column-Level & Row-Level Controls 10.4 Secure Data Sharing with Analytics Hub (BigQuery Sharing)

12 Chapter 11: Lifecycle Management, Reliability & Compliance

11.1 Object Lifecycle Management & Storage Tiering 11.2 High Availability & Disaster Recovery Architectures 11.3 Data Retention, Time Travel, and Point-in-Time Recovery 11.4 Encryption Key Management & Privacy Compliance

9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers

Key Takeaways Explain This Topic AI

A BigQuery subscription writes Pub/Sub messages straight into a BigQuery table through the Storage Write API, with no Dataflow job or subscriber code.

BigQuery subscriptions deliver at least once and can map fields with the topic schema or the table schema and write message metadata such as message_id and publish_time.

Use a Dataflow streaming pipeline (for example the Pub/Sub to BigQuery template) when messages need parsing, validation, enrichment, or windowed aggregation before loading.

Eventarc routes CloudEvents, such as google.cloud.storage.object.v1.finalized, to Cloud Run, Cloud Run functions, Workflows, GKE, or internal HTTP endpoints.

Dead-letter topics (5 delivery attempts by default, configurable from 5 to 100) isolate poison messages so the rest of the stream keeps flowing.

Last updated: October 2026

9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers

Core Focus: The exam guide asks you to identify use cases for event-driven ingestion from Pub/Sub to BigQuery and to use Eventarc triggers in event-driven pipelines. Event-driven designs start work when data arrives instead of on a clock, which cuts latency and avoids paying for jobs that find nothing to do.

Traditional pipelines run on fixed schedules such as a nightly cron job. That adds latency, wastes runs when no new data exists, and breaks when upstream producers are late. Event-driven pipelines react to a message or a file the moment it arrives.

Event-Driven vs. Time-Based Scheduling Time-Based (Cron / Polling):

[ Clock: 02:00 UTC ] ---> (Triggers Pipeline) ---> [ Reads Cloud Storage Bucket ]

- Problem 1: If data lands at 02:05 UTC, it waits almost 24 hours.

- Problem 2: If data is late, the job processes empty or stale input.

Event-Driven (Pub/Sub / Eventarc):

[ File Lands in GCS ] ---> [ Eventarc: object finalized ] ---> (Triggers Pipeline)

- Processing starts seconds after the upload completes.

- Nothing runs (and nothing is billed) while waiting.

  • Dimension Time-Based Scheduling Event-Driven Architecture
  • Trigger Wall-clock time (cron) A state change: a message published, a file finalized
  • Freshness Bounded by the schedule interval Seconds after the event
  • Idle cost Jobs run even with no new data Little or none while idle
  • Late producers Need guessed buffer windows Work starts when data actually arrives

Typical tools Cloud Scheduler, BigQuery scheduled queries Pub/Sub, Eventarc, Cloud Run, Cloud Run functions, Workflows

Pub/Sub Basics

Pub/Sub is Google Cloud's managed, asynchronous messaging service. Publishers send messages to a topic ; each subscription on the topic receives its own copy of every message.

  • Subscription type How messages are delivered Typical consumer
  • Pull The consumer requests messages and acknowledges them Dataflow streaming jobs, custom workers
  • Push Pub/Sub sends each message as an HTTPS POST Cloud Run services, Cloud Run functions
  • BigQuery (export) Pub/Sub writes messages straight into a BigQuery table No consumer code at all

Cloud Storage (export) Pub/Sub writes batches of messages to files in a bucket Raw archives, data lake landing zones

A dead-letter topic catches messages that fail delivery more than a configured number of attempts (5 by default, configurable from 5 to 100), so one bad message does not block the rest.

  • Three Ways to Get Pub/Sub Data into BigQuery
  • 1. BigQuery Subscription (No Pipeline)

A BigQuery subscription writes each message to an existing BigQuery table as it arrives, using the Storage Write API behind the scenes. There is no Dataflow job or subscriber client to run.

Schema options: use the topic schema (Avro or Protocol Buffers) mapped to table columns; use the table schema to map JSON message fields to matching columns; or, with neither option, write the raw message to a data column.

Metadata: optionally write the subscription name, message ID, publish time, and attributes to extra columns for auditing.

Failures: messages that cannot be written are negatively acknowledged and retried, and move to the dead-letter topic if one is configured.

Delivery: at least once , so a downstream view or MERGE should deduplicate on message_id or a business key if duplicates matter.

Light changes: a Single Message Transform (a small JavaScript function) can adjust a message before it is written.

Permissions: the Pub/Sub service agent (or a service account you specify) needs permission to write to the table, such as BigQuery Data Editor on it. gcloud pubsub subscriptions create orders-to-bq \

  • -topic=orders \
  • -bigquery-table=my-project:sales.orders_raw \
  • -use-table-schema \
  • -write-metadata \
  • -dead-letter-topic=orders-dlq
  • 2. Dataflow Streaming Pipeline

Use Dataflow (for example the Google-provided Pub/Sub to BigQuery template, or your own Apache Beam pipeline) when messages need real transformation : parsing, validation, enrichment with lookups, windowed aggregation, or routing bad records to a dead-letter output. The Pub/Sub-to-BigQuery template enforces exactly-once processing by default; a BigQuery subscription is at-least-once.

3. Custom Subscriber

A Cloud Run service on a push subscription can call the Storage Write API itself. Choose this only when you need custom logic that neither option above provides.

  • Requirement Best choice
  • Land JSON events in a table with no transformation and no code BigQuery subscription
  • Parse, validate, enrich, or aggregate (windows) before loading Dataflow streaming pipeline
  • Keep a raw copy of every message as files Cloud Storage subscription
  • Load batches of files on a schedule, not events Load jobs or BigQuery Data Transfer Service (Section 3.1)
  • Eventarc: Routing Events to Pipelines

Eventarc routes events in the open CloudEvents format from Google Cloud services, Pub/Sub topics, and other sources to a destination.

Direct events: For example, google.cloud.storage.object.v1.finalized , emitted when an upload to a bucket completes. Filters narrow it to one bucket.

Cloud Audit Logs events: React to API calls recorded in audit logs (for example, a BigQuery job completing or a table being created) without changing the producer.

Pub/Sub messages: Any message published to a topic can trigger a destination.

Destinations: Cloud Run services, Cloud Run functions (formerly Cloud Functions), Workflows, GKE services, and internal HTTP endpoints in a VPC network.

Triggering Data Services from Eventarc

The exam guide mentions Eventarc with Dataform, Dataflow, Cloud Functions, Cloud Run, and Cloud Composer. Eventarc sends events to Cloud Run, Cloud Run functions, or Workflows, and that target then starts the data service through its API:

Goal Eventarc destination What the destination does

Run a Dataflow template for each uploaded file Cloud Run function or Workflows Calls the Dataflow API to launch a Flex Template with the file path

Refresh Dataform models when new data lands Workflows Compiles the repository and creates a Dataform workflow invocation

  • Start an Airflow DAG in Cloud Composer Cloud Run function Calls the Airflow REST API to trigger a DAG run
  • Validate and load a file into BigQuery Cloud Run service Checks the file, then starts a BigQuery load job

Loading diagram...

Failure Handling in Event-Driven Pipelines

Idempotency: Events can be delivered more than once. Use MERGE on a business key, deterministic job IDs, or deduplicating views so a retry does not double-count.

Dead-letter handling: Route messages that keep failing to a dead-letter topic or bucket instead of retrying forever, and alert on its size.

Backoff: Retry calls to rate-limited APIs (HTTP 429 or 503) with exponential backoff and jitter.

Common Exam Traps

Building a Dataflow job just to copy messages into BigQuery: If no transformation is needed, a BigQuery subscription removes the pipeline entirely.

Assuming exactly-once from a BigQuery subscription: It is at-least-once; deduplicate downstream when it matters.

Polling a bucket: A function that lists a bucket every minute adds latency and API cost; an Eventarc trigger on object finalization starts work immediately.

Silently dropping bad messages: Use a dead-letter topic so failures are kept, counted, and replayable.