9.2 Deploying Pub/Sub, Dataflow & Loading Data into Google Cloud

Google Cloud Associate Cloud Engineer ACE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Introduction & Exam Overview

1.1 About the GCP Associate Cloud Engineer Exam 1.2 How to Use This Study Guide & Build a Study Plan

2 Chapter 2: Resource Hierarchy, IAM Basics & Cloud Identity

2.1 The GCP Resource Hierarchy: Organizations, Folders & Projects 2.2 Organization Policies & Granting IAM Roles 2.3 Cloud Identity: Managing Users and Groups

3 Chapter 3: APIs, Quotas & Billing Configuration

3.1 Enabling APIs and Provisioning Cloud Operations Suite 3.2 Assessing and Requesting Quota Increases 3.3 Billing Accounts, Project Linking & Billing Exports 3.4 Billing Budgets and Alerts

4 Chapter 4: Planning Compute & Storage Resources

4.1 Choosing the Right Compute Option: GCE vs. GKE vs. Cloud Run vs. Cloud Functions 4.2 Spot VMs, Custom Machine Types & Cost-Optimized Compute Planning 4.3 Choosing a Data Storage Product: Cloud SQL, BigQuery, Firestore, Spanner & Bigtable 4.4 Cloud Storage Classes & Persistent Disk Types

5 Chapter 5: Planning Network Resources

5.1 Load Balancing Options and Use Cases 5.2 Regional Resource Placement & High Availability Planning 5.3 Network Service Tiers: Premium vs. Standard

6 Chapter 6: Deploying Compute Engine Workloads

6.1 Launching Compute Engine Instances: Disks, SSH Keys & Availability Policies 6.2 Managed Instance Groups, Instance Templates & Autoscaling 6.3 OS Login and VM Manager for Fleet Configuration

7 Chapter 7: Deploying Google Kubernetes Engine Workloads

7.1 Installing and Configuring kubectl 7.2 Deploying GKE Clusters: Autopilot, Regional, Private & GKE Enterprise 7.3 Deploying Containerized Applications to GKE

8 Chapter 8: Deploying Serverless & Event-Driven Applications

8.1 Deploying Applications to Cloud Run and Cloud Functions 8.2 Event-Driven Architectures: Pub/Sub, Cloud Storage Triggers & Eventarc

9 Chapter 9: Deploying Data Solutions

9.1 Deploying Data Products: Cloud SQL, Firestore, Spanner, BigQuery & AlloyDB 9.2 Deploying Pub/Sub, Dataflow & Loading Data into Google Cloud

10 Chapter 10: Deploying Networking Resources & Infrastructure as Code

10.1 Creating VPCs, Subnets & Shared VPC 10.2 Firewall Rules, Cloud VPN & VPC Network Peering 10.3 Infrastructure as Code: Terraform, Config Connector, Cloud Foundation Toolkit & Helm

11 Chapter 11: Managing Compute Engine & GKE in Production

11.1 Connecting to and Managing Running Compute Engine Instances 11.2 Working with Snapshots and Custom Images 11.3 Managing GKE Clusters: Node Pools, Workloads & Artifact Registry Access 11.4 Horizontal and Vertical Pod Autoscaling in GKE

12 Chapter 12: Managing Cloud Run, Storage & Database Operations

12.1 Managing Cloud Run Revisions, Traffic Splitting & Scaling 12.2 Managing and Securing Cloud Storage Objects & Lifecycle Policies 12.3 Querying, Estimating Costs, Backup & Job Monitoring for Databases (Cloud SQL, BigQuery, Spanner, Dataflow)

13 Chapter 13: Networking Operations, Monitoring & Logging

13.1 Networking Operations: Subnets, Static IPs, Cloud DNS & Cloud NAT 13.2 Cloud Monitoring: Alerts, Custom Metrics & Dashboards 13.3 Cloud Logging: Buckets, Log Analytics, Routers & Filtering 13.4 Diagnostics, Ops Agent, Managed Service for Prometheus & Audit Logs

14 Chapter 14: IAM Policies & Role Types

14.1 IAM Policies: Structure, Bindings & Policy Hierarchy 14.2 IAM Role Types: Basic, Predefined & Custom Roles

15 Chapter 15: Service Accounts & Security Configuration

15.1 Creating Service Accounts & Assigning Least-Privilege Permissions 15.2 Assigning Service Accounts to Resources & Managing SA IAM Policies 15.3 Service Account Impersonation & Short-Lived Credentials

16 Chapter 16: Exam Readiness — Scenarios & Test-Day Strategy

16.1 Common ACE Exam Scenarios & Decision Frameworks 16.2 Test-Day Logistics, Time Management & Final Review Checklist

9.2 Deploying Pub/Sub, Dataflow & Loading Data into Google Cloud

Key Takeaways Explain This Topic AI

Pub/Sub topics are append-only message stores; subscriptions (pull or push) deliver to subscribers, with a default 7-day (up to 31-day) retention window and dead-letter topics for repeated delivery failures.

Dataflow runs Apache Beam pipelines in batch or streaming mode; Flex templates (Docker image plus a Cloud Storage spec file) are the current recommended packaging over classic templates.

BigQuery load jobs cap out at 15 TB and 10,000 files per job across CSV/JSON/Avro/Parquet/ORC, with gzip as the only supported CSV/JSON compression.

Storage Transfer Service is the managed, code-free choice for large or recurring transfers from S3, Azure Blob Storage, HDFS, or on-prem POSIX sources -- reach for Dataflow only when the data needs transformation in flight.

Match the loading method to volume and frequency: command-line tools for ad hoc small files, BigQuery load jobs for staged batch files, Storage Transfer Service for ongoing bulk migration.

Last updated: July 2026

Why This Section Matters

The exam guide's data-products bullet also names Pub/Sub and Dataflow among the things you must know how to deploy, and a separate bullet -- "Loading data (e.g., command line upload, load data from Cloud Storage, Storage Transfer Service)" -- tests how you move data into Google Cloud in the first place. Together these form the "plumbing" half of Domain 3's data content: messaging between services, transforming data in flight, and bulk-loading it from wherever it currently lives. Expect scenario questions that describe a data volume, source system, and freshness requirement, then ask you to pick the right ingestion tool -- confusing Storage Transfer Service (a managed copy job) with Dataflow (a transformation pipeline) is one of the most common ACE traps.

Pub/Sub -- Asynchronous Messaging

Pub/Sub decouples publishers from subscribers using two objects:

A topic -- an append-only named resource that publishers send messages to.

A subscription -- attached to a topic, delivers each message to its subscriber (a topic can have many subscriptions, giving fan-out to multiple independent consumers). gcloud pubsub topics create order-events

gcloud pubsub subscriptions create order-events-sub \
-topic=order-events --ack-deadline=30 \
-message-retention-duration=7d

Delivery works in two modes:

Mode How it works Best for

Pull Subscriber (an app or a Dataflow job) actively requests messages and acknowledges them Batch-style or high-throughput consumers that control their own pace

Push Pub/Sub sends an HTTP POST to your endpoint; a 2xx response acknowledges the message Lightweight, webhook-style consumers such as Cloud Run or Cloud Functions

Three deployment details the exam likes to test:

Message retention : the default is 7 days after publish, and messages (acknowledged or not, depending on configuration) can be retained for up to 31 days , which lets a newly created subscription "replay" recent history instead of only seeing messages published after it existed.

Ordering keys : attaching an ordering key to messages guarantees per-key delivery order, but push subscriptions allow only one outstanding message per ordering key at a time , which can add latency -- pull is usually preferred for high-volume ordered streams.

Dead-letter topics : after a configurable maximum delivery attempts count, Pub/Sub forwards a message it cannot get acknowledged to a separate dead-letter topic for offline investigation, instead of retrying it forever.

Dataflow -- Managed Apache Beam Pipelines

Dataflow runs data-processing pipelines written against the open source Apache Beam model, in either batch mode (bounded data, such as a nightly file drop) or streaming mode (unbounded data, such as a live Pub/Sub topic) using the same programming model for both.

You rarely write a pipeline from scratch for common extract-transform-load (ETL) work: Google publishes ready-to-run Dataflow templates , and you launch one instead of authoring custom code: gcloud dataflow jobs run pubsub-to-bq \

-gcs-location=gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery \
-region=us-central1 \

--parameters inputSubscription=projects/my-proj/subscriptions/order-events-sub,outputTableSpec=my-proj:sales_ds.orders

There are two template families:

Classic templates -- the original format; parameters are effectively fixed once the template is staged.

Flex templates -- Google's current recommendation for any new template. The pipeline is packaged as a Docker image in Artifact Registry , referenced by a small specification file in Cloud Storage; this allows more flexible runtime parameters and support for arbitrary custom sources and sinks.

A common ACE-style scenario reads: "events land continuously in a Pub/Sub topic and must be transformed and written into BigQuery in near real time." The answer is a Dataflow streaming job -- ideally launched from a Google-provided template such as Pub/Sub Subscription to BigQuery -- not a manual bq load , which is a batch-only path that cannot consume a live topic. Note also that Google-managed template jobs now run on Dataflow Prime (an autotuning execution mode) by default; if a scenario calls for the older, standard Dataflow execution behavior, you disable Prime with the enable_prime=false service option.

Loading Data Into Google Cloud

The exam guide names three loading paths, each suited to a different volume and frequency profile:

Method Typical use Notes

Command-line upload Small-to-moderate one-off files, ad hoc work gcloud storage cp (the current CLI, replacing gsutil cp ) for Cloud Storage objects; bq load to load directly into a BigQuery table from a local file or a gs:// path

Load data from Cloud Storage Recurring batch loads of files already staged in a bucket BigQuery load jobs read CSV, newline-delimited JSON, Avro, Parquet, or ORC ; a single load job can process up to 15 TB across up to 10,000 files ; gzip is the only supported compression for CSV/JSON

Storage Transfer Service Large, recurring, or scheduled transfers from another cloud or on-prem system Managed, code-free transfers from Amazon S3 , Azure Blob Storage , HDFS, and on-prem POSIX filesystems into Cloud Storage; optimized for transfers over 1 TiB ; supports both one-time and repeating scheduled jobs

A frequent exam trap is reaching for a custom Dataflow pipeline to "just copy files" from Amazon S3 into Cloud Storage -- that is exactly what Storage Transfer Service already does, managed and without custom code. Save Dataflow for when the data genuinely needs transformation in flight, not just relocation. For extremely large, one-time, offline migrations (measured in hundreds of terabytes or petabytes) where shipping data over the network is impractical, Google also offers the Transfer Appliance , a physical device you load on-site and ship back to Google -- rarely the right answer on the ACE exam, but useful to recognize as an outlier compared to the three primary loading paths above.