Study guide

9.4 Monitoring Pipelines: Dataflow Job UI, Cloud Logging & Cloud Monitoring

Google Cloud Associate Data Practitioner Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

About the Associate Data Practitioner Exam

2 Chapter 1: Data Processing Paradigms & Ingestion Strategies

1.1 ETL, ELT, and ETLT Transformation Paradigms 1.2 Data Transfer and Migration Tools 1.3 Data Quality, Cleaning, and Validation

3 Chapter 2: Data Formats & Storage Solution Selection

2.1 File Formats & Serialization Characteristics 2.2 Cloud Storage Buckets, Location Types & Storage Classes 2.3 Selecting Google Cloud Databases and Storage Services

4 Chapter 3: Loading Data & Modern Ingestion Architectures

3.1 Batch Loading Data into BigQuery 3.2 Streaming Ingestion & Deduplication Patterns 3.3 BigQuery External Tables, BigLake & Federated Queries 3.4 Schema Definition & Evolution for Loaded Data

5 Chapter 4: BigQuery SQL Analytics & Query Foundations

4.1 BigQuery SQL Syntax & Analytical Functions 4.2 Complex Data Types: Arrays, Structs, and JSON 4.3 Answering Business Questions with BigQuery: Worked Analyses

6 Chapter 5: BigQuery Performance Optimization & Cost Governance

5.1 Table Partitioning Strategies 5.2 Table Clustering & Data Skipping 5.3 Query Optimization, Execution Plans, and Cost Controls

7 Chapter 6: Interactive Analysis, Notebooks & Visual Presentation

6.1 Exploratory Analysis with Jupyter & Colab Enterprise 6.2 Looker Studio (Data Studio) Reporting & Dashboard Design 6.3 Enterprise BI with Looker & LookML Foundations 6.4 Building, Modifying & Sharing Looker Dashboards

8 Chapter 7: Machine Learning with BigQuery ML & Vertex AI

7.1 Planning an ML Project: BigQuery ML, AutoML, or Pretrained Models? 7.2 Creating, Training & Evaluating Models with BigQuery ML 7.3 Running Inference and Organizing Models in Model Registry 7.4 Using Pretrained Gemini Models from BigQuery with Remote Connections

9 Chapter 8: Pipeline Engines & Transformation Services

8.1 Transformation Engine Selection Matrix 8.2 Apache Beam & Cloud Dataflow Core Concepts 8.3 Stream Processing: Windowing, Watermarks, and Triggers

10 Chapter 9: Workflow Orchestration, Automation & Monitoring

9.1 SQL Pipelines with Dataform & Scheduled Queries 9.2 Workflow Orchestration with Cloud Composer & Workflows 9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers 9.4 Monitoring Pipelines: Dataflow Job UI, Cloud Logging & Cloud Monitoring

11 Chapter 10: Access Control, Governance & Data Sharing

10.1 IAM Principles & Role Hierarchies for Data Services 10.2 Cloud Storage Security & Access Controls 10.3 Fine-Grained Security: Column-Level & Row-Level Controls 10.4 Secure Data Sharing with Analytics Hub (BigQuery Sharing)

12 Chapter 11: Lifecycle Management, Reliability & Compliance

11.1 Object Lifecycle Management & Storage Tiering 11.2 High Availability & Disaster Recovery Architectures 11.3 Data Retention, Time Travel, and Point-in-Time Recovery 11.4 Encryption Key Management & Privacy Compliance

9.4 Monitoring Pipelines: Dataflow Job UI, Cloud Logging & Cloud Monitoring

Key Takeaways Explain This Topic AI

The Dataflow job UI's job graph, execution details, job metrics, autoscaling charts, and pipeline logs show where a job is slow or failing.

Rising data freshness, system latency, and backlog together mean input is outpacing processing; check for a single bottleneck step before adding workers.

Cloud Logging supports log queries, log-based metrics, and sinks that route logs to BigQuery, Cloud Storage, or Pub/Sub.

INFORMATION_SCHEMA.JOBS answers BigQuery usage questions such as which queries consumed the most slot time or bytes billed.

Cloud Monitoring alerting policies pair a metric condition, such as oldest_unacked_message_age above 900 seconds for 5 minutes, with notification channels.

Last updated: October 2026

9.4 Monitoring Pipelines: Dataflow Job UI, Cloud Logging & Cloud Monitoring

Core Focus: The exam guide asks you to monitor Dataflow pipeline progress using the Dataflow job UI and to review and analyze logs in Cloud Logging and Cloud Monitoring . A pipeline that runs without observation fails silently; this section shows where to look and what each signal means.

The Dataflow Monitoring Interface

Open Dataflow > Jobs in the console to see running, completed, and failed jobs (with jobs from the last 30 days), then open a job:

View What it shows Use it to

Job graph The pipeline's steps and stages, with a job summary, job logs, and per-step details See which step is slow or failing and read its throughput

Execution details Stages over time, data freshness for streaming jobs, worker progress for batch jobs Find the stage that is holding back the pipeline

Job metrics Charts such as throughput, CPU utilization, data freshness, system latency, backlog, and I/O Watch trends and spot spikes

Autoscaling Current and target workers and the reason for scaling decisions Tell whether the job is short of workers or capped by max_num_workers or quota

  • Estimated cost Cost from resource usage so far Catch an expensive runaway job
  • Recommendations Suggestions for performance, cost, and errors Fix common misconfigurations
  • Pipeline logs Job logs from the service and worker logs from your code Read the stack trace behind a failure
  • Data sampling Sampled elements at each step Check what malformed input actually looks like
  • Reading Streaming Health

Data freshness is how far behind real time the pipeline's output is, based on the oldest data still being processed.

System latency is how long elements wait to be processed.

Backlog (for example, unacknowledged Pub/Sub messages) shows input piling up.

Rising freshness, latency, and backlog together mean input is arriving faster than the pipeline can process it. Before adding workers, check whether one step is the bottleneck: a slow external API or a sink at its quota creates backpressure on upstream steps, and more workers will not help. Dataflow also offers turnkey alerts for streaming jobs that monitor backlog and resource use.

Cloud Logging: Finding What Went Wrong

All the data services write logs to Cloud Logging : Dataflow job and worker logs, Composer (Airflow) task logs, Dataform invocation logs, and Cloud Audit Logs for BigQuery jobs. resource.type="dataflow_step"

  • resource.labels.job_id="2026-10-08_12_34_56-1234567890"
  • severity>=ERROR resource.type="bigquery_project"
  • protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"
  • severity=ERROR
  • Feature Purpose
  • Logs Explorer Search logs with the logging query language by resource, severity, job ID, or text
  • Log-based metrics Count matching log entries (for example, "dead-letter writes") to chart and alert on

Log sinks Route logs to BigQuery (analysis with SQL), Cloud Storage (cheap long-term retention), or Pub/Sub (other tools)

Log buckets and retention Keep logs longer than the default for audits

For BigQuery itself, the INFORMATION_SCHEMA.JOBS views are often faster than logs for usage questions: -- The 10 queries that consumed the most slot time in the last 7 days

  • SELECT user_email, job_id, total_slot_ms, total_bytes_billed, query
  • FROM `region-us`.INFORMATION_SCHEMA.JOBS
  • WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  • AND job_type = 'QUERY'
  • ORDER BY total_slot_ms DESC
  • LIMIT 10;
  • Cloud Monitoring: Dashboards and Alerts

Cloud Monitoring collects metrics from every service and turns them into dashboards and alerting policies.

  • Metric What it tells you
  • pubsub.googleapis.com/subscription/num_undelivered_messages How many messages are waiting in a subscription

pubsub.googleapis.com/subscription/oldest_unacked_message_age How old the oldest waiting message is, in seconds (a direct freshness SLA signal)

  • dataflow.googleapis.com/job/system_lag The longest time an item has waited to be processed in a streaming job
  • dataflow.googleapis.com/job/data_watermark_age How far the pipeline's data watermark is behind real time
  • bigquery.googleapis.com/slots/allocated Slots currently allocated to a project or reservation

An alerting policy combines a condition (for example, oldest_unacked_message_age above 900 seconds for 5 minutes) with notification channels such as email, SMS, Slack, PagerDuty, Pub/Sub, or webhooks. Conditions can be built in the console or written in PromQL. Uptime checks and log-based alerts cover services and log events that have no metric.

Troubleshooting Playbook

Symptom Where to look Likely fix

A batch Dataflow job failed Job graph, then worker logs for the failing step Fix the code or bad input; add a dead-letter output

Streaming freshness keeps growing Execution details and job metrics; check one step's throughput Remove the bottleneck (batch API calls, raise sink quota) or raise max_num_workers

Pub/Sub backlog age climbing Monitoring chart for oldest_unacked_message_age Scale or fix the subscriber; check for poison messages

A scheduled query stopped updating a table Scheduled queries run history; Cloud Logging for the transfer run Fix permissions (service account), quota, or SQL errors

BigQuery costs jumped INFORMATION_SCHEMA.JOBS by user and bytes billed Add partition filters, maximum_bytes_billed , or materialized views

Common Exam Traps

Scaling workers to fix backpressure: If one sink or API limits throughput, more workers only pile up more waiting work. Fix the bottleneck.

Confusing system lag with per-element processing time: Lag measures how long the oldest item has been waiting, not how long your code takes per element.

Keeping logs only in the default bucket: Default retention is short for Data Access logs (30 days); route logs to BigQuery or Cloud Storage with a sink when audits need longer.

Alerting on a single spike: Use a duration window (for example, 5 minutes) so alerts fire on sustained problems, not momentary blips.