Study guide
9.2 Workflow Orchestration with Cloud Composer & Workflows
Google Cloud Associate Data Practitioner Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
About the Associate Data Practitioner Exam
2 Chapter 1: Data Processing Paradigms & Ingestion Strategies
1.1 ETL, ELT, and ETLT Transformation Paradigms 1.2 Data Transfer and Migration Tools 1.3 Data Quality, Cleaning, and Validation
3 Chapter 2: Data Formats & Storage Solution Selection
2.1 File Formats & Serialization Characteristics 2.2 Cloud Storage Buckets, Location Types & Storage Classes 2.3 Selecting Google Cloud Databases and Storage Services
4 Chapter 3: Loading Data & Modern Ingestion Architectures
3.1 Batch Loading Data into BigQuery 3.2 Streaming Ingestion & Deduplication Patterns 3.3 BigQuery External Tables, BigLake & Federated Queries 3.4 Schema Definition & Evolution for Loaded Data
5 Chapter 4: BigQuery SQL Analytics & Query Foundations
4.1 BigQuery SQL Syntax & Analytical Functions 4.2 Complex Data Types: Arrays, Structs, and JSON 4.3 Answering Business Questions with BigQuery: Worked Analyses
6 Chapter 5: BigQuery Performance Optimization & Cost Governance
5.1 Table Partitioning Strategies 5.2 Table Clustering & Data Skipping 5.3 Query Optimization, Execution Plans, and Cost Controls
7 Chapter 6: Interactive Analysis, Notebooks & Visual Presentation
6.1 Exploratory Analysis with Jupyter & Colab Enterprise 6.2 Looker Studio (Data Studio) Reporting & Dashboard Design 6.3 Enterprise BI with Looker & LookML Foundations 6.4 Building, Modifying & Sharing Looker Dashboards
8 Chapter 7: Machine Learning with BigQuery ML & Vertex AI
7.1 Planning an ML Project: BigQuery ML, AutoML, or Pretrained Models? 7.2 Creating, Training & Evaluating Models with BigQuery ML 7.3 Running Inference and Organizing Models in Model Registry 7.4 Using Pretrained Gemini Models from BigQuery with Remote Connections
9 Chapter 8: Pipeline Engines & Transformation Services
8.1 Transformation Engine Selection Matrix 8.2 Apache Beam & Cloud Dataflow Core Concepts 8.3 Stream Processing: Windowing, Watermarks, and Triggers
10 Chapter 9: Workflow Orchestration, Automation & Monitoring
9.1 SQL Pipelines with Dataform & Scheduled Queries 9.2 Workflow Orchestration with Cloud Composer & Workflows 9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers 9.4 Monitoring Pipelines: Dataflow Job UI, Cloud Logging & Cloud Monitoring
11 Chapter 10: Access Control, Governance & Data Sharing
10.1 IAM Principles & Role Hierarchies for Data Services 10.2 Cloud Storage Security & Access Controls 10.3 Fine-Grained Security: Column-Level & Row-Level Controls 10.4 Secure Data Sharing with Analytics Hub (BigQuery Sharing)
12 Chapter 11: Lifecycle Management, Reliability & Compliance
11.1 Object Lifecycle Management & Storage Tiering 11.2 High Availability & Disaster Recovery Architectures 11.3 Data Retention, Time Travel, and Point-in-Time Recovery 11.4 Encryption Key Management & Privacy Compliance
9.2 Workflow Orchestration with Cloud Composer & Workflows
Key Takeaways Explain This Topic AI
Cloud Composer is a fully managed Apache Airflow service hosted on Google Kubernetes Engine (GKE), providing rich programmatic DAG authoring in Python for cross-system data orchestration.
Airflow core components include Directed Acyclic Graphs (DAGs), Operators (BigQueryInsertJobOperator, DataflowCreateJavaJobOperator), Tasks, Sensors (GCSObjectExistenceSensor), and connection Hooks.
Task dependencies in Airflow are established programmatically using bitshift operators (upstream >> downstream), supporting complex multi-stage fan-out and fan-in workflows.
Cloud Workflows provides lightweight, fully serverless orchestration defined in YAML/JSON, offering sub-second startup latency and pay-per-step billing ideal for event-driven microservices and REST API coordination.
Last updated: October 2026
Workflow Orchestration with Cloud Composer & Workflows
Core Focus: Enterprise data pipelines rarely operate within a single tool. Ingesting, processing, modeling, and publishing analytical data requires coordinating multi-system workflows spanning transactional databases, object storage, streaming compute, data warehouses, and visualization platforms. On Google Cloud, Cloud Composer (managed Apache Airflow) and Cloud Workflows serve as the two primary orchestrators. Understanding their architectural differences, operational costs, and implementation patterns is critical for data practitioners.
While tools like Dataform excel at managing transformations inside BigQuery, enterprise data engineering demands end-to-end orchestration across disparate platforms. A representative enterprise pipeline might extract records from Cloud SQL, stage files in Cloud Storage, launch a Cloud Dataflow batch validation job, run BigQuery transformations, and finally trigger a Looker dashboard cache refresh or a Vertex AI model evaluation. Orchestrating these heterogeneous steps requires dedicated workflow engines that manage state, dependencies, retries, and logging.
The Role of Cross-Service Orchestration
Data pipelines that span multiple services require an external coordinator to enforce execution boundaries. In a multi-stage architecture: [ Cloud SQL / Third-Party APIs ]
- |
- v (Extract & Stage)
- [ Cloud Storage (Landing Bucket) ]
- |
- v (Batch Validation & Tokenization)
- [ Cloud Dataflow Job ]
- |
- v (Load)
- [ BigQuery Raw Dataset ]
- |
- v (Dataform / SQL Transformation)
- [ BigQuery Curated Marts ]
- |
- +---> [ Looker Dashboard Refresh ]
- +---> [ Vertex AI Pipeline Ingestion ]
Without a centralized orchestrator, teams rely on fragmented cron jobs or ad-hoc scripts, introducing single points of failure, lack of unified logging, and severe difficulties in diagnosing pipeline disruptions. Cross-service orchestrators bridge this gap by treating each distinct Google Cloud service as an individual task within a unified, monitored workflow.
Cloud Composer: Managed Apache Airflow
Cloud Composer is a fully managed workflow orchestration service built on the open-source Apache Airflow framework. Cloud Composer automates the provisioning, configuration, scaling, and maintenance of complete Airflow environments. +-----------------------------------------------------------------------------------------+
- | Cloud Composer Architecture |
- | |
- | +-----------------------------------------------------------------------------------+ |
- | | Google Kubernetes Engine (GKE) Cluster | |
- | | | |
- | | [ Airflow Webserver ] [ Airflow Schedulers ] [ Airflow Triggerers ] | |
- | | | | |
- | | v | |
- | | [ Autoscaling Airflow Workers ] | |
- | | - Worker 1, Worker 2, Worker N | |
- | +-----------------------------------------------------------------------------------+ |
- | | |
- | +----------------------------------+----------------------------------+ |
- | | | | |
- | v v v |
- | [ Cloud SQL ] [ Cloud Storage ] [ Cloud Logging ] |
- | - Metadata DB - DAG Bucket (`/dags`) - Centralized Log |
- | - Task instance state - Plugins & requirements Aggregation |
- | - Variables & connections - Cloud Storage FUSE mount |
- +-----------------------------------------------------------------------------------------+
- Architectural Components
Google renamed Cloud Composer Managed Service for Apache Airflow in April 2026 (the exam guide still says Cloud Composer). Composer 3 ("Managed Airflow Gen 3") is the current generation, and many Composer 2 environments are still running. An environment has four core pillars:
Google Kubernetes Engine (GKE): Runs Airflow (in Composer 3 this cluster sits in a Google-managed tenant project, so you no longer see or manage it), including Schedulers (which parse DAGs and schedule tasks), Triggerers (which handle asynchronous event-based operators), the Airflow Webserver UI, and dynamic Airflow Workers.
Cloud SQL Instance: Serves as the central Airflow metadata database, recording the execution state of DAG runs, task instances, variables, and connection credentials.
Cloud Storage DAG Bucket: Every Composer environment is paired with a dedicated Cloud Storage bucket. Uploading Python DAG files to the gs://[environment-bucket]/dags directory triggers automatic synchronization to all GKE worker and scheduler nodes via Cloud Storage FUSE.
Cloud Logging & Monitoring: All Airflow worker stdout/stderr logs, scheduler logs, and task outputs stream automatically into Cloud Logging, accessible directly from the Airflow UI or Google Cloud console.
Core Airflow Concepts
DAG (Directed Acyclic Graph): A collection of all tasks organized with directional dependencies, authored entirely in standard Python code. DAGs must be strictly acyclic (no infinite loops or circular dependencies).
Operators: Define the template for a single unit of work. Common Google Cloud operators include:
BigQueryInsertJobOperator : Submits BigQuery SQL queries, scripts, or export jobs asynchronously.
CloudDataTransferServiceCreateJobOperator : Initiates automated transfers between external storage or AWS S3 and Cloud Storage.
DataflowStartFlexTemplateOperator / DataflowTemplatedJobStartOperator : Launch Dataflow jobs from Flex or classic templates. (The older DataflowCreatePythonJobOperator and DataflowCreateJavaJobOperator were removed from the Google provider; BeamRunPythonPipelineOperator runs Beam code directly.)
DataprocSubmitJobOperator : Submits Spark, PySpark, or Hive jobs to a Dataproc cluster.
BashOperator and PythonOperator : Execute local bash commands or Python callables directly within the Airflow worker container.
Tasks: Concrete instances of operators bound to a specific DAG run.
Sensors: Specialized operators that halt downstream execution until a specific external condition is met. For example, GCSObjectExistenceSensor continually polls a Cloud Storage bucket until a specific file arrives, while ExternalTaskSensor waits for a task in a separate DAG to complete.
Hooks: Low-level interfaces that abstract authentication and API calls to external platforms (such as Google Cloud APIs, AWS, Snowflake, or Salesforce).
Defining Task Dependencies and Authoring DAGs
Dependencies in Apache Airflow are declared cleanly using Python bitshift operators ( >> for upstream-to-downstream, << for downstream-to-upstream): from datetime import datetime, timedelta
- from airflow import DAG
- from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
- from airflow.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensor
- from airflow.providers.google.cloud.operators.dataflow import DataflowStartFlexTemplateOperator
- default_args = {
- 'owner': 'data-engineering',
- 'depends_on_past': False,
- 'retries': 3,
- 'retry_delay': timedelta(minutes=5),
- 'retry_exponential_backoff': True,
- }
- with DAG(
- dag_id='customer_analytics_pipeline',
- default_args=default_args,
- description='End-to-end customer batch ETL pipeline',
- schedule='0 3 * * *', # Daily at 03:00 UTC
- start_date=datetime(2026, 1, 1),
- catchup=False,
- tags=['retail', 'daily'],
) as dag:
- # Step 1: Wait for raw partner data landing in Cloud Storage
- wait_for_landing_file = GCSObjectExistenceSensor(
- task_id='wait_for_landing_file',
- bucket='enterprise-raw-landing',
- object='partner_data/{{ ds }}/transactions.csv',
- timeout=3600,
- poke_interval=60,
- mode='reschedule',
- )
- # Step 2: Trigger Dataflow job to validate and sanitize raw records
- run_dataflow_validation = DataflowStartFlexTemplateOperator(
- task_id='run_dataflow_validation',
- project_id='analytics-prod',
- location='us-central1',
- body={
- 'launchParameter': {
- 'jobName': 'sanitize-tx-{{ ds_nodash }}',
- 'containerSpecGcsPath': 'gs://enterprise-code-repo/templates/sanitize_transactions.json',
- 'parameters': {'input': 'gs://enterprise-raw-landing/partner_data/{{ ds }}/*.csv'},
- }
- },
- )
- # Step 3: Execute BigQuery transformation to aggregate daily analytics
- run_bigquery_marts = BigQueryInsertJobOperator(
- task_id='run_bigquery_marts',
- configuration={
- 'query': {
- 'query': """
- MERGE INTO `analytics.daily_sales` T
- USING `staging.sanitized_tx` S
- ON T.transaction_id = S.transaction_id
- WHEN MATCHED THEN UPDATE SET amount = S.amount
- WHEN NOT MATCHED THEN INSERT ROW;
- """,
- 'useLegacySql': False,
- }
- },
- )
- # Defining DAG Dependency Graph using bitshift operators
- wait_for_landing_file >> run_dataflow_validation >> run_bigquery_marts
- Airflow Scheduling, Backfill, and Failure Recovery
catchup and backfill : By default, Airflow attempts to execute all un-run historical intervals between start_date and the current date ( catchup=True ). In data pipelines where reprocessing history would trigger redundant or costly computations, setting catchup=False ensures only the most recent scheduled interval executes upon deployment.
Retries and Exponential Backoff: Network blips, API rate limits, or transient downstream database locks can cause temporary task failures. Setting retries=3 alongside retry_exponential_backoff=True prevents overwhelming failing systems while ensuring automatic resilience.
Airflow Sensors ( poke vs reschedule ): In poke mode, a sensor blocks the Airflow worker slot continuously while waiting, consuming memory and compute resources. In reschedule mode, the sensor yields its worker slot back to the cluster between evaluation checks, preventing worker starvation across the environment.
Worker Autoscaling (Composer 2 and 3): Cloud Composer automatically scales Airflow workers up and down based on task queue depth and resource demand. Organizations define minimum and maximum worker limits, allowing environments to absorb batch surges and scale down during off-peak hours.
Cloud Workflows: Serverless API and Microservice Orchestration
While Cloud Composer is ideal for complex, stateful data pipelines with extensive third-party library requirements, Cloud Workflows provides a lightweight, fully serverless alternative designed for orchestrating Google Cloud APIs, Cloud Run services, and Cloud Functions. +-------------------------------------------------------------------------+
- | Cloud Workflows |
- | |
- | [ Eventarc / HTTP Request ] ---> [ Workflow Engine (YAML / JSON) ] |
- | | |
- | v |
- | +---------------------------------+ |
- | | |
- | v v |
- | [ Call Cloud Run ] [ BigQuery REST API ] |
- | | | |
- | +----------------+----------------+ |
- | | |
- | v |
- | [ Conditional Branching ] |
- | - switch / condition |
- | | |
- | v |
- | [ Publish to Cloud Pub/Sub ] |
- +-------------------------------------------------------------------------+
- Architecture and Characteristics
Declarative YAML/JSON Definitions: Workflows are written declaratively, outlining sequential steps, conditional branching ( switch ), parallel branches ( parallel ), and error catching ( try/retry/except ).
No Continuous Running Cost: Cloud Workflows is completely serverless and scales to zero. Pricing is strictly pay-per-step (with a generous free tier), eliminating the fixed cluster infrastructure costs associated with Cloud Composer.
Sub-Second Execution Latency: Workflows starts executing in milliseconds, making it ideal for event-driven, low-latency microservice pipelines (such as processing an e-commerce order or reacting to an uploaded document).
Built-in Connectors: Google Cloud provides native connectors for BigQuery, Cloud Storage, Pub/Sub, Cloud Tasks, and Firestore, simplifying authenticated API invocations without custom authentication boilerplate. # Sample Cloud Workflows definition coordinating an automated BigQuery export
main:
params: [args]
steps:
- init:
assign:
- projectId: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
- dataset: "curated_mart"
- runBigQueryJob:
call: googleapis.bigquery.v2.jobs.insert
args:
projectId: ${projectId}
body:
configuration:
query:
query: "SELECT * FROM `curated_mart.daily_kpis` WHERE date = CURRENT_DATE()"
destinationTable:
- projectId: ${projectId}
- datasetId: ${dataset}
- tableId: "export_cache"
- writeDisposition: "WRITE_TRUNCATE"
- useLegacySql: false
- result: jobResponse
- notifySuccess:
call: googleapis.pubsub.v1.projects.topics.publish
args:
topic: ${"projects/" + projectId + "/topics/pipeline-notifications"}
body:
messages:
- data: ${base64.encode(json.encode({"status": "SUCCESS", "jobId": jobResponse.id}))}
- done:
- return: ${jobResponse.id}
- Comparison: Cloud Composer vs. Cloud Workflows
The following matrix outlines the technical and operational differences between Cloud Composer and Cloud Workflows:
Architectural Attribute Cloud Composer (Managed Airflow) Cloud Workflows
Underlying Engine Apache Airflow hosted on Google Kubernetes Engine (GKE). Google-proprietary, serverless state execution engine.
Authoring Paradigm Programmatic Python code defining DAGs and operators. Declarative YAML or JSON defining steps and transitions.
Startup Latency Seconds to minutes (DAG scheduling cycles). Milliseconds (instantaneous invocation).
Pricing & Cost Model Always-on environment fee (the environment runs even when no DAG is executing) plus scaling workers. Pay-per-step executed; scales to zero with zero idle cost.
Ecosystem & Connectors Vast open-source library (Airflow providers for GCP, AWS, Azure, dbt, Snowflake, Spark). Native Google Cloud connectors and generic HTTP/REST endpoints.
Execution Context Executes custom Python/Bash code inside worker containers. Orchestrates external APIs; cannot run arbitrary internal code.
Primary Use Cases Heavyweight, complex, batch data pipelines spanning multi-cloud platforms. Event-driven microservices, API chaining, and serverless ingestion workflows.
Dataproc Workflow Templates
A Dataproc workflow template is a reusable definition of a DAG of Spark, PySpark, Hive, or Hadoop jobs plus where to run them. Creating the template runs nothing; instantiating it starts a workflow:
Managed cluster: The workflow creates an ephemeral cluster, runs the jobs in dependency order, and deletes the cluster when they finish.
Cluster selector: The workflow runs on an existing cluster chosen by labels and leaves that cluster running.
Parameterized: The template declares parameters (such as an input path) so each run can pass new values.
Inline: gcloud can instantiate a YAML definition directly without saving a template. gcloud dataproc workflow-templates instantiate nightly-spark-etl \
- -region=us-central1 \
- -parameters=INPUT_PATH=gs://landing/2026-10-08/
Workflow templates orchestrate only Dataproc jobs and have no built-in clock: trigger them from Cloud Scheduler, Workflows, or a Composer task.
Selecting an Orchestration Solution
Option Scope Authoring Cost model Exam cue
BigQuery scheduled queries One SQL statement or script SQL in the console or bq Only the query cost "Refresh this summary table every night"
Dataform workflows Dependent SQL models in BigQuery SQLX in Git Only the query cost "Dozens of dependent tables with assertions and version control"
Dataproc workflow templates A DAG of Spark/Hadoop jobs on one cluster YAML or API Cluster time only while it runs "Run three dependent Spark jobs on a cluster that exists only for the run"
Workflows Calls to Google Cloud APIs and HTTP services YAML/JSON steps Per step executed; nothing when idle "Serverless, event-driven chain of API calls"
Cloud Composer Complex DAGs across many systems Python (Airflow) Always-on environment "Cross-system pipeline with sensors, backfills, and many operators"
Common Exam Traps & Real-World Scenarios
Exam Tip: Look for keywords in question stems. If the prompt describes a serverless , low-cost , event-driven , or HTTP-centric workflow that executes infrequently, choose Cloud Workflows . If the prompt requires an open-source , Python-authored , complex cross-cloud pipeline with sophisticated sensors and backfill capabilities , choose Cloud Composer .
Trap 1: Processing Heavy Data Inside Airflow Workers
The Trap: Writing Python code inside an Airflow PythonOperator to download a 50 GB CSV from Cloud Storage, load it into a pandas DataFrame, transform the records in memory, and upload it to BigQuery.
The Reality: Airflow is an orchestrator, not a distributed compute engine. Airflow worker pods have limited memory and CPU. Performing data transformations inside workers causes Out-Of-Memory (OOM) crashes and worker node eviction. Airflow tasks should merely dispatch work to scalable engines (e.g., launching a Dataflow job or triggering a BigQuery SQL job).
Trap 2: Choosing Composer for Low-Latency Event Processing
The Trap: Using Cloud Composer to trigger a sub-second response whenever a single user submits an online registration form.
The Reality: Composer schedulers operate on polling intervals and are ill-suited for real-time, low-latency microservices. Cloud Workflows, triggered via Eventarc or Cloud Functions, provides instant sub-second execution with zero idle compute cost.
Trap 3: Neglecting Sensor Reschedule Mode
The Trap: Leaving Airflow sensors configured with default mode='poke' when waiting for external files that might take hours to arrive.
The Reality: In poke mode, the sensor continuously occupies an Airflow worker execution slot for the entire duration, blocking other tasks and driving up autoscaling costs. Using mode='reschedule' frees up the worker slot between checks.