Study guide

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow

Google Cloud Professional Machine Learning Engineer PMLE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & the Agent Platform Map

1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map

2 Chapter 2: Low-Code ML with BigQuery ML

2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery

3 Chapter 3: AutoML & Google Cloud AI APIs

3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs

4 Chapter 4: Foundation Models in Model Garden

4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability

5 Chapter 5: Exploring, Preprocessing & Protecting Data

5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows

6 Chapter 6: Notebooks, Prototyping & Experiment Tracking

6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata

7 Chapter 7: Evaluating Predictive & Generative AI Solutions

7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge

8 Chapter 8: Choosing the Modeling Approach

8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements

9 Chapter 9: Training Data & Training SDKs

9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning

10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune

11 Chapter 11: Training Hardware & Distributed Training

11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism

12 Chapter 12: Deploying Models for Inference

12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers

13 Chapter 13: Model Versions, Rollouts & Inference Logic

13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing

14 Chapter 14: Scaling Online Serving

14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving

15 Chapter 15: End-to-End ML Pipelines

15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving

16 Chapter 16: Orchestration with Managed Airflow & Ray

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads

17 Chapter 17: Automating Retraining & CI/CD/CT

17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build

18 Chapter 18: Securing AI Systems, Responsible AI & Explainability

18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform

19 Chapter 19: Monitoring AI Solutions

19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production

16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow

Key Takeaways Explain This Topic AI

Managed Service for Apache Airflow, formerly Cloud Composer, is a fully managed Apache Airflow service for authoring, scheduling, and monitoring workflows as Python DAGs.

A Managed Airflow environment runs Airflow schedulers, triggerers, and workers on a GKE cluster and stores DAGs, logs, and plugins in an environment Cloud Storage bucket.

Airflow 3 is generally available in Managed Airflow (Gen 3) environments, with Airflow 2 builds still offered.

The Apache Airflow Google provider includes operators such as RunPipelineJobOperator, CreateCustomContainerTrainingJobOperator, CreateBatchPredictionJobOperator, and DeployModelOperator.

Choose Managed Airflow when ML steps are part of a broader cross-system data workflow, and Agent Platform Pipelines when the workflow is ML-centric and needs artifact lineage.

Last updated: September 2026

The exam guide names Managed Service for Apache Airflow as a pipeline orchestration option. On April 15, 2026 , Google announced that Cloud Composer is evolving to become Managed Service for Apache Airflow (Managed Airflow) to reflect its open-source foundation. Older materials and code still say "Composer," and the Airflow Google provider keeps compatibility names alongside new ManagedAirflow... operator aliases.

What Managed Airflow Provides

Apache Airflow defines workflows as DAGs (directed acyclic graphs) of tasks in Python files. Managed Airflow runs Airflow for you:

Component Role

GKE cluster (managed by the service) Runs Airflow schedulers , triggerers , and workers , plus monitoring agents

  • Airflow web server The Airflow UI
  • Airflow database Airflow metadata (DAG runs, task states)
  • Environment bucket (Cloud Storage) DAGs , logs, custom plugins, and data

Environment generations are named Managed Airflow (Gen 3) , (Gen 2) , and (Legacy Gen 1) . Airflow 3 is generally available in Gen 3 environments, alongside Airflow 2 builds. You manage environments through the console, gcloud, the API (still called the Cloud Composer API), or Terraform. Logs go to Cloud Logging and metrics to Cloud Monitoring .

DAGs for ML Workflows

The Apache Airflow Google provider includes operators for Agent Platform, BigQuery, Dataflow, Managed Spark, and more. Agent Platform-related operators include:

  • Need Operators
  • Run an Agent Platform Pipeline RunPipelineJobOperator

Custom training CreateCustomTrainingJobOperator , CreateCustomContainerTrainingJobOperator , CreateCustomPythonPackageTrainingJobOperator

AutoML training CreateAutoMLTabularTrainingJobOperator , CreateAutoMLImageTrainingJobOperator , CreateAutoMLForecastingTrainingJobOperator

  • Tuning CreateHyperparameterTuningJobOperator
  • Batch scoring CreateBatchPredictionJobOperator
  • Serving CreateEndpointOperator , DeployModelOperator
  • Model versions AddVersionAliasesOnModelOperator , ListModelVersionsOperator
  • Features CreateFeatureOnlineStoreOperator , CreateFeatureViewOperator , FeatureViewSyncSensor

Distributed compute CreateRayClusterOperator , DeleteRayClusterOperator with DAG("nightly_churn", schedule="0 2 * * *", start_date=datetime(2026, 9, 1), catchup=False):

  • load = BigQueryInsertJobOperator(task_id="build_features", configuration={...})
  • wait = GCSObjectExistenceSensor(task_id="wait_for_crm_export", bucket="crm", object="{{ ds }}/export.csv")
  • train = RunPipelineJobOperator(task_id="train_pipeline", display_name="churn-{{ ds }}",
  • template_path="gs://ml/pipelines/churn.yaml",
  • parameter_values={"run_date": "{{ ds }}"}, region="us-central1", project_id="p")
  • score = CreateBatchPredictionJobOperator(task_id="score", ...)
  • publish = BigQueryInsertJobOperator(task_id="publish_scores", configuration={...})
  • [load, wait] >> train >> score >> publish

Airflow features that matter for ML operations:

Schedules (cron or presets) with catchup and backfill for past periods.

Sensors that wait for files, tables, or external jobs.

Retries , SLAs , and alerting callbacks.

Templating (for example, {{ ds }} ) to pass the logical run date into queries and pipeline parameters.

Event-based triggering , such as DAGs triggered by Cloud Storage changes.

Custom plugins and Python dependencies installed in the environment.

Managed Airflow vs. Agent Platform Pipelines

  • Dimension Managed Airflow Agent Platform Pipelines
  • Primary purpose General workflow orchestration across many systems ML workflows

Infrastructure A persistent environment (GKE-based) that you size and pay for continuously Serverless . Pay per run plus task resources

ML artifacts and lineage Data lineage can integrate with Knowledge Catalog, but models and ML artifacts aren't tracked in ML Metadata ML Metadata lineage , artifact passing, and experiment integration built in

Caching of ML steps Task-level logic you implement Execution caching by step interface

Ecosystem Hundreds of operators for databases, SaaS, and other clouds Google Cloud Pipeline Components, KFP, TFX

Scheduling Rich cron, sensors, backfills, cross-DAG dependencies Scheduler API, Pub/Sub and Eventarc triggers

Best fit ML steps inside a larger data platform schedule End-to-end ML training, evaluation, and deployment workflows

The common hybrid pattern

Let Managed Airflow orchestrate the enterprise data workflow (ingest from SaaS, wait for upstream systems, build warehouse tables), then trigger an Agent Platform Pipeline for the ML portion with RunPipelineJobOperator . The ML pipeline keeps lineage, caching, and evaluation gates, and Airflow keeps cross-system dependencies and backfills.

Operational Considerations

Environment sizing: scale workers and schedulers to DAG volume. Heavy compute belongs in BigQuery, Dataflow, Managed Spark, or Agent Platform jobs, not on Airflow workers .

Identity: give the environment's service account least-privilege roles for the services its DAGs call.

Networking: use private IP environments and VPC Service Controls where required.

Version upgrades: plan Airflow 2 to Airflow 3 migrations, because DAG APIs changed.

Cost: the environment runs continuously, which suits organizations already running many DAGs but is expensive for a single weekly ML job.

Designing Reliable ML DAGs

Idempotent tasks: rerunning a task for the same logical date should overwrite, not duplicate, outputs (for example, write to a date-partitioned table).

Pass the logical date into every query and pipeline parameter, so backfills reproduce historical runs correctly.

Keep tasks thin: Airflow tasks should submit and monitor work in managed services, not do the heavy processing.

Separate data readiness from ML logic: sensors and data checks in Airflow, and ML validation gates inside the Agent Platform Pipeline.

Alert on failures and SLA misses so late data doesn't silently produce stale models.

Worked Scenario

A retailer's data platform team runs 400 Airflow DAGs that load POS, e-commerce, and supplier data. The ML team needs weekly demand model retraining that must start only after all supplier feeds for the week land, and must backfill four weeks after an outage.

Add a DAG in the existing Managed Airflow environment with sensors for the supplier feeds and catchup/backfill for missed weeks.

Trigger the existing Agent Platform Pipeline (validation, training, evaluation, and conditional deployment) with RunPipelineJobOperator , passing the week as a parameter.

The ML team keeps lineage and evaluation gates in Pipelines. The platform team keeps one orchestration layer for data dependencies.