Study guide
2.8 Choosing the Development and Experimentation Environment
Google Cloud Professional Machine Learning Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap
2 Section 1: Architecting Low-Code AI Solutions (~13%)
1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability
3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)
2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance
4 Section 3: Scaling Prototypes into ML Models (~21%)
3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization
5 Section 4: Serving and Scaling Models (~20%)
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput
6 Section 5: Automating and Orchestrating ML Pipelines (~18%)
5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
7 Section 6: Monitoring AI Solutions (~13%)
6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production
2.8 Choosing the Development and Experimentation Environment
Key Takeaways Explain This Topic AI
Experiments on Agent Platform tracks parameters, metrics, and artifacts across runs and is the lightweight choice while the workflow is still changing.
Agent Platform Pipelines is the right environment once the workflow is stable, multi-step, and must be reproducible or scheduled.
Kubeflow Pipelines on GKE is chosen when the team needs cluster-level control, existing GKE investment, or portability across clouds.
Framework matters: TFX is the natural pipeline authoring layer for TensorFlow workflows, while KFP is framework-agnostic and preferred for PyTorch, scikit-learn, and mixed stacks.
The escalation path is notebook plus Experiments, then a training job, then a pipeline — moving too early adds ceremony, moving too late blocks reproducibility.
Last updated: September 2026
2.8 Choosing the Development and Experimentation Environment
Blueprint reference: Section 2.3, "Choosing the appropriate Google Cloud environment for development and experimentation (e.g., Experiments on Gemini Enterprise Agent Platform, Gemini Enterprise Agent Platform Pipelines, and Kubeflow Pipelines) given the framework."
Note the last three words. The official bullet ties this choice to the framework in use, which is a hint that the exam will present a stack and ask which environment fits it.
The Three Environments
Environment What it is Choose it when
Experiments on Agent Platform A tracking layer that records parameters, metrics, and artifacts per run, comparable in the console The workflow is still changing; you need to compare many quick attempts without building infrastructure
Agent Platform Pipelines Serverless managed execution of KFP or TFX pipelines, per-run billing, no cluster The workflow is stable and multi-step, must be reproducible, scheduled, or triggered, and you do not want to operate infrastructure
Kubeflow Pipelines on GKE Self-managed Kubeflow on a cluster you run You need cluster-level control, have existing GKE investment or on-premises parity requirements, or need portability across environments
The decision is mostly about operational burden versus control . Agent Platform Pipelines removes the cluster entirely and charges per pipeline run plus the resources each step consumes; there is nothing to patch, upgrade, or size when idle. Self-managed Kubeflow on GKE keeps the cluster — and its cost — running continuously, and you own upgrades, but you get full control over the environment, custom operators, and the ability to run the same pipeline definition elsewhere.
That difference produces a reliable exam signal: "we do not want to manage infrastructure" or "we have no platform team" points to Agent Platform Pipelines; "we must run the same pipeline on-premises as well" or "we already operate GKE and need custom cluster configuration" points to Kubeflow on GKE.
Experiments Is Not an Alternative to Pipelines
A common misreading treats these as three mutually exclusive options. In practice Experiments is a tracking layer that composes with both : a notebook logs to it, a training job logs to it, and a pipeline step logs to it. What it is not is an orchestrator — it has no scheduling, no DAG, and no dependency management.
So the real question is usually "is this workload still exploratory, or is it a workflow?" Exploration → Notebook + Experiments
- Repeatable training → Custom training job + Experiments
- Multi-step workflow → Pipeline (Agent Platform or Kubeflow) + Experiments + ML Metadata
- Framework Drives the Authoring Layer
Agent Platform Pipelines executes pipelines authored in either of two SDKs, and the framework in use should decide which.
Authoring SDK Best fit Why
KFP (Kubeflow Pipelines SDK) PyTorch, scikit-learn, XGBoost, JAX, mixed-language, or any containerized step Framework-agnostic; a component is just a container with typed inputs and outputs
TFX TensorFlow end-to-end workflows Opinionated standard components for ingestion, validation, transform, training, evaluation, and pushing, with built-in data and model validation
TFX brings a great deal for free when the stack is TensorFlow — schema inference and anomaly detection through data validation, a transform step whose graph is embedded into the served model, and a model-blessing evaluator. Applied to a PyTorch workflow, most of that machinery is inapplicable and the abstraction becomes overhead. The exam expects the mapping: TensorFlow shop → TFX; heterogeneous or PyTorch shop → KFP.
Both run on the same managed Agent Platform Pipelines service, so the SDK choice does not change where the pipeline executes.
Cost Shapes
Experiments — effectively free tracking; you pay for whatever compute produced the runs.
Agent Platform Pipelines — a small per-run charge plus the compute each step consumes, and nothing at all between runs.
Kubeflow on GKE — the cluster's node pool cost continuously, whether or not pipelines are running, plus the operational cost of maintaining it.
For a team running a handful of pipelines a week, a permanently-running GKE cluster is the wrong economic shape. For a team running hundreds of jobs an hour on a cluster that is already there for other reasons, it may be the right one.
Ray as a Fourth Option
For distributed Python workloads that are not naturally a DAG — hyperparameter sweeps, reinforcement learning, large-scale batch inference in Python — Ray on Agent Platform provides a managed Ray cluster. Unlike a custom training job, which releases its resources at completion, a Ray cluster persists until deleted, which makes it well suited to interactive distributed development and poorly suited to being forgotten about. It is covered in Chapter 5.
Exam Traps
Treating Experiments as an orchestrator. It tracks; it does not schedule.
Choosing self-managed Kubeflow "for flexibility" when the scenario complains about operational burden.
Using TFX for a PyTorch pipeline. KFP is the framework-agnostic choice.
Building a pipeline on day one for a workflow that is still changing daily.
Ignoring idle cluster cost when comparing GKE-hosted Kubeflow to managed pipelines.