Study guide
6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
Google Cloud Professional Machine Learning Engineer PMLE
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & the Agent Platform Map
1.1 PMLE Exam Facts, Policies & the June 2026 Blueprint 1.2 From Vertex AI to Gemini Enterprise Agent Platform: The Product Name Map
2 Chapter 2: Low-Code ML with BigQuery ML
2.1 Choosing, Training & Predicting with BigQuery ML Models 2.2 Feature Engineering & Feature Selection in BigQuery ML 2.3 Fine-Tuning & Evaluating Gemini Models from BigQuery
3 Chapter 3: AutoML & Google Cloud AI APIs
3.1 Training Models with Agent Platform AutoML 3.2 Building with Document AI, Vision, Translation & Other AI APIs
4 Chapter 4: Foundation Models in Model Garden
4.1 Evaluating & Selecting Models in Model Garden 4.2 Building & Tuning Solutions with Gemini, Imagen, Veo & MaaS 4.3 Optimizing Gemini Applications for Cost, Latency & Availability
5 Chapter 5: Exploring, Preprocessing & Protecting Data
5.1 Organizing & Exploring Tabular, Text & Image Data 5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python 5.3 Creating & Consolidating Features in Agent Platform Feature Store 5.4 Protecting PII & Sensitive Data in ML Workflows
6 Chapter 6: Notebooks, Prototyping & Experiment Tracking
6.1 Workbench vs. Colab Enterprise: Secure, Collaborative Notebooks 6.2 Prototyping with PyTorch, scikit-learn, JAX & Model Garden in Notebooks 6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
7 Chapter 7: Evaluating Predictive & Generative AI Solutions
7.1 Evaluation Metrics for Predictive Models 7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
8 Chapter 8: Choosing the Modeling Approach
8.1 Choosing the Model Type: ARIMA, Trees, DNNs & LLMs 8.2 Choosing the Product & Deployment Strategy 8.3 Modeling Techniques for Interpretability Requirements
9 Chapter 9: Training Data & Training SDKs
9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery 9.2 Custom Training on Agent Platform: Containers, Jobs & Worker Pools 9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices
10 Chapter 10: Troubleshooting, Hyperparameter Tuning & Fine-Tuning
10.1 Troubleshooting ML Training Failures 10.2 Hyperparameter Tuning on Agent Platform 10.3 Fine-Tuning Foundation Models & When to Tune
11 Chapter 11: Training Hardware & Distributed Training
11.1 Choosing CPUs, GPUs & TPUs for Training 11.2 Distributed Training: Data & Model Parallelism
12 Chapter 12: Deploying Models for Inference
12.1 Batch vs. Online Inference on Agent Platform 12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE 12.3 Prebuilt & Custom Serving Containers
13 Chapter 13: Model Versions, Rollouts & Inference Logic
13.1 Organizing & Versioning Models in Model Registry 13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits 13.3 Inference Preprocessing & Postprocessing
14 Chapter 14: Scaling Online Serving
14.1 Serving Features Online with Feature Store 14.2 Public, Dedicated & Private Endpoints 14.3 Serving Hardware & Autoscaling for Throughput 14.4 Tuning Models for Production Training & Serving
15 Chapter 15: End-to-End ML Pipelines
15.1 Validating Data & Models in Pipelines 15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching 15.3 Ensuring Consistent Preprocessing Between Training & Serving
16 Chapter 16: Orchestration with Managed Airflow & Ray
16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow 16.2 Ray on Agent Platform for Distributed ML Workloads
17 Chapter 17: Automating Retraining & CI/CD/CT
17.1 Choosing a Retraining Policy 17.2 CI/CD/CT for ML with Cloud Build
18 Chapter 18: Securing AI Systems, Responsible AI & Explainability
18.1 Protecting AI Systems from Data & Model Exfiltration 18.2 Defending Gen AI Apps: Safety Filters, Model Armor & Regex Guardrails 18.3 Responsible AI: Detecting & Monitoring Bias 18.4 Model Explainability on Agent Platform
19 Chapter 19: Monitoring AI Solutions
19.1 Configuring Model Monitoring on Agent Platform 19.2 Training-Serving Skew, Data Drift, Concept Drift & Attribution Drift 19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
6.3 Experiment Environments, Tracking & Lineage with Experiments and ML Metadata
Key Takeaways Explain This Topic AI
Experiments on Agent Platform tracks the steps, inputs such as parameters and datasets, and outputs such as models and metrics of each experiment run.
Autologging in the Agent Platform SDK uses MLflow autologging and captures parameters and metrics from frameworks including Keras, scikit-learn, XGBoost, LightGBM, and PyTorch Lightning.
Time-series metrics logged with log_time_series_metrics are stored in a managed TensorBoard instance tied to the experiment or run.
ML Metadata records lineage as a graph in which artifacts and executions are nodes, events are edges, and contexts group related nodes such as a pipeline run.
Experiment runs cost nothing extra; you pay only for the resources used during the experiment.
Last updated: September 2026
Section 2.3 of the exam guide covers choosing the appropriate Google Cloud environment for development and experimentation (Experiments on Agent Platform, Agent Platform Pipelines, and Kubeflow Pipelines) and tracking and comparing model artifacts, versions, and lineage (Experiments and ML Metadata). Evaluation metrics themselves are covered in Chapter 7.
Choosing the Experimentation Environment
Environment Best for Trade-offs
Notebook + Experiments SDK (Colab Enterprise or Workbench) Early, interactive exploration: trying features, algorithms, and prompts Easy to lose reproducibility if code and data versions aren't logged
Agent Platform Pipelines (serverless KFP) Repeatable multi-step experiments: data prep → train → evaluate, with automatic metadata and lineage. Runs can be grouped into an experiment Needs code packaged as components. Best once the workflow stabilizes
Kubeflow Pipelines on GKE (self-managed) Organizations standardized on Kubeflow, needing portability across on-premises and clouds, or deep Kubernetes customization You operate the cluster, upgrades, and security
Custom training jobs with Experiments Full-scale or distributed training runs that must be compared Longer iteration loop than notebooks
Pipelines built with the Kubeflow Pipelines (KFP) SDK can run on Agent Platform Pipelines, so teams can prototype locally or on self-managed Kubeflow and move to the managed service. Given the exam's preference for managed services, Agent Platform Pipelines is usually the right answer unless the scenario explicitly requires self-managed Kubeflow.
Framework matters too. Experiments works with any Python framework through the SDK and integrates deeply with TensorFlow. Autologging , built on MLflow autologging, captures parameters and metrics from Fastai, Gluon, Keras, LightGBM, PyTorch Lightning, scikit-learn, Spark, Statsmodels, and XGBoost .
Experiments on Agent Platform
An experiment groups experiment runs . Each run records:
- Steps (preprocessing, training)
- Inputs (algorithm, parameters, dataset)
- Outputs (models, checkpoints, metrics)
Experiment runs cost nothing extra. You pay only for compute and storage used. from google.cloud import aiplatform
- aiplatform.init(project=PROJECT, location=REGION,
- experiment="churn-xgb", experiment_tensorboard=TB_RESOURCE)
- aiplatform.start_run("xgb-depth6-lr0p05")
- aiplatform.log_params({"max_depth": 6, "learning_rate": 0.05, "data_version": "v12"})
# ... train ...
- aiplatform.log_metrics({"auc_pr": 0.61, "recall_at_p90": 0.44})
- aiplatform.end_run()
- API Records
- log_params Hyperparameters and configuration
- log_metrics Summary metrics for the run
log_time_series_metrics Per-step metrics such as loss curves, stored in a managed TensorBoard instance (documented as Vertex AI TensorBoard)
- autolog() Automatic parameter and metric capture for supported frameworks
- PipelineJob.submit(experiment=...) Adds a pipeline run to an experiment so pipeline runs can be compared
The console shows runs side by side, and the SDK returns runs as DataFrames for analysis. TensorBoard adds loss curves, histograms, computation graphs, embedding projections, and a shareable link, and the TensorBoard profiler helps diagnose slow training.
ML Metadata and Lineage
Agent Platform ML Metadata builds on the open-source ML Metadata (MLMD) library and stores metadata as a graph inside a regional MetadataStore (usually one per project).
- Concept Meaning Example
- Artifact Data produced or consumed by the workflow Dataset, model, evaluation metrics
- Execution A workflow step with runtime parameters Training step, validation step
- Event Edge linking an artifact to an execution as input or output "Training execution consumed dataset v12"
- Context Groups artifacts and executions One pipeline run, or one experiment run
MetadataSchema Type definition (system schemas such as system.Model , or custom schemas) Validates metadata fields
Pipelines record this graph automatically . Custom code can write it with the SDK. Lineage answers audit questions such as:
Which dataset and hyperparameters produced the model now in production?
Which models were trained on a dataset that turned out to have a labeling bug?
Which model version produced a specific batch prediction?
Versions and the Model Registry
Experiments and metadata track how a model was made. Model Registry tracks which versions exist and which is deployed. Log the winning run's model to the registry as a new version, with a link back to its experiment run. Use aliases such as default or champion to show which version is promoted (Chapter 13).
Worked Scenario
A regulated lender must show auditors, for any production credit model, the training data snapshot, preprocessing code version, hyperparameters, evaluation results, and approver.
Training runs as an Agent Platform Pipeline submitted with experiment="credit-risk" , so every run is comparable and ML Metadata records the lineage automatically.
The pipeline takes the managed dataset version and the container image digest as inputs, so both show up in lineage.
Evaluation metrics are logged to the run. The approved model goes to Model Registry with labels for the approver and ticket ID.
Auditors follow lineage from the deployed model version back to its dataset and run, with no spreadsheets involved.
What to Log in Every Run
A run is only reproducible if you can rebuild it. Log at least:
- Category Examples
- Data Managed dataset version or BigQuery snapshot time, row counts, label distribution
- Code Git commit, container image digest, preprocessing version
- Configuration Hyperparameters, random seeds, hardware (machine type, accelerator count)
- Results Summary metrics, per-slice metrics, loss curves, evaluation artifacts
- Context Owner, purpose ("test focal loss for class imbalance"), links to tickets
- Exam Traps
Tracking experiments in spreadsheets or notebook markdown when the scenario asks for comparable, auditable runs.
Choosing self-managed Kubeflow on GKE when nothing requires it. Agent Platform Pipelines removes cluster operations.
Confusing Experiments (compare runs) with Model Registry (manage versions for deployment) and ML Metadata (lineage graph). Production systems use all three together.