Study guide
5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage
Google Cloud Professional Machine Learning Engineer Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
0.1 Exam Overview, Blueprint & Certification Logistics 0.2 Study Strategy, Scenario Mechanics & 8-Week Roadmap
2 Section 1: Architecting Low-Code AI Solutions (~13%)
1.1 BigQuery ML: Model Types, SQL Syntax and In-Database Evaluation 1.2 Feature Engineering and Feature Selection in BigQuery ML 1.3 Generating Predictions with BigQuery ML 1.4 AutoML on Agent Platform: Tabular, Image, Text and Video Training 1.5 Fine-Tuning Gemini Models from BigQuery 1.6 Model Garden: Evaluating and Selecting the Right Model 1.7 Industry-Specific AI APIs: Document AI, Vision, Translate and Speech 1.8 Generative Media and Models as a Service: Gemini, Imagen and Veo 1.9 Optimizing Gemini Applications for Cost, Latency and Availability
3 Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)
2.1 Organizing and Exploring Tabular, Text, Image and Video Data 2.2 Choosing the Preprocessing Tool: BigQuery, Dataflow, Spark and In-Memory Python 2.3 Creating and Consolidating Features in Agent Platform Feature Store 2.4 Data Privacy, PII Handling and Sensitive Information 2.5 Notebook Environments: Collaboration and Security Best Practices 2.6 Prototyping in Workbench and Colab Enterprise with PyTorch, scikit-learn and JAX 2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks 2.8 Choosing the Development and Experimentation Environment 2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation 2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance
4 Section 3: Scaling Prototypes into ML Models (~21%)
3.1 Choosing the Model Type: ARIMA, DNN, LLM and Classical Baselines 3.2 Choosing the Product: AutoML, BigQuery ML, Custom Training and Pipelines 3.3 Choosing the Deployment Strategy at Design Time 3.4 Modeling Techniques for Interpretability Requirements 3.5 Organizing Training Data on Cloud Storage and BigQuery 3.6 Ingesting Structured and Unstructured Data into Training Pipelines 3.7 Training SDKs and Services: Custom Training, Kubeflow on GKE, AutoML and Tabular Workflows 3.8 Troubleshooting ML Model Training Failures 3.9 Hyperparameter Tuning, Bayesian Optimization and Vizier 3.10 Model Evaluation Metrics, Transfer Learning and Fine-Tuning 3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU 3.12 Distributed Training Strategies, Hardware Acceleration and Performance Optimization
5 Section 4: Serving and Scaling Models (~20%)
4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE 4.2 Rollout Strategies: Canary, Traffic Splitting, Shadow and Blue-Green 4.3 Tuning Models for Production: Vector Search and Model Compression 4.4 Deploying Models to Public and Private Endpoints 4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge 4.6 Scaling the Serving Backend for Throughput
6 Section 5: Automating and Orchestrating ML Pipelines (~18%)
5.1 Validating Data and Models Inside Pipelines 5.2 Agent Platform Pipelines: Kubeflow Pipelines (KFP) and TFX Architecture 5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage 5.4 Ray on Agent Platform for Distributed Python Workloads 5.5 MLOps CI/CD Automation, Retraining Triggers and Skew Prevention
7 Section 6: Monitoring AI Solutions (~13%)
6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor 6.2 Responsible AI, Explainable AI, Fairness Metrics and Model Governance 6.3 Model Monitoring: Feature Skew and Drift Detection 6.4 Operational Troubleshooting, Performance Profiling and Cost Optimization 6.5 Monitoring, Testing and Evaluating Generative AI in Production
5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage
Key Takeaways Explain This Topic AI
Lightweight Python Function components (@dsl.component) allow rapid authoring by packaging standalone Python code and installing dynamic pip packages into a base image at runtime.
Containerized Components (@dsl.container_component) provide complete environment isolation and performance optimization by executing pre-built custom Docker container images with compiled C++, CUDA, or proprietary dependencies.
KFP v2 enforces a strict architectural boundary between Parameters (small primitive values passed via command-line arguments and logged in task specs) and Artifacts (large, complex objects like Datasets and Models stored in Cloud Storage and passed by URI reference).
Vertex ML Metadata (MLMD) tracks the four fundamental relational entities of ML systems: Contexts, Executions, Artifacts, and Events, enabling full auditability, reproducibility, and visual lineage tracking.
Vertex AI Pipelines step caching hashes the component specification, container image, input parameter values, and upstream input artifact IDs to skip execution of unchanged DAG nodes, dramatically reducing training costs and cycle times.
Last updated: September 2026
5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage
Building enterprise-grade ML pipelines requires rigorous modularity, robust artifact handling, and comprehensive auditability. In production environments, an ML pipeline is not merely a script that runs; it is a software system where every data transformation, trained weight matrix, evaluation metric, and deployment action must be tracked, reproducible, and debuggable.
The Kubeflow Pipelines (KFP) v2 SDK paired with Vertex ML Metadata (MLMD) provides the foundation for component reusability, artifact immutability, and end-to-end data provenance across the Google Cloud AI ecosystem.
1. KFP v2 Component Architectures: Lightweight vs. Containerized
When developing reusable pipeline tasks in KFP v2, ML engineers choose between two primary component authoring paradigms: Lightweight Python Function-based Components and Custom Containerized Components . +---------------------------------------------------------------------------------------------------------+
- | KFP v2 COMPONENT AUTHORING SPECTRUM |
- +------------------------------------+------------------------------------+-------------------------------+
- | ATTRIBUTE | LIGHTWEIGHT FUNCTION COMPONENT | CUSTOM CONTAINERIZED COMPONENT|
- +------------------------------------+------------------------------------+-------------------------------+
- | Decorator Syntax | `@dsl.component` | `@dsl.container_component` |
- | Implementation | Self-contained Python function | Custom Dockerfile & entrypoint|
- | Environment Setup | `packages_to_install` at runtime | Pre-built, baked container |
- | Startup Latency | Higher (downloads pip packages) | Minimal (instant startup) |
- | Complex Dependencies | Limited to pip wheels | Full OS/C++/CUDA/binary access|
- | Maintenance Overhead | Very low; no Docker build pipeline | Moderate; requires CI builds |
- | Best Suited For | Rapid prototyping, standard Python | Production, heavy C++, CUDA |
- +------------------------------------+------------------------------------+-------------------------------+
- 1. Lightweight Python Function-Based Components
Lightweight components allow developers to define pipeline tasks directly in Python without manually writing Dockerfiles, building container images, or maintaining container registries. The KFP SDK automatically serializes the Python function, embeds it into a generated task spec, and executes it inside the specified base_image . from kfp.dsl import component, Input, Output, Dataset, Metrics, Model
- @component(
- base_image="python:3.10-slim",
- packages_to_install=["pandas==2.1.0", "scikit-learn==1.3.0", "joblib==1.3.2"]
- )
- def evaluate_model(
- test_dataset: Input[Dataset],
- model_artifact: Input[Model],
- metrics: Output[Metrics],
- threshold: float = 0.85
) -> bool:
- import pandas as pd
- import joblib
- from sklearn.metrics import accuracy_score, precision_score, recall_score
- # Read input dataset via GCS local mount path
- df = pd.read_parquet(test_dataset.path)
- X_test, y_test = df.drop(columns=["target"]), df["target"]
- # Load model weights
- clf = joblib.load(f"{model_artifact.path}/model.joblib")
- predictions = clf.predict(X_test)
- acc = accuracy_score(y_test, predictions)
- prec = precision_score(y_test, predictions)
- rec = recall_score(y_test, predictions)
- # Log scalar metrics directly into Vertex ML Metadata
- metrics.log_metric("accuracy", float(acc))
- metrics.log_metric("precision", float(prec))
- metrics.log_metric("recall", float(rec))
- return bool(acc >= threshold)
- 2. Custom Containerized Components
For enterprise production pipelines requiring strict environment immutability, complex non-Python binaries (e.g., OpenCV with custom C++ extensions, CUDA 12.2 drivers, proprietary C++ libraries), or sub-second cold start times, Containerized Components provide full control. The container image is pre-built, vulnerability-scanned, and pushed to Google Artifact Registry . from kfp.dsl import container_component, ContainerSpec, Input, Output, Dataset, Model
- @container_component
- def run_distributed_cpp_preprocessor(
- raw_data: Input[Dataset],
- processed_data: Output[Dataset],
- num_shards: int = 16
):
- return ContainerSpec(
- image="us-central1-docker.pkg.dev/my-gcp-project/ml-repo/cpp-preprocessor:v1.2.0",
- command=["/usr/local/bin/preprocess_engine"],
- args=[
- "--input_path", raw_data.path,
- "--output_path", processed_data.path,
- "--shards", str(num_shards)
- ]
- )
- 2. Artifacts vs. Parameters: Data Passing Mechanics
In KFP v2, data passing between DAG nodes is strictly typed and categorized into two distinct primitives: Parameters and Artifacts . +---------------------------------------------------------------------------------------------------------+
- | PARAMETERS VS. ARTIFACTS IN KFP v2 |
- +------------------------------------+------------------------------------+-------------------------------+
- | DIMENSION | PARAMETER | ARTIFACT |
- +------------------------------------+------------------------------------+-------------------------------+
- | Data Scale | Small primitives (< kilobyte) | Arbitrary size (MB, GB, TB) |
- | Data Types | `int`, `float`, `str`, `bool`, `dict` | `Dataset`, `Model`, `Metrics` |
- | Storage Location | Vertex ML Metadata DB / JSON Spec | Cloud Storage (gs:// bucket) |
- | Passing Mechanism | Command-line CLI flag / Env Var | Cloud Storage URI path pointer|
- | Lineage Representation | Stored in Execution node metadata | Stored as distinct MLMD Node |
- | UI Visualization | Table of key-value pairs | Interactive graphs, confusion |
- | | | matrices, markdown, tables |
- +------------------------------------+------------------------------------+-------------------------------+
- Standard KFP v2 Artifact Types
Dataset ( Input[Dataset] , Output[Dataset] ): Represents raw, processed, or partitioned tabular/image data on Cloud Storage.
Model ( Input[Model] , Output[Model] ): Represents serialized model weight directories (SavedModel, joblib, PyTorch .pt , ONNX).
Metrics ( Output[Metrics] ): Scalar numeric key-value pairs (accuracy, RMSE, loss) logged via metrics.log_metric() , rendered in comparison tables.
ClassificationMetrics ( Output[ClassificationMetrics] ): Specialized artifact supporting rich visual curves in the Vertex AI Console (Confusion Matrices, ROC curves, Precision-Recall curves) via log_confusion_matrix() and log_roc_curve() .
HTML & Markdown ( Output[HTML] , Output[Markdown] ): Rich visual diagnostic reports rendered directly in the Vertex AI Pipelines execution dashboard. [ Component A (Data Splitter) ]
- |
- | Produces: Output[Dataset] (Writes dataframe to gs://my-bucket/artifacts/.../data.csv)
- v
- [ Cloud Storage Artifact Store ] <====== Immutable GCS URI Pointer
- |
- | Consumes: Input[Dataset] (Reads dataframe from GCS mount)
- v
- [ Component B (Model Trainer) ]
- 3. Vertex ML Metadata (MLMD): Provenance and Lineage
Vertex ML Metadata (MLMD) is a managed metadata repository that automatically records every action, parameter, and output generated across Vertex AI Pipelines, Custom Training Jobs, and Model Registry deployments. +---------------------------------------------------------------------------------------------------------+
- | VERTEX ML METADATA (MLMD) CORE GRAPH ENTITIES |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | +-------------------------------------------------------------------------------------------------+ |
- | | CONTEXT (e.g., PipelineRun: 'churn-pipeline-20260901-0830', Experiment: 'churn-hyperopt-v2') | |
- | +-------------------------------------------------------------------------------------------------+ |
- | | |
- | | Groups & Scopes |
- | v |
- | +-------------------+ EVENT (Input) +-------------------+ |
- | | ARTIFACT (Data) | -------------------------------------> | EXECUTION (Task) | |
- | | Type: Dataset | | Type: Trainer | |
- | | URI: gs://.../raw | | State: COMPLETE | |
- | +-------------------+ +-------------------+ |
- | | |
- | | EVENT (Output) |
- | v |
- | +-------------------+ |
- | | ARTIFACT (Model) | |
- | | Type: Model | |
- | | URI: gs://.../mod | |
- | +-------------------+ |
- +---------------------------------------------------------------------------------------------------------+
- The Four Core MLMD Entities
Artifacts : Concrete entities or data objects created or consumed by an ML workflow (e.g., datasets, model binaries, evaluation metric JSONs, vocabulary files). Each artifact possesses a unique URI in Cloud Storage and a schema type.
Executions : Individual execution steps or computational tasks within a workflow (e.g., a data preprocessing container run, a distributed training job, a batch prediction scoring run). Executions record runtime parameters, start/end timestamps, container images, and completion status.
Events : Directed relational edges that connect Executions and Artifacts . An Event denotes whether an artifact was an INPUT consumed by an execution or an OUTPUT produced by an execution.
Contexts : Logical groupings that cluster executions and artifacts together (e.g., a specific PipelineRun , an Experiment , or an ExperimentRun ).
Why Lineage Matters for Enterprise Governance
Reproducibility & Debugging : If a production model begins returning anomalous predictions, ML engineers can query MLMD to trace the exact lineage: identifying the specific training execution ID, the exact Git commit SHA embedded in metadata, the hyperparameter dictionary, and the specific raw GCS data shards consumed during training.
Regulatory Auditability : In regulated industries (finance, healthcare), MLMD provides immutable proof of data provenance, ensuring that models were trained strictly on compliant, audited data partitions.
4. Vertex AI Pipelines Execution Caching
One of the most powerful cost- and time-saving features of Vertex AI Pipelines is Execution Caching . +---------------------------------------------------------------------------------------------------------+
- | EXECUTION CACHE FINGERPRINTING |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | COMPONENT CACHE HASH = HASH( |
- | 1. Component Specification (Python code, commands, entrypoints) |
- | 2. Container Image URI & Digest (e.g., us-docker.pkg.dev/.../preprocessor@sha256:abc...) |
- | 3. Input Parameter Values (e.g., learning_rate=0.01, max_depth=5) |
- | 4. Upstream Input Artifact IDs & Checksums (e.g., dataset_artifact_id_89412) |
- | ) |
- | |
- | [ Cache Hit Check ] |
- | * If Hash Matches Previous Execution in MLMD: REUSE OUTPUT ARTIFACTS (Execution Time: 0s, Cost: $0)|
- | * If Hash Differs (Code changed, data updated, params altered): EXECUTE NEW CONTAINER |
- +---------------------------------------------------------------------------------------------------------+
- Caching Mechanics and Best Practices
Cache Key Calculation : Vertex AI computes a deterministic fingerprint for each task based on four factors: the component specification/code, the container image URI/digest, all input parameter values, and the upstream input artifact IDs.
Cache Hits : When a component is submitted with identical inputs and code, Vertex AI skips container execution entirely, instantly linking the previously generated output artifacts into the current pipeline run. A 4-hour feature extraction step on 100 GB of data executes in 0 seconds.
- Configuring Caching in Code : # Enable or disable caching at the individual component level
- train_task = train_model(dataset=ingest_task.output)
- train_task.set_caching_options(enable_caching=True) # Or False to force re-execution
- # Configure pipeline-level caching during submission
- job = aiplatform.PipelineJob(
- display_name="churn-training-job",
- template_path="churn_pipeline.yaml",
- enable_caching=True # Enables caching across all tasks in DAG
- )
- job.submit()
Important
When to Disable Caching : Always disable caching ( enable_caching=False ) for components that read non-deterministic external data sources directly (e.g., querying CURRENT_DATE() from BigQuery, fetching real-time API feeds, or downloading from static GCS URIs that get overwritten in-place without new artifact IDs).