Study guide

6.2 Implement Training Pipelines

Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling

2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM

2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces

3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries

3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries

4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML

4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces

5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps

5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning

6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison

6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs

7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management

7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving

8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback

8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback

9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts

9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts

10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC

10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI

11 Chapter 11: Deploying and Managing Foundation Models

11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads

12 Chapter 12: Prompt Design, Variants, and Git Version Control

12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git

13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows

13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows

14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging

14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging

15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation

15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing

16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production

16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models

6.2 Implement Training Pipelines

Key Takeaways Explain This Topic AI

A pipeline job YAML has type: pipeline and a required jobs map that composes command or sweep steps into a directed acyclic graph. Connect steps with ${{parent.jobs.<step>.outputs.<name>}} and ${{parent.inputs.<name>}}.

Pass data as typed outputs (uri_folder, uri_file, mltable, mlflow_model), not ad-hoc copies. Components declare the same types on their interface so teams can develop steps independently.

Reuse requires is_deterministic true, settings.force_rerun false (the default), and identical code, environment, inputs, outputs, and run settings. force_rerun true reruns every step; those child jobs cannot be reused by later pipelines.

Schedules are time-based only (recurrence or cron). Azure Machine Learning v2 schedules do not support event-based triggers; cron DAYS and MONTHS values are ignored. Disable a schedule before you delete it.

Use a pipeline when you have multiple reusable steps; use a single command job for one training script. The exam prefers YAML or SDK v2 over Designer-only answers. Azure Machine Learning pipelines are not Azure Pipelines and not Azure Data Factory.

Last updated: August 2026

Implement Training Pipelines

Quick Answer: An Azure Machine Learning pipeline is a directed acyclic graph (DAG) of components (usually type: command steps, optionally sweeps). The job YAML itself is type: pipeline . Steps pass typed outputs ( uri_folder , mltable , mlflow_model ) through ${{parent.jobs.<step>.outputs.<name>}} . Unchanged deterministic steps reuse by default; settings.force_rerun: true reruns everything. Schedules are recurrence or cron, not event triggers. Prefer YAML / SDK v2 over Designer-only designs on this exam.

Exam AI-300 Domain 2 asks you to implement training pipelines . A single command job is the right unit for one script. A pipeline is the right unit when preprocess, train, evaluate, and (later) register or deploy must run as one workflow with lineage , reuse , and schedules .

Why a pipeline instead of one command job

Microsoft's pipeline concept page gives two operational reasons: standardize Machine Learning Operations (MLOps) so data engineers, scientists, and ML engineers own different steps, and cut cost by reusing outputs and by placing each step on the cheapest adequate compute.

  • Need Single command job Pipeline job
  • One training script, one environment Yes — Chapter 5 Unnecessary ceremony

Prep on CPU, train on GPU, evaluate on CPU Awkward; one compute for the whole job Each child job can override compute

  • Change only the trainer, keep last week's cleaned data You rerun everything Unchanged prep reuses
  • Weekly retrain External orchestrator or you click Run Workspace schedule on the pipeline
  • Versioned, reusable steps across teams Copy a folder Registered components

Do not confuse Azure's three "pipeline" products:

Azure Machine Learning pipelines — data to model. Caching, distribution, component reuse. This exam bullet.

Azure Data Factory pipelines — data to data. Strongly typed movement.

Azure Pipelines (Azure DevOps) — code and model to app. CI/CD, approvals, GitHub Actions' cousin.

A GitHub Actions workflow that calls az ml job create --file pipeline.yml is DevOps around an Azure Machine Learning pipeline, not a substitute for the DAG.

YAML anatomy ( type: pipeline )

Schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json . Required keys are type: pipeline and jobs . $schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json

  • type: pipeline
  • display_name: claims-train-eval

settings:

  • default_compute: azureml:cpu-cluster
  • force_rerun: false
  • continue_on_step_failure: false

inputs:

raw_images:

  • type: uri_folder
  • path: azureml:claims-images:2026-08-15

outputs:

trained_model:

type: mlflow_model

jobs:

prep:

  • type: command
  • component: azureml:claims_prep@latest

inputs:

raw: ${{parent.inputs.raw_images}}

outputs:

cleaned:

type: uri_folder

train:

  • type: command
  • component: ./train.yml
  • compute: azureml:gpu-cluster

inputs:

data: ${{parent.jobs.prep.outputs.cleaned}}

outputs:

model: ${{parent.outputs.trained_model}}

Submit with az ml job create --file pipeline.yml . SDK v2 uses an @pipeline decorated function whose body calls component callables and returns named outputs; set pipeline_job.settings.default_compute = "cpu-cluster" or "serverless" . For serverless in YAML, default_compute: azureml:serverless .

Current child job types inside jobs are command and sweep . A distributed trainer is still a command step that happens to include a distribution block from section 6.1. The pipeline does not become type: pytorch .

Settings you will be tested on:

default_compute — every step inherits it unless the step sets its own compute . That is how prep stays on CPU and train moves to GPU.

default_datastore — where pipeline outputs land if you do not pin a path; otherwise the workspace blob datastore.

continue_on_step_failure — schema default is true (later independent steps still run). Production retrains often set false so a failed prep cancels the graph.

force_rerun — default false . Covered below.

Studio shows a graph of child jobs under the parent pipeline job. Each child is a real job with its own logs, metrics, and outputs.

Components, typed data, and the DAG

A component is a versioned step interface: name, inputs, outputs, environment, command, code. You studied creating components in Chapter 3. Pipelines consume them.

Literal inputs ( string , number , integer , boolean ) are hyperparameters ( max_epochs , learning_rate ). They do not draw graph edges.

Object inputs/outputs ( uri_file , uri_folder , mltable , mlflow_model ) do draw edges. The prep step's uri_folder cleaned output becomes the train step's input via ${{parent.jobs.prep.outputs.cleaned}} . Pipeline-level inputs use ${{parent.inputs.raw_images}} . Bind a child output up to the pipeline with ${{parent.outputs.trained_model}} on that child's outputs map, as in the CLI v2 examples.

Modes match other v2 jobs: inputs ro_mount / download / direct ; outputs rw_mount / upload . Passing mltable is the right type when the step needs a table blueprint rather than a raw folder. Passing mlflow_model is how evaluate consumes what train just wrote without a side-channel registry write.

is_deterministic on a component defaults to true (reuse by default). Set it false when the step is inherently non-repeatable: pulling from a live URL, drawing fresh random synthetic data, or hitting an external API whose bytes change under the same URI.

Register components with az ml component create --file train.yml and reference component: azureml:my_train@latest (or a pinned version) so the DAG does not embed a local folder path. Local component: ./train.yml is fine while iterating.

Force rerun versus reuse

Reuse is how pipelines save money. Criteria, all required:

Component is_deterministic is true.

Pipeline settings.force_rerun is false.

Component code snapshot , environment definition , inputs and parameters , output settings , and run settings are the same as a previous successful run.

When reuse hits, the service copies status, logs, metrics, and outputs from the original child; Studio draws a recycle icon. It does not re-read a lake folder that you overwrote in place under the same URI. If the asset identity did not change, reuse will happily hand you last week's bytes. That is the same immutability lesson as data assets: pin a new version or a new path when the data actually changed.

force_rerun: true reruns every child. Microsoft also notes that children of a force-rerun pipeline cannot be reused by later jobs. Do not leave force rerun on in production schedules or you pay to recompute prep forever and you poison the reuse cache.

Debug unexpected reruns in this order: force_rerun on either job, is_deterministic , ContentSnapshotId (code changed), environment definition, then Studio graph compare for inputs, parameters, and run settings.

Schedules for retraining

When the DAG is good enough for production, attach a schedule . v2 schedules work on v1 or v2 pipeline jobs and do not require publishing a classic pipeline endpoint.

Recurrence: trigger.type: recurrence with frequency ( minute / hour / day / week / month ) and interval . Optional schedule.hours , minutes , weekdays , start_time , end_time , time_zone (default UTC).

Cron: trigger.type: cron with a crontab expression ( MINUTES HOURS DAYS MONTHS DAYS-OF-WEEK ). In Azure Machine Learning, DAYS and MONTHS values are ignored and treated as * . 15 16 * * 1 is 16:15 UTC every Monday.

Hard limitation: Azure Machine Learning v2 schedules do not support event-based triggers (no "new blob arrived"). If you need storage events, use an external orchestrator and often a batch endpoint wrapping the pipeline.

create_job can point at a YAML file or azureml:<job-name> . You may override settings , inputs , outputs , tags, and experiment name so the scheduled run uses production data while your debug run used a sample. Macro expressions: ${{name}} in output paths; ${{creation_context.trigger_time}} in string inputs.

Manage with az ml schedule create|list|show|update|enable|disable|delete . Disable before delete ; deletion is permanent. Even after you assign a managed identity to the schedule, the author must keep job-run permission or the schedule stops. Recurrence on day 31 skips short months. Triggered jobs are named <schedule_name>-YYYYMMDDThhmmssZ .

Designer versus code

You can author the same component DAG in Azure Machine Learning studio Designer (drag and drop), in Python SDK v2 , or in CLI v2 YAML . Designer is useful for demos and for people who will not touch YAML. This exam's tooling profile is SDK/CLI v2 . When a stem asks how to implement a reusable, scheduled training pipeline in source control, the answer is a type: pipeline YAML (or SDK equivalent) in Git, submitted by az ml job create or GitHub Actions — not "only Designer."

Exam scenario

Claims images land weekly in Azure Data Lake Storage. Prep (CPU, Spark-ish Python) writes a uri_folder of tiles; train (GPU, PyTorch DDP from 6.1) writes an mlflow_model ; eval writes a metrics JSON. Scientists kept pasting three command jobs by hand and rerunning prep for every learning-rate change. The MLOps engineer registers claims_prep , claims_train , and claims_eval components, authors pipeline.yml with default_compute: azureml:cpu-cluster , overrides compute on the train step to the GPU cluster, binds outputs with ${{parent.jobs.prep.outputs.cleaned}} , and creates a cron schedule 0 4 * * 1 that overrides the pipeline input to azureml:claims-images@latest . Force rerun stays false so unchanged prep is reused. They do not wait for a blob-created event — v2 schedules cannot fire on that.

Common trap

Calling Azure Pipelines or Data Factory the Azure Machine Learning training DAG. Sibling traps: leaving force_rerun: true on a weekly schedule; expecting reuse after you overwrite files at the same URI; using Designer as the only source of truth with no YAML in Git; assuming cron DAYS / MONTHS fields work; treating a pipeline as type: pytorch instead of putting distribution on the train child command .

Loading diagram... Training pipeline DAG with typed outputs and a schedule