ExamFinch.com My Study

Study guide

6.3 Compare Model Performance Across Jobs

Practice this exam

Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling

2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM

2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces

3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries

3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries

4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML

4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces

5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps

5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning

6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison

6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs

7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management

7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving

8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback

8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback

9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts

9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts

10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC

10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI

11 Chapter 11: Deploying and Managing Foundation Models

11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads

12 Chapter 12: Prompt Design, Variants, and Git Version Control

12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git

13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows

13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows

14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging

14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging

15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation

15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing

16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production

16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models

6.3 Compare Model Performance Across Jobs

Key Takeaways Explain This Topic AI

Compare jobs in Azure Machine Learning studio (select jobs, compare parameters, metrics, and tags) or in code with MLflow mlflow.search_runs. Azure Machine Learning SDK v2 has no native metric query API — use MLflow.

Sweeps and pipelines create parent/child runs. List trials with filter_string tags.mlflow.parentRunId equal to the parent run ID. Register only the winning child, not the parent job and not every trial.

Agree a primary metric name and direction (maximize val_auroc, minimize log_loss) before you sort. search_runs returns only the last logged value of each metric; use MlflowClient.get_metric_history for curves.

Tags, experiment names, and Studio lineage (data assets in, models out) are how you reconstruct why a candidate won. Custom tags such as candidate or validation_set=holdout-2026-08 help filters.

Never treat two scalar metrics as comparable if they were computed on different validation sets, different splits, or different preprocessing. Equal names do not mean equal meaning.

Last updated: August 2026

Compare Model Performance Across Jobs

Quick Answer: Compare training jobs in Azure Machine Learning studio (multi-select jobs → compare parameters, metrics, tags) or with MLflow mlflow.search_runs . Sweeps and pipelines produce parent/child runs; filter tags.mlflow.parentRunId . Sort on one primary metric computed on the same validation set . Register only the winner. SDK v2 does not query metrics natively — MLflow does.

Exam AI-300 Domain 2 asks you to compare model performance across jobs . Chapter 5 taught you to log with MLflow. This section is the decision that follows: which run is champion, and what you are allowed to conclude from the numbers.

Studio compare versus MLflow search

Two complementary tools; the exam expects both.

Studio. On the Jobs page, select two or more jobs (or child jobs under an experiment) and open compare. The compare experience (studio documents it as a preview panel you enable) lines up parameters , metrics , and tags so a human can see that run A used lr=0.01 and scored val_auroc=0.81 while run B used lr=0.001 and scored 0.84 . Graph compare for pipelines (section 6.2) is the sibling feature: it diffs topology, inputs, and settings when you are debugging reuse, not when you are picking a model.

MLflow. Azure Machine Learning Python SDK v2 does not provide native logging or metric-query APIs. Query the workspace tracking store with the MLflow SDK ( mlflow plus azureml-mlflow ), pointed at the workspace tracking URI. import mlflow

  • runs = mlflow.search_runs(
  • experiment_names=["claims-seg"],
  • filter_string="metrics.val_auroc > 0.8 and params.backbone = 'resnet50'",
  • output_format="pandas",
  • )
  • winners = runs.sort_values("metrics.val_auroc", ascending=False)

search_runs returns a pandas DataFrame by default: run identity columns, params.<name> , metrics.<name> (the last logged scalar), and tags. That last-value rule matters: a loss curve collapses to the final point. For the full series, use MlflowClient().get_metric_history(run_id, "val_loss") .

Useful filters (AND only; OR is not supported when MLflow is connected to Azure Machine Learning):

Parameters: params.num_boost_round='100' (operators = , != , like ).

Metrics: metrics.auc>0.8 (numeric comparators).

Tags: tags.framework='torch' .

Attributes: attributes.status = 'Finished' , attributes.user_id = '...' , attributes.duration , attributes.run_id IN (...) .

Status names differ from Studio. Map them before you write a filter:

  • Azure Machine Learning job status MLflow attributes.status
  • Not started / Queue / Preparing Scheduled
  • Running Running
  • Completed Finished
  • Failed Failed
  • Canceled Killed

order_by supports attributes ( attributes.start_time DESC , attributes.duration DESC — duration is an Azure Machine Learning convenience). It does not currently support metrics.* / params.* / tags.* expressions against Azure Machine Learning. Sort those in pandas: runs.sort_values("metrics.val_auroc", ascending=False) .

Search across experiments with experiment_ids=[...] (an array — useful when two people logged the same model in different experiments) or search_all_experiments=True . If you pass none of those, MLflow searches only the active experiment ( mlflow.set_experiment ).

Parent and child runs

Hyperparameter sweeps (Chapter 5) and pipelines (section 6.2) mint a parent job plus children. The parent is the orchestrator: it does not usually hold the metric you care about. Each trial or step is a child.

  • MLflow records the relationship on the child as tag mlflow.parentRunId : child_runs = mlflow.search_runs(
  • filter_string=f"tags.mlflow.parentRunId='{parent_run_id}'"
  • )
  • best = child_runs.sort_values("metrics.val_auroc", ascending=False).iloc[0]

Compare siblings that share a parent when you are picking a sweep winner. Comparing a pipeline's prep step metric to a train step metric is meaningless. Comparing two unrelated command jobs is valid only if they logged the same metric on the same holdout.

Primary metric, tags, and lineage

Pick one primary metric before the meeting, the same way AutoML makes you pick one. Accuracy on an imbalanced fraud set is a vanity number; val_auroc or val_pr_auc is a decision number. Direction matters: higher AUROC wins; lower log loss wins. If one job logged accuracy and another logged val_auroc , you do not have a comparison — you have two posters.

Tags are the cheap index you will thank yourself for:

  • framework=torch , backbone=resnet50 , data=claims-images:2026-08-15
  • role=candidate versus role=champion
  • validation_set=holdout-2026-08 so a later engineer can filter to comparable runs

Studio lineage on a job shows data assets consumed and models produced. When a candidate looks too good, open lineage first: did it train on the evaluation set? Did it use a different data-asset version than the champion? Lineage plus tags beat screenshot metrics in an audit.

Download artifacts ( mlflow.artifacts.download_artifacts ) when the metric is not enough — confusion matrices, reliability plots, Responsible AI dashboards (Chapter 7). Load a logged model with mlflow.<flavor>.load_model(f"runs:/{run_id}/{artifact_path}") for a local sanity check before you promote.

Register only the winner

The compare step ends in a promotion decision , not a bulk register. Workflow:

Filter to completed children of the sweep or to comparable command jobs.

Confirm every remaining run used the same validation asset and split .

Sort on the primary metric; break ties with a secondary metric or cost (GPU hours, latency).

Spot-check lineage, tags, and a few artifacts on the top run.

Register that run's model into the workspace model registry (MLflow model format is the AI-300 default — Chapter 7). Do not register every trial "for completeness." Cluttered registries become the next outage.

The parent sweep job is not the model. Registering the parent by mistake packages an empty or incomplete artifact. Register the winning child run .

Scheduled pipelines from 6.2 should compare (or at least gate on a metric threshold) before they register. A weekly retrain that always registers model:latest without beating the champion is how production quietly regresses.

The validation-set trap

This is the skill-measured bullet's most important caution: a higher number is not a better model if the metric was computed on different data.

Concrete failure modes:

Job A scored on a 10 percent random split of last week's tiles; job B scored on a hand-picked "easy" folder.

Job A used azureml:claims-val:3 ; job B used azureml:claims-val:4 after a labeling correction.

Job A reported training accuracy; job B reported validation AUROC; someone compared 0.97 to 0.84 and kept A.

Prep changed (new augmentation) for B, but both jobs still named the metric val_auroc .

Equal metric names do not mean equal meaning . Before you sort, pin the validation data asset version (or an mltable that lists exact files) in both jobs and tag it. If you cannot prove the holdout is the same, you are not comparing model performance — you are comparing experimental setups.

Exam scenario

A sweep of 40 trials trains claims-segmentation models under parent job gentle_oak . Trials log val_auroc and val_loss via MLflow. A scientist opens the parent, sees val_auroc=0.79 on the parent row (a leftover diagnostic), and almost registers the parent. The MLOps engineer instead runs mlflow.search_runs with tags.mlflow.parentRunId='<gentle_oak_id>' , keeps attributes.status = 'Finished' , sorts metrics.val_auroc descending in pandas, and inspects the top three in studio compare. Two of the three used azureml:claims-val:2 ; the 0.91 run used a different, smaller val folder. They discard 0.91, register the 0.88 run that shares the agreed holdout, and tag it role=candidate pending Responsible AI checks in Chapter 7.

Common trap

Comparing metrics that were not computed on the same validation set — or comparing training loss to validation AUROC because both are "the number in studio." Sibling traps: registering every sweep child; querying metrics with Azure Machine Learning SDK v2 instead of MLflow; filtering MLflow status with Studio words ( Completed instead of Finished ); using OR in filter_string ; treating the pipeline parent as the model; ranking on search_runs last-value metrics when you actually needed the best checkpoint mid-run from get_metric_history .