Study guide
5.1 Configure Experiment Tracking with MLflow
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
5.1 Configure Experiment Tracking with MLflow
Key Takeaways Explain This Topic AI
Azure Machine Learning SDK v2 has no native logging API; install mlflow plus the azureml-mlflow plugin and log with MLflow APIs.
Command jobs auto-start an MLflow run; notebooks and other interactive sessions must call mlflow.start_run() (or a logging API that creates a run).
The workspace is the MLflow tracking server (azureml:// URI). Azure compute configures it automatically; local, Databricks, and Synapse clients must set the tracking URI.
mlflow.autolog() captures metrics, parameters, and models for supported frameworks, but some flavors skip large models — use log_models=False and log the model yourself when needed.
High-volume metrics should use asynchronous logging (mlflow.config.enable_async_logging() or synchronous=False); Azure Machine Learning still guarantees order and flushes pending metrics when the job finishes.
Last updated: August 2026
Configure Experiment Tracking with MLflow
Quick Answer: Azure Machine Learning SDK v2 has no native logging API . Install mlflow and azureml-mlflow , then log with mlflow.log_metric , log_param , log_artifact , log_model , or mlflow.autolog() . Submitted jobs auto-start an MLflow run; notebooks need mlflow.start_run() . The workspace is the tracking server ( azureml:// URI). Use async logging for high-volume metrics.
Domain 2 of Exam AI-300 asks you to configure experiment tracking with MLflow . That skill is the difference between a training job you can compare, promote, and debug six months later and a GPU bill with no lineage. Machine Learning Operations (MLOps) lives on metrics, parameters, artifacts, and models that are stored against a run identity — not on print() statements in std_log.txt .
SDK v2 has no logging API of its own
If you learned Azure Machine Learning SDK v1, you remember Run.get_context() and run.log() . That API is gone in v2. Microsoft’s current guidance is explicit: there is no logging functionality in the Azure Machine Learning SDK for Python (v2). You track experiments with MLflow Tracking plus the azureml-mlflow plugin that teaches MLflow how to talk to a workspace.
Install both packages in the environment that actually trains: pip install mlflow azureml-mlflow
For a smaller footprint on tracking-only nodes, mlflow-skinny is enough. Asynchronous metric logging additionally needs MLflow 2.8.0 or later and azureml-mlflow 1.55 or later .
Your training script then imports MLflow, not azure.ai.ml logging helpers: import mlflow
- mlflow.log_param("num_epochs", 20)
- mlflow.log_metric("val_auc", 0.91, step=3)
- mlflow.log_artifact("plots/roc.png")
- mlflow.sklearn.log_model(model, artifact_path="model")
Parameters can be any type ( log_param / log_params with a dictionary). Metrics are numeric . Booleans are logged as 0 or 1 . Curves are the same metric logged many times with a step (or a log_batch of Metric entities). Files are artifacts . An MLflow model is a folder that packages the weights plus the flavor files needed to load them — that package is what you later register and deploy with no-code scoring.
Experiments, runs, jobs, and where the URI lives
MLflow organizes work as experiments that contain runs . In Azure Machine Learning, a run is a job . The workspace is already an MLflow server . Each workspace has a tracking URI that starts with azureml:// . You do not stand up a separate tracking cluster.
How you get that URI depends on where the process runs:
Where the code runs Do you set the tracking URI? How a run starts
Command job on a compute cluster or serverless compute No — Azure Machine Learning injects it Automatically; do not call start_run()
Notebook or Jupyter on a compute instance No — already configured You call mlflow.start_run() (or a logging API that creates a run)
Local laptop, Azure Databricks, Azure Synapse Analytics Yes — mlflow.set_tracking_uri(...) You start the run
Get the URI with CLI v2 ( az ml workspace show --query mlflow_tracking_uri ), with SDK v2 ( ml_client.workspaces.get(name).mlflow_tracking_uri ), from the Azure portal Essentials blade, or by constructing azureml://{region}.api.azureml.ms/mlflow/v1.0/subscriptions/... . Private-link workspaces use a different URI shape — do not hand-build the public template; ask the SDK or CLI. You can also export MLFLOW_TRACKING_URI so every process on a shared cluster points at the same workspace.
Jobs submitted with CLI v2 set the experiment from YAML ( experiment_name: ). You do not have to call mlflow.set_experiment inside the training script. Interactive sessions should call mlflow.set_experiment("claims-fraud-2026") or export MLFLOW_EXPERIMENT_NAME . If you set nothing, runs land in an experiment named Default .
Interactive notebooks versus training jobs
On a compute-instance notebook the pattern is: set the experiment, start a run, log, end the run. A context manager is the cleanest form: mlflow.set_experiment("iris-classifier")
with mlflow.start_run(run_name="iris-rf-depth12") as run:
- mlflow.log_params({"n_estimators": 200, "max_depth": 12})
- mlflow.log_metric("accuracy", 0.97)
run_name becomes the display name in studio, which is how humans find the trial later. Technically a logging API creates a run if none is active, and mlflow.active_run() returns it — still prefer an explicit start_run so you control the name and the lifetime.
Inside a command job , the platform already opened the run. Call mlflow.autolog() and log_metric directly. Calling start_run() there creates a nested run, which is almost never what the exam (or a production parent job) wants.
Exam scenario
A fraud team trains LightGBM on a compute cluster with CLI v2. The YAML has experiment_name: fraud-lgbm . The script calls mlflow.autolog() and mlflow.log_metric("pr_auc", score) . Six child jobs later, studio Jobs shows one experiment with six jobs, each with parameters, a metrics chart, and an MLflow model folder. The MLOps engineer did not call start_run and did not set a tracking URI in the script — both were inherited from the workspace.
Common trap
Do not reach for a v1 Run.log helper, and do not assume mlflow.get_run(id).data.metrics["loss"] is a time series. That dictionary returns only the most recently logged value for each metric name. Use MlflowClient.get_metric_history() when you need the full curve. Another trap: logging a 4 GB model with the default 300-second artifact timeout ( AZUREML_ARTIFACTS_DEFAULT_TIMEOUT ). Raise it before log_model on large artifacts.
Autolog, explicit logs, and large models
mlflow.autolog() before training tells supported frameworks (scikit-learn, LightGBM, XGBoost, PyTorch, TensorFlow, and others) to log typical metrics, parameters, and a model. You can keep the convenience and skip the model with mlflow.autolog(log_models=False) when you want to call mlflow.sklearn.log_model (or another flavor) yourself after extra packaging. Some flavors disable automatic model logging when the trained model exceeds an internal size boundary. If autolog produced metrics but no model folder, that is expected — log the model explicitly.
Images go through log_image (numpy or PIL) or log_figure (matplotlib). Text, dictionaries, and existing files use log_text , log_dict , log_artifact , and log_artifacts . Prefer logging an MLflow model over a raw pickle: the model folder is what no-code deployment and the Responsible AI dashboard expect.
Asynchronous logging for high volume
Synchronous log_metric waits until the backend accepts the value. That is fine for a dozen scalars. It is a bottleneck when tens of nodes log hundreds of thousands of points. Enable async globally with mlflow.config.enable_async_logging() or export MLFLOW_ENABLE_ASYNC_LOGGING=True , or pass synchronous=False on a single call. Control returns as soon as the operation is accepted ; the value is not immediately readable. You can wait() on the returned operation if a later line must read that metric. Azure Machine Learning still guarantees metric order , and it waits for pending async metrics when the job is about to finish — completed jobs have a complete metric store.
Do not call log_metric in a tight loop if you can batch with MlflowClient.log_batch .
Authentication and authorization
The plugin authenticates with azure-identity . The chain is environment variables, then managed identity, Azure CLI, Azure PowerShell, then an interactive browser. Interactive browser blocks and is wrong for unattended jobs. For those, set a service principal : AZURE_TENANT_ID , AZURE_CLIENT_ID , and AZURE_CLIENT_SECRET (or a certificate path). Store secrets in Azure Key Vault, not in the training script. Built-in roles such as AzureML Data Scientist and Contributor already cover MLflow. A custom role needs Microsoft.MachineLearningServices/workspaces/experiments/* and .../jobs/* for tracking, plus .../models/*/* if you also use the MLflow model registry. First contact with the service is often set_experiment or start_run — turn on logging.getLogger("azure").setLevel(logging.DEBUG) when a prompt appears at a surprising time.
Studio Jobs UI versus the MLflow client
Both views read the same workspace store.
Studio: Jobs tab, filter by experiment, open a job, use Metrics to chart and share a layout, and Outputs and logs for files. user_logs/std_log.txt is stdout/stderr from your script (the first place you look). system_logs is platform diagnostics. Multi-node jobs add a folder per node IP.
MLflow client: mlflow.get_run(run_id) for the latest metrics, params, and tags; MlflowClient.list_artifacts / mlflow.artifacts.download_artifacts for files; search_runs to compare trials in Python.
Use studio when a human needs a chart. Use the client when a promotion script must pick the best pr_auc without clicking.