5.2 Use Automated Machine Learning to Explore Optimal Models

Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling

2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM

2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces

3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries

3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries

4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML

4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces

5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps

5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning

6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison

6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs

7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management

7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving

8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback

8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback

9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts

9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts

10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC

10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI

11 Chapter 11: Deploying and Managing Foundation Models

11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads

12 Chapter 12: Prompt Design, Variants, and Git Version Control

12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git

13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows

13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows

14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging

14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging

15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation

15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing

16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production

16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models

5.2 Use Automated Machine Learning to Explore Optimal Models

Key Takeaways Explain This Topic AI

AutoML tasks include classification, regression, forecasting, computer vision (multi-class, multi-label, object detection, instance segmentation), and NLP (text classification, named entity recognition).

Featurization — scaling, missing-value handling, text-to-numeric — is baked into the trained model and is applied automatically at inference.

Voting and stacking ensembles are on by default; stacking uses LogisticRegression (classification) or ElasticNet (regression/forecasting) as the meta-model.

You stop a job with limits (timeout_minutes, trial_timeout_minutes, max_trials, enable_early_termination) and a primary_metric the task type allows.

Studio is the no-code path; SDK/CLI v2 (type: automl, often MLTable data) is the code path. AutoML is a strong baseline, not a guarantee it beats a well-tuned custom model.

Last updated: August 2026

Use Automated Machine Learning to Explore Optimal Models

Quick Answer: Automated machine learning (AutoML) tries many algorithms and parameters against a primary metric you choose. Tasks cover classification , regression , forecasting , computer vision , and NLP . Featurization becomes part of the model. Voting and stacking ensembles are on by default. Stop with exit criteria . Use studio (no-code) or SDK/CLI v2 . AutoML is a search, not a promise it beats a well-tuned custom model.

Domain 2 of Exam AI-300 asks you to use automated machine learning to explore optimal models . The exam word is explore . AutoML is how an MLOps engineer gets a high-quality baseline — and often a production candidate — without personally iterating every estimator. It does not retire custom training scripts or sweep jobs (section 5.4).

How AutoML works

You identify the problem type, choose studio or SDK/CLI v2, point at labeled data, set the primary metric and limits, and submit a job. Azure Machine Learning then launches many child trials in parallel . Each trial pairs an algorithm with feature handling and hyperparameters. Each trial gets a score on your metric. The job stops when it hits exit criteria you defined (or when it stops making progress if you defined none).

The training job produces a serialized model (commonly a Python .pkl ) that includes both the estimator and the preprocessing . Logged metrics for every child appear on the parent job. You inspect them in studio the same way you inspect any experiment: Jobs, then the AutoML parent, then child runs.

A minimal CLI v2 job looks like this: $schema: https://azuremlsdk2.blob.core.windows.net/preview/0.0.1/autoMLJob.schema.json

• type: automl
• task: classification

training_data:

• path: ./train_data
• type: mltable
• compute: azureml:cpu-cluster
• primary_metric: AUC_weighted
• target_column_name: is_fraud

limits:

• timeout_minutes: 180
• max_trials: 40
• enable_early_termination: true

Submit with az ml job create --file automl-classification-job.yml . SDK v2 uses factory helpers such as automl.classification(...) plus set_limits and set_training , then ml_client.jobs.create_or_update .

Task types you must name on the exam

Pick the task from the business problem, not from the library you like.

Classification — categorical label. Fraud, churn, image-or-text class. Featurizers include deep neural network text featurizers for tabular-plus-text problems.

Regression — numeric target. Price, delay minutes, risk score.

Forecasting — time series treated as multivariate regression : lagged values are pivoted into features along with extra predictors. Advanced knobs include holiday detection, Auto-ARIMA, Prophet, ForecastTCN, many-models grouping, rolling-origin cross validation, lags, and rolling-window aggregates.

Computer vision — authored from the Python SDK (studio still shows the resulting jobs). Tasks are multi-class image classification (one label per image), multi-label image classification (several labels per image), object detection (bounding boxes), and instance segmentation (pixel-level polygons). It integrates with Azure Machine Learning data labeling.

Natural language processing (NLP) — text classification and named entity recognition (NER) . End-to-end deep networks on pretrained BERT models, 104-language multilingual support, distributed training with Horovod, and the same labeling integration.

Tabular AutoML v2 wants MLTable training data (a folder with an MLTable file plus the CSV or Parquet). Vision and NLP jobs use the labeled datasets those task guides describe. You are not required to memorize every supported estimator; you are required to know that you do not pick the algorithm unless you allow-list or block-list it ( allowed_training_algorithms / blocked_training_algorithms ).

Featurization is part of the model

Featurization is scaling, normalization, missing-value handling, and converting text to numbers. AutoML does this automatically ( mode: auto ), you can turn it off , or you can supply custom transformers. Whatever ran during training is serialized into the model . At scoring time the same steps run on new rows. That is why a raw CSV scored against an AutoML endpoint does not need a separate preprocessing service — and why you must not “simplify” the pickle by stripping preprocessors.

Validation is also automatic if you omit validation_data :

Training set larger than 20,000 rows — 10% holdout for validation.

1,000–20,000 rows — 3-fold cross-validation.

Fewer than 1,000 rows — 10-fold cross-validation.

You can pass an explicit validation set. A separate test dataset to evaluate the final recommended model is a preview feature — teach it as optional and experimental, not as a GA requirement.

Ensembles, metrics, and exit criteria

Ensemble models are enabled by default and usually appear as the last iterations. Two methods:

• Method How it combines models Default meta-model
• Voting Weighted average of class probabilities (classification) or of regression targets Not applicable

Stacking Heterogeneous models feed a second-level learner LogisticRegression (classification); ElasticNet (regression/forecasting)

Selection follows the Caruana ensemble selection algorithm with sorted initialization: start with up to five best individual models that sit within a 5% band of the best score, then add a model only when it improves the ensemble. You can disable vote or stack ensembles in set_training when you need a single interpretable model.

The primary metric must be one the task allows. Classification often uses accuracy , AUC_weighted , average_precision_score_weighted , norm_macro_recall , or precision_score_weighted . On small or highly skewed labels, thresholded metrics such as accuracy can be a poor objective — AUC_weighted is the usual safer choice for fraud. Regression and forecasting lean on r2_score , normalized_root_mean_squared_error , and normalized_mean_absolute_error . NLP NER currently supports Accuracy as the primary metric. The name you put in YAML must match the metric the service actually logs.

Exit criteria live under limits :

timeout_minutes — whole job; default is six days (8,640 minutes) . Ensembling and explainability after the last trial are not counted in that timeout.

trial_timeout_minutes — each child; default 43,200 minutes (one month) if unset.

max_trials — default 1,000 .

max_concurrent_trials — match this to nodes on the cluster so children actually run in parallel; default is one concurrent child.

enable_early_termination — stop if the short-term score is not improving.

SDK/CLI AutoML jobs run on a compute cluster or compute instance , not on your laptop kernel. Studio’s automated ML wizard is the no-code equivalent for tabular problems; vision and NLP authoring is SDK-first with studio for monitoring.

Studio versus SDK, and what AutoML is not

Use studio when an analyst should launch a classification run without Python. Use SDK/CLI v2 when the AutoML job is a pipeline step, when you need vision/NLP, or when you must pin blocked_training_algorithms , custom featurization, ONNX-compatible models, or distributed LightGBM / TCNForecaster on large data. You can convert many tabular winners to ONNX for C# / ML.NET scoring without a REST hop.

AutoML is not magic. Algorithms have inherent randomness, so two identical configs can differ slightly. A specialist with a well-designed feature set and a tuned LightGBM can beat AutoML on that problem. AutoML’s job in MLOps is to search the space quickly, apply sound defaults, and give you a model whose featurization is already packaged . You still register, evaluate with responsible AI, and compare against the custom candidate (later chapters).

Exam scenario

A bank wants a first fraud model on a 2-million-row tabular set with a 0.8% positive rate. The MLOps engineer creates an MLTable, submits type: automl with task: classification , primary_metric: AUC_weighted (not accuracy), enable_early_termination: true , and max_concurrent_trials equal to the cluster’s node count. Studio shows child trials plus a final voting ensemble. The engineer registers the MLflow model and, in parallel, keeps a custom XGBoost job as a challenger — they do not assume AutoML is automatically production.

Common trap

Do not say “AutoML always wins.” Do not pick accuracy as the primary metric on a brutal class imbalance just because it is the default in a tutorial. Do not ship the estimator and drop the featurizers — missing-value imputation and scaling are the model . Do not point tabular v2 AutoML at a raw uri_file and expect it to behave like MLTable. Do not confuse AutoML (the service searches algorithms ) with a sweep job (you bring one training script and search its hyperparameters).