Study guide
5.4 Automate Hyperparameter Tuning
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
5.4 Automate Hyperparameter Tuning
Key Takeaways Explain This Topic AI
A sweep job (type: sweep) runs many child trials of your training script over a search_space; sampling is random, grid, or Bayesian.
Grid sampling requires discrete choice parameters. Random supports discrete and continuous (optional Sobol). Bayesian supports choice, uniform, and quniform and works best with enough trials (about 20 per hyperparameter) and modest concurrency.
Early termination policies are Bandit (slack_factor or slack_amount), median stopping, and truncation selection; omit a policy to let every trial finish.
The objective is primary_metric plus goal Maximize or Minimize; the name must match mlflow.log_metric in the script exactly.
Sweep searches hyperparameters of a script you provide. AutoML searches algorithms (and their parameters) for you. They are not interchangeable exam answers.
Last updated: August 2026
Automate Hyperparameter Tuning
Quick Answer: A sweep job ( type: sweep ) launches child jobs of your training script across a search space . Sample with random , grid , or Bayesian . Stop losers with Bandit , median stopping , or truncation selection . Optimize a primary metric ( Maximize or Minimize ) that the script logs with MLflow under the exact same name . Sweep ≠ AutoML: you bring the script; AutoML brings the algorithms.
Domain 2 asks you to automate hyperparameter tuning . In SDK/CLI v2 that product is the SweepJob , not the v1 HyperDrive object name. You start from a command job (section 5.3), replace some inputs with parameter expressions , and call .sweep() — or you author YAML with type: sweep .
Search space: discrete versus continuous
Hyperparameters are the knobs the optimizer does not learn as weights: learning rate, tree depth, batch size, number of layers. You declare a search space of expressions:
Discrete / choice — a list, a range, or comma-separated values ( batch_size: [16, 32, 64] ).
Quantized discrete — QUniform , QLogUniform , QNormal , QLogNormal (round a draw to a step q ).
Continuous — Uniform / LogUniform / Normal / LogNormal over a real interval.
CLI YAML: search_space:
learning_rate:
- type: uniform
- min_value: 0.01
- max_value: 0.2
boosting:
- type: choice
- values: [gbdt, dart]
SDK v2 uses Uniform , Choice , and friends from azure.ai.ml.sweep , then command_job(learning_rate=Uniform(...), boosting=Choice(...)).sweep(...) .
Every trial restarts training from scratch , including data loaders. Push heavy featurization into a pipeline step or a data asset so each child is not recopying a lake.
Sampling algorithms
Sampler What it supports Early termination When to use
Random Discrete and continuous Yes First pass; optional Sobol ( RandomSamplingAlgorithm(seed=..., rule="sobol") ) for repeatable space-filling
Grid choice only Yes Small discrete grids you can afford to exhaust
Bayesian choice , uniform , quniform Uses prior results When you have budget; Microsoft recommends at least about 20 trials per hyperparameter
Grid on a uniform learning rate is an exam fail: grid cannot enumerate an interval. Bayesian with a loguniform (or other unsupported distribution) is the other fail. Bayesian also converges better with fewer concurrent trials , because new samples need finished predecessors. Random is the safe default when the space is mixed or you want aggressive early stopping.
Objective, logging, and the best child
The sweep’s objective is:
primary_metric — a string that must equal the metric name in mlflow.log_metric inside the trial script.
goal — Maximize or Minimize .
If the script logs val_auc and the sweep asks for AUC_weighted , the service never sees a usable score and early termination / best-child selection is nonsense. Log often enough for policies to have intervals (each log_metric of that name is one interval). After the parent finishes, download the best trial outputs ( ml_client.jobs.download(sweep_name, output_name="model") or az ml job download --name <sweep> --output-name model ). Studio charts — metrics over time, parallel coordinates, 2D/3D scatter — are how you explain why that child won.
Early termination and limits
Policies compare children on the primary metric and cancel losers:
Bandit — slack around the current best ( slack_factor ratio or slack_amount absolute). Example: maximize, best is 0.8, slack_factor 0.2 → children below about 0.67 can die.
Median stopping — stop if the trial’s best is worse than the median of running averages across jobs.
Truncation selection — at each evaluation, cancel the worst truncation_percentage (1–99). Optional exclude_finished_jobs .
None (default) — every trial runs to completion.
Shared knobs: evaluation_interval (apply every N metric reports; docs default 0 if you omit it, so set it on purpose) and delay_evaluation (give every configuration a minimum number of intervals). A conservative starter that Microsoft cites as roughly 25–35% savings without hurting the primary metric is median stopping with evaluation_interval=1 and delay_evaluation=5 . Bandit with tight slack, or truncation with a large percentage, is more aggressive.
Resource limits on the parent:
max_total_trials — 1 to 1000 .
max_concurrent_trials — 1 to 1000; cannot exceed what the compute actually has .
timeout — seconds for the entire sweep .
trial_timeout — seconds for one child.
If both max_total_trials and timeout are set, the sweep ends on whichever hits first .
Sweep versus AutoML
Keep this contrast memorized:
AutoML — you choose a task and data. The service searches algorithms (and hyperparameters, and featurization, and ensembles).
Sweep — you already chose the algorithm (your python train.py ). The service searches that script’s hyperparameters .
A LightGBM sweep is the right answer when the team standardized on LightGBM. An AutoML classification job is the right answer when you still need to know whether LightGBM, XGBoost, or a stack ensemble should win. You can even sweep inside a pipeline by calling .sweep() on a command component ; that is still “you brought the script.”
- YAML sketch: $schema: https://azuremlschemas.azureedge.net/latest/sweepJob.schema.json
- type: sweep
trial: ...
search_space:
learning_rate:
- type: uniform
- min_value: 0.01
- max_value: 0.2
- sampling_algorithm: random
objective:
- goal: minimize
- primary_metric: test-multi_logloss
limits:
- max_total_trials: 20
- max_concurrent_trials: 4
- timeout: 7200
early_termination:
- type: median_stopping
- evaluation_interval: 1
- delay_evaluation: 5
- compute: azureml:cpu-cluster
SDK equivalent: sweep_job = command_job_for_sweep.sweep(compute="cpu-cluster", sampling_algorithm="random", primary_metric="test-multi_logloss", goal="Minimize") , then set_limits(...) and early_termination = MedianStoppingPolicy(...) .
Exam scenario
A claims model will ship as LightGBM. The engineer wraps src/train.py (already logging mlflow.log_metric("val_logloss", ...) ) as a command job, replaces learning_rate with Uniform(0.01, 0.2) and num_leaves with Choice([31, 63, 127]) , and sweeps with Bayesian sampling, goal="Minimize" , primary_metric="val_logloss" , max_total_trials=40 , max_concurrent_trials=4 on an 8-node cluster (leaving headroom), and median stopping delayed 5 evaluations. They download the best child’s MLflow model. They do not open AutoML for this run because the algorithm is already chosen.
Common trap
The classic failure is a name mismatch between primary_metric and log_metric . Next is using grid on continuous parameters, or Bayesian on unsupported distributions. Next is setting max_concurrent_trials larger than cluster nodes and wondering why Bayesian looks random. Next is treating a sweep as AutoML (or vice versa). Last is logging the metric only once at the end — Bandit and median stopping never get an interval to act on, so you pay for every doomed epoch.