Study guide
2.3 Create and Manage Compute Targets
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
2.3 Create and Manage Compute Targets
Key Takeaways Explain This Topic AI
A compute instance is a single-owner, single-node cloud workstation with a 120 GB OS disk; enable idle shutdown, but a stopped instance still bills disk, public IP, and the standard load balancer.
A compute cluster is managed, multi-node, and autoscaling; set min_instances to 0 (or use serverless compute) so you are not charged for idle VMs.
Serverless compute is the default training target when you omit the compute name; Azure Machine Learning creates, scales, and deletes the nodes, and jobs consume the same Azure Machine Learning compute quota.
Managed compute is compute instance, compute cluster, and serverless; unmanaged compute (Kubernetes, remote VMs, Databricks, HDInsight, and similar) is created outside Azure Machine Learning and then attached.
Training and inference use different targets: clusters, instances, serverless, and attached Kubernetes for training; managed online/batch endpoints or Kubernetes for production inference.
Last updated: August 2026
Create and Manage Compute Targets
Quick Answer: Instance = personal workstation (one owner, 120 GB OS disk, idle shutdown; stopped still bills disk/IP/load balancer). Cluster = named autoscaling nodes (min 0). Serverless = omit the compute name and let Azure Machine Learning create capacity. Managed means instance, cluster, and serverless; Kubernetes and remote VMs are unmanaged attaches.
A compute target is the machine, or pool of machines, that runs a training script or hosts a deployment. The point of the abstraction is that you can change where the job runs without rewriting the training code. Attach compute to a workspace ; everyone in that workspace can submit jobs to shared targets. A compute instance is the exception to “shared hardware”: it has one owner , even though notebook files on the workspace file share are visible to other instances.
AI-300 Domain 1 asks you to create and manage these targets, not to memorize every retired NV-series SKU. Know the managed versus unmanaged split, when to use instance versus cluster versus serverless, how billing behaves when something looks “off,” and which targets are valid for training versus inference.
Managed versus unmanaged
Managed compute is created and patched by Azure Machine Learning. Only three kinds qualify:
- Compute instance
- Compute cluster ( AmlCompute )
- Serverless compute
You create instances and clusters from studio, SDK v2, CLI v2 ( az ml compute create ), or ARM/Bicep. You do not create serverless compute as a persistent resource; you use it by omitting the compute name on a job (or setting pipeline default_compute to serverless / azureml:serverless ).
Unmanaged compute is created outside Azure Machine Learning and then attached : remote virtual machines, Azure HDInsight, Azure Databricks, Azure Data Lake Analytics, and Kubernetes . You own patching, scaling, and often the identity story. Attach Kubernetes when inference or training must land on an existing on-premises, cloud, or edge cluster.
Apache Spark pools can appear as a training target (preview, with limits for AutoML). Do not invent Spark as a replacement for a compute cluster on a standard tabular command job unless the stem is about Spark.
Training targets versus inference targets
Training (SDK/CLI v2) commonly uses:
- Training target AutoML Pipelines Designer Notes
- Compute cluster Yes Yes Yes Named, autoscaling, reusable
- Serverless compute Yes Yes Yes Default when compute is omitted
Compute instance Yes (SDK) Yes Yes Single node; must enable root access at create if you will submit training jobs to it
Kubernetes (attached) — Yes Yes Unmanaged
Remote VM, Databricks, HDInsight, Batch, Data Lake Analytics Partial Often yes Often no Unmanaged; scenario-specific
Inference (production) is a shorter list:
Managed online endpoints and batch endpoints — Azure Machine Learning hosts the container on managed (serverless) compute. GPU is supported. This is the default production answer for Domain 2.
Azure Machine Learning Kubernetes — real-time or batch inference on your cluster.
A compute instance can host local test inference for debug. It is not the production online-endpoint answer. When you size inference VMs, Microsoft’s rule of thumb is scale up first (about 150 percent of the RAM the model needs), then scale out for concurrency.
Hub and project workspaces do not support compute clusters . On a hub, training jobs that would have used a cluster should use serverless compute . That limitation is a favorite distractor if the stem already established a Foundry/hub layout in section 2.1.
Compute instance: the personal workstation
A compute instance is a fully managed Ubuntu workstation with Jupyter, JupyterLab, VS Code (including VS Code for the Web), the Azure CLI, Docker, CUDA stacks on GPU SKUs, and the Azure Machine Learning Python SDK v2 environment. Administrators can create an instance on behalf of a user (disable SSO in that flow) and can push a setup script for packages and mounts.
Facts the exam likes:
One owner. Other people in the workspace do not share the VM as a multi-user login. They can still open notebooks from the workspace file share , because that share is mounted as the default working directory on every instance.
120 GB OS disk (P10). You cannot change the OS disk type. Customer-managed key encryption is not currently supported for that OS disk. Temporary data can go to /mnt (size depends on the VM SKU) or /tmp . Do not fill / with training data. If the disk fills, idle shutdown and studio stop/restart can fail ; clear at least several GB from the terminal ( df -h ), then reboot. Do not sudo shutdown as your stop mechanism—that does not deallocate through Azure Machine Learning.
Idle shutdown and schedules. Enable idle shutdown so an abandoned Jupyter session does not burn GPU hours. Idle means no Jupyter kernel, no Jupyter terminal, no Azure Machine Learning run, no VS Code connection, and no custom app. Stopping the VM stops compute-hour billing. You still pay for the disk , the public IP , and the standard load balancer . Every instance and cluster gets a load balancer, and that balancer exists even while the compute is stopped.
Quota. Dedicated cores for instances and clusters are a shared regional quota. Stopping an instance does not release quota, so you can restart it. If quota is exhausted, you cannot create another instance in that VM family until you delete something or request more quota ( az ml compute list-sizes shows what the region will offer you).
Hub reuse. On a hub, a compute instance can be reused across the hub’s project workspaces. That is hub-specific sharing of a still-single-owner workstation, not a multi-user cluster.
If you will submit training jobs to the instance (not just run cells), enable root access at create time . You cannot retrofit that later.
Compute cluster: named, autoscaling, reusable
A cluster is the classic “cpu-cluster” or “gpu-cluster” you pass as compute="cpu-cluster" in SDK v2. It can be one node or many. It autoscales when jobs arrive. Set:
min_instances: 0 so the cluster scales to zero between jobs. A min of 1 is a warm pool you pay for 24/7.
max_instances to cap spend and to stay inside quota.
idle_time_before_scale_down (seconds) so a burst of jobs can reuse nodes without waiting for a cold scale-up every time.
- CLI v2 YAML sketch: $schema: https://azuremlschemas.azureedge.net/latest/amlCompute.schema.json
- name: cpu-cluster
- type: amlcompute
- size: STANDARD_DS3_v2
- min_instances: 0
- max_instances: 4
- idle_time_before_scale_down: 120
identity:
- type: system_assigned
- az ml compute create --file cpu-cluster.yml
Clusters support CPU and GPU SKUs. GPU jobs need a matching CUDA version in the environment (Hopper H100/H200 families need CUDA 12.0+, Ampere A100 needs 11.0+, and so on). Isolated SKUs such as Standard_M128ms exist for compliance workloads that must be the only tenant on the host.
Identity on a cluster is either one system-assigned identity or one or more user-assigned identities, not both at once. The default identity mounts datastores and pulls ACR images. Code inside the job can use DEFAULT_IDENTITY_CLIENT_ID with ManagedIdentityCredential to reach Azure Storage without an account key.
Serverless compute: omit the name
If you leave compute off a command, sweep, AutoML, or parallel job, the job runs on serverless compute . Azure Machine Learning creates, scales, patches, and deletes the nodes. You can still set resources.instance_type and resources.instance_count , and queue_settings.job_tier of Standard (dedicated) or Spot (low priority). If you specify nothing, the service picks a CPU VM size from quota, cost, performance, and disk size.
Why serverless shows up as the “reduce management overhead” answer:
- No cluster objects to replicate in every workspace for every VM size
- No waiting for a named cluster to scale down before your job can scale up on those nodes
Quota is still required (workspace and subscription). Usage appears as Serverless in the Azure portal quota blade. Insufficient family quota fails the job with a message to shrink instance_count or request quota.
Billing matches Azure Machine Learning compute. Identities: user passthrough or workspace user-assigned managed identity. No system-assigned identity on the serverless job.
Pipeline jobs set settings.default_compute: azureml:serverless (CLI) or pipeline_job.settings.default_compute = "serverless" (SDK). Studio job submit and Designer also have a Serverless choice.
You still need Contributor or AzureML Data Scientist on the workspace to submit. Serverless does not bypass RBAC.
Subresources you pay for even when “nothing is running”
Creating instances and clusters also creates subresources : the VMs, a load balancer per instance or cluster, virtual network plumbing, and bandwidth charges for cross-region egress. That is why a “I stopped everything” invoice still shows disk, IP, and load balancer for instances, and why deleting unused compute (not merely stopping it) is a cost-control action.
Retired VM series (NC, NCv2, ND, NV from 2023; Av1 and HB from 2024; NCv3 from 30 September 2025) should be recreated on a supported size. If a GPU SKU is missing from the picker, it is often a quota problem, not an unsupported series.
Exam scenario
A team trains nightly on GPUs, experiments in notebooks during the day, and deploys a real-time classifier. Give each scientist a compute instance with idle shutdown for notebooks. Run the nightly training as a command job with compute omitted (serverless) or on a cluster with min_instances: 0 . Deploy to a managed online endpoint , not to the compute instance. If the workspace is a hub project, do not look for a cluster create button; use serverless.
Common trap
Setting cluster min_instances to 1 “so jobs start faster,” then being surprised by 24/7 VM charges. The matching instance trap is enabling idle shutdown and assuming the invoice goes to zero: disk, public IP, and load balancer still bill . A third trap is filling the 120 GB OS disk until the instance cannot shut down automatically.