Study guide

8.1 Deploy Models to Managed Online Endpoints

Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling

2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM

2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces

3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries

3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries

4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML

4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces

5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps

5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning

6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison

6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs

7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management

7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving

8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback

8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback

9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts

9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts

10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC

10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI

11 Chapter 11: Deploying and Managing Foundation Models

11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads

12 Chapter 12: Prompt Design, Variants, and Git Version Control

12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git

13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows

13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows

14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging

14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging

15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation

15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing

16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production

16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models

8.1 Deploy Models to Managed Online Endpoints

Key Takeaways Explain This Topic AI

An Azure Machine Learning endpoint is a stable scoring URL plus authentication; a deployment is the model, environment, instance_type, and instance_count that actually score. Online endpoints can host several deployments and split traffic among them.

Managed online endpoints serve synchronous low-latency HTTP. They cannot scale to zero. Autoscale is Azure Monitor resource use (CPU, memory, requests, or a schedule). You pay for VMs while they run.

Production managed online endpoints should use auth_mode aad_token (Microsoft Entra). key never expires and is weakest; aml_token is short-lived. aad_token is managed-only — Kubernetes online endpoints cannot use it.

Custom models need a scoring script with init() and run(). MLflow and Triton support no-code. Do not confuse dedicated managed-online VMs with Domain 3 standard/serverless foundation-model deployments (quota-less, billed per token).

High availability wants instance_count of at least 3. Many SKUs reserve ceil(1.2 times instances) extra VM quota for upgrades. Endpoint names must be unique in the Azure region.

Last updated: August 2026

Deploy Models to Managed Online Endpoints

Quick Answer: A managed online endpoint is a durable HTTPS URL plus authentication. A deployment is the model, environment, instance_type , and instance_count that actually score. Clients call the endpoint; Azure Machine Learning routes traffic to one or more deployments. Online scoring is synchronous, low-latency HTTP . Instances cannot scale to zero — you pay while they run. For production on managed endpoints, set auth_mode: aad_token ( Microsoft Entra ). MLflow models can deploy with no scoring script ; custom models need init() and run() .

Exam AI-300 Domain 2 asks you to deploy models as real-time (managed online) endpoints . Chapter 7 registered and versioned models. This section is the first production surface those models hit: a live HTTPS service that applications call request by request.

Endpoint versus deployment

Keep these two objects separate. The exam loves mixing them.

An endpoint is the contract your consumers keep:

  • A stable scoring URI such as https://<endpoint-name>.<region>.inference.ml.azure.com/score
  • An authentication mechanism ( key , aml_token , or aad_token )
  • Optional traffic and mirror_traffic maps (section 8.3)
  • A Swagger document for the scoring contract

A deployment is the implementation behind that contract:

  • A registered model (or an inline path used for demos)
  • An environment (curated image plus Conda, or bring-your-own container)
  • Optional code_configuration.scoring_script
  • instance_type (the virtual machine SKU) and instance_count
  • Request settings, probes, and the identity used to pull storage and the container registry

One endpoint can host several deployments. Alice can run a CPU TensorFlow model as blue while Bob tests a GPU PyTorch model as green under the same URL. Consumers do not change their scoring URI when you swap implementations. The endpoint also has a routing mechanism that can send a percentage of requests to each deployment.

Endpoint names must be unique in the Azure region , not just in the workspace. Two workspaces in eastus cannot both own fraud-score .

SDK/CLI v2 is the exam surface: az ml online-endpoint / az ml online-deployment , or ManagedOnlineEndpoint and ManagedOnlineDeployment in azure-ai-ml . Do not author Azure Container Instances or Azure Kubernetes Service (AKS) v1 webservices as the current path. Kubernetes online endpoints still exist if you attach your own cluster; they do not get traffic mirroring or Microsoft Entra aad_token . Managed online endpoints provision the VMs, patch the host OS, recover nodes, and expose Azure Monitor metrics and per-deployment cost. You pay for those VMs and for networking; there is no extra Azure Machine Learning surcharge on the SKU.

When online is the right shape

Use a managed online endpoint when:

  • Latency must stay in the HTTP request/response (hundreds of milliseconds, not minutes)
  • The payload fits in the HTTP body
  • You need to scale the number of concurrent requests
  • The model answers in a short wall-clock time

Do not use online for overnight scoring of a data lake. That is a batch endpoint (section 8.2). Do not confuse this with standard (serverless) deployments of catalog foundation models. Those consume no VM quota from your subscription, bill per token (plus a small per-minute endpoint fraction), do not host multiple deployments, do not support traffic split or mirroring, and do not let you customize the inference stack. Domain 3 covers Microsoft Foundry serverless APIs and managed-compute foundation models. On AI-300, if the item is a classic registered scikit-learn or MLflow model with a scoring script, the answer is a managed online endpoint , not a Foundry serverless API.

  • Feature Standard / serverless (catalog) Managed online endpoint Batch endpoint
  • Invocation Synchronous HTTP Synchronous HTTP Starts an async job
  • Multiple deployments No Yes, traffic split Yes, default switch
  • Mirror traffic No Yes No
  • Auth Key Key, aml_token , or aad_token Microsoft Entra
  • Compute None (serverless) Dedicated VM instances Cluster instances
  • Scale to zero Built-in No Yes
  • Autoscale Built-in Resource use Job count
  • Cost basis Per token Running VM instances Nodes consumed by the job
  • Local testing No Yes (Docker) No
  • Authentication: key, aml_token, aad_token

Scoring is a data-plane operation. Create/update/delete of the endpoint is control plane and always uses a Microsoft Entra token against https://management.azure.com . Data-plane scoring uses one of three auth_mode values.

  • Mode Security Lifetime Endpoint types Scoring RBAC
  • key Lowest — static secrets Never expires; rotate with regenerateKeys Managed and Kubernetes Not required

aml_token Medium — service-issued Short-lived; refresh after refreshAfterTimeUtc Managed and Kubernetes Not required

aad_token Highest — identity and role Follows Entra policy (often 60–90 minutes) Managed only Yes: onlineEndpoints/score/action

Microsoft's current guidance: for production workloads on managed online endpoints, use aad_token . Keys are fine for development. aad_token on a Kubernetes online endpoint is a trap — it is not supported.

Callers send Authorization: Bearer <key-or-token> . az ml online-endpoint get-credentials returns primaryKey / secondaryKey for key mode, or accessToken plus expiry for token modes. Data-plane Entra tokens must be issued for audience https://ml.azure.com , not management.azure.com . Mixing those audiences yields HTTP 401. Built-in AzureML Data Scientist includes both control-plane CRUD and score/action . A caller with only Contributor on a different resource cannot score an aad_token endpoint.

Scoring script, no-code, and bring-your-own container

Custom models need a Python scoring file with:

  • init() — runs once when the container starts; load the model from the AZUREML_MODEL_DIR environment variable
  • run(raw_data) — runs on every invoke; parse the HTTP body and return JSON-serializable output

No-code deployments skip that file for MLflow and Triton models: Azure Machine Learning autogenerates the inference server. Automated machine learning (AutoML) models also ship a scoring script you can deploy as-is. Low-code means you bring the script plus a curated image. Include the azureml-inference-server-http package in a custom Conda file or the container will not serve. Bring Your Own Container (BYOC) puts the whole stack in a custom image and still rides managed autoscale, logs, and rollout.

If you later delete the registered model or the environment image a deployment still references, reimaging during a host OS patch can fail . Update the deployment before you delete those assets. Microsoft patches base images for known vulnerabilities; you must redeploy to pick up a patched image. If you brought your own image, you own the patch cycle.

Compute, quota, autoscale, and cost

You choose instance_type from the managed online SKU list (CPU or GPU) and instance_count . For high availability Microsoft recommends at least three instances . For many SKUs the service reserves 20 percent extra VM quota for upgrades and recovery: requesting 10 Standard_DS3_v2 (4 cores) needs quota for ceil(1.2 * 10) * 4 = 48 cores. Some SKUs are exempt; check the SKU list. That reserved capacity is not billed unless the upgrade actually runs.

Autoscale is Azure Monitor autoscale on resource use — CPU, memory, request rate — or a schedule such as peak business hours. It is not the batch job-count scaler. Managed online deployments cannot scale to zero. Idle instances still bill. That is the number-one cost surprise versus batch.

A shared quota pool exists only for short testing of selected catalog models (Llama, Phi, Mistral, and similar) from studio, and only on Enterprise Agreement subscriptions. Do not plan production capacity on shared quota.

A workspace managed virtual network can isolate inbound scoring (workspace private endpoint) from outbound pulls (managed VNet private endpoints). Expect extra private-link and fully qualified domain name outbound charges. Customer-managed keys are supported on online and batch, not on standard serverless deployments.

  • YAML shape (CLI v2)
  • A minimal endpoint: $schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
  • name: fraud-online
  • auth_mode: aad_token
  • A minimal deployment: $schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
  • name: blue
  • endpoint_name: fraud-online
  • model: azureml:fraud-xgb:3

code_configuration:

  • code: onlinescoring
  • scoring_script: score.py
  • environment: azureml:fraud-env:2
  • instance_type: Standard_DS3_v2
  • instance_count: 3

Production practice: register the model and environment, then reference azureml:<name>:<version> . Inline path: uploads work for demos and local Docker tests. Local endpoints ( --local / local=True ) need Docker Engine, support one deployment, and do not implement traffic rules or auth. Studio and Azure Resource Manager templates cannot deploy locally.

Create the endpoint, then the deployment. --all-traffic on az ml online-deployment create is a development convenience that points 100 percent of traffic at the new deployment; production rollouts set traffic explicitly (section 8.3).

Exam scenario

A payments team registered fraud-xgb:3 as an MLflow model. The API gateway needs p99 latency under 200 ms and will send a small JSON body. The security baseline forbids long-lived keys in production. Deploy a managed online endpoint with auth_mode: aad_token , a CPU or GPU SKU that meets latency, instance_count of at least 3, and either no-code MLflow or a reviewed score.py . Do not pick a batch endpoint, and do not pick a Foundry serverless deployment unless the item is explicitly a catalog foundation model.

Common trap

Candidates mix online autoscale with batch scale-to-zero , or they pick key for a production managed endpoint because tokens expire. Tokens are supposed to expire. Another trap: treating standard/serverless foundation-model deployments as the way to host every registered workspace model. Serverless is quota-less and per-token, but only for catalog models that support it — Domain 3, not this Domain 2 bullet. A third trap: putting aad_token on a Kubernetes online endpoint.

Loading diagram... Online endpoint with two deployments and a traffic split