8.2 Deploy Models to Batch Endpoints

Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling

2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM

2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces

3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries

3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries

4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML

4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces

5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps

5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning

6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison

6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs

7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management

7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving

8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback

8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback

9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts

9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts

10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC

10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI

11 Chapter 11: Deploying and Managing Foundation Models

11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads

12 Chapter 12: Prompt Design, Variants, and Git Version Control

12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git

13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows

13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows

14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging

14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging

15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation

15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing

16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production

16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models

8.2 Deploy Models to Batch Endpoints

Key Takeaways Explain This Topic AI

A batch endpoint starts an asynchronous scoring job over files in storage or data assets; results land in blob. Routing is a default deployment switch, not a traffic split, and there is no mirror traffic.

Compute clusters can scale to zero (min_instances: 0). Autoscale is based on job count. Azure Machine Learning does not bill for idle batch endpoints; cost is the cluster nodes consumed while the job runs.

Batch can deploy a model or a pipeline component. MLflow models skip the scoring script. Custom scoring uses init() plus run(mini_batch) over a list of file paths, not a single HTTP body.

Parallelism is file-level via mini_batch_size and max_concurrency_per_instance. Per-job overrides can change instance count, mini-batch size, retries, timeout, and error threshold without editing the deployment.

Authentication is Microsoft Entra ID. Jobs run as the invoker. Low-priority clusters (allocated as Spot VMs after 31 March 2026) can cut cost but evict nodes; reschedule is at mini-batch granularity, and a fully preempted cluster cancels the job.

Last updated: August 2026

Deploy Models to Batch Endpoints

Quick Answer: A batch endpoint is a durable HTTPS URL that starts an asynchronous scoring job , not a real-time HTTP response. Inputs are files in storage or data assets ; outputs land in blob . Routing is a default deployment switch , not a traffic split. Clusters can scale to zero . Auth is Microsoft Entra . You can deploy a model or a pipeline component . Parallelism is mini-batches of files .

Exam AI-300 Domain 2 splits batch from online because the product is different: jobs, files, and clusters versus always-on HTTP servers. The registered model from Chapter 7 can serve both; the operational contract is not the same.

Endpoint, default deployment, and invoke

The batch endpoint still gives you a stable URL and authentication. The deployment still holds the model (or pipeline), environment, and compute. The routing difference is the exam discriminator:

Online: traffic percentages that must sum to 100 (or 0 to disable)

Batch: one default deployment . invoke without a name hits the default. Pass --deployment-name (CLI) or deployment_name (SDK) to test a non-default deployment without flipping production.

Create a second Keras deployment, invoke it by name, confirm outputs, then az ml batch-endpoint update --set defaults.deployment_name=keras-dpl . Consumers keep the same URL. There is no traffic: map and no mirror traffic on batch endpoints.

Azure Machine Learning does not bill you for the endpoint or deployment objects . Organize as many as you need. Endpoints and deployments can share a cluster or use independent clusters. Endpoint names, like online names, must be unique in the Azure region because the name is in the URI.

SDK/CLI v2 types are BatchEndpoint plus ModelBatchDeployment or PipelineComponentBatchDeployment . Studio can create model deployments; Kubernetes batch targets need CLI or SDK.

When batch is the right shape

Use batch when:

Inference is expensive or long-running
Data is many files in a storage account or a registered data asset
You do not need HTTP latency
You want parallelization across cluster nodes
You want to operationalize an entire pipeline graph , not only a single model

A nightly fraud-file drop of two million CSV rows is batch. A checkout-page risk call is online. Overcapacity is queued on batch (jobs wait for cluster nodes) versus throttled on online (HTTP 429).

Model deployments versus pipeline component deployments

Model deployment scores a registered model at scale. Required pieces:

Model
Compute cluster ( azureml:<cluster-name> )
Scoring script (optional for MLflow )
Environment (optional for MLflow)

Reuse the same registered model you deployed online; batch runs it over files. If the model needs light pre- or post-processing, put it in the batch scoring script rather than standing up a second service.

Pipeline component deployment hosts a whole directed graph: preprocess, retrieve features, score, post-process. Specify the pipeline component plus cluster configuration. You can also promote an existing pipeline job into a component, but Microsoft calls explicit component authoring the Machine Learning Operations (MLOps) best practice so the graph is versioned independently of one ad-hoc job.

Curated environments are not supported for batch; author a custom environment. Include azureml-core and azureml-dataset-runtime[fuse] plus whatever the model needs. AutoML's generated scoring script is for online endpoints. Reusing it on batch is a documented failure mode — write a batch driver.

Scoring script: files, not HTTP bodies

Online run(raw_data) receives the HTTP payload. Batch run(mini_batch) receives a list of file paths . init() still loads the model once from AZUREML_MODEL_DIR . Return a pandas DataFrame or an array; each row corresponds to a successful file in that mini-batch.

Parallelism is file-level . A folder of 100 files with mini_batch_size: 10 yields 10 mini-batches, regardless of file size. Huge files need splitting; the runtime does not rebalance skewed file sizes. Tune these knobs together:

Setting Role
resources.instance_count Nodes requested for each scoring job
max_concurrency_per_instance Parallel run() workers per node
mini_batch_size Files per run() call
output_action append_row merges into output_file_name ; summary_only skips the merge
retry_settings.max_retries / timeout Per mini-batch
error_threshold File failures allowed before the job dies ( -1 means unlimited)
logging_level warning , info , or debug

Override instance count, mini-batch size, retries, timeout, and error threshold per job ( --mini-batch-size , --instance-count , or SDK params_override ) without editing the deployment. That is how you give a 1-million-file run more nodes than a 10,000-file smoke test.

Outputs must go to a blob datastore . The default is the workspace blob store under a job GUID. You can set --output-path to azureml://datastores/<name>/paths/<folder>/ and a new output_file_name . Duplicate output paths fail the job — pick a unique file name.

Inputs can be Azure Machine Learning datastores, data assets, or storage URIs ( uri_folder , uri_file ). Model deployments take one data input; pipeline deployments can take a dictionary of named inputs.

Compute, scale-to-zero, autoscale, and cost

Batch runs on Azure Machine Learning compute clusters ( AmlCompute ) or Kubernetes. Create the cluster with min_instances: 0 and a max_instances cap. Nodes provision when the job starts and deallocate when it finishes . Queued jobs consume no compute. idle_time_before_scale_down still applies if you leave min_instances above zero; production batch clusters almost always start at zero.

Autoscale is based on job count , not CPU. That is the opposite of online. Cost equals nodes actually running during the job, capped by the cluster maximum. Idle min_instances: 0 clusters cost nothing. Creating the endpoint itself does not start VMs.

A typical cluster YAML: $schema: https://azuremlschemas.azureedge.net/latest/amlCompute.schema.json
name: batch-cluster
type: amlcompute
size: STANDARD_DS3_v2
min_instances: 0
max_instances: 5
idle_time_before_scale_down: 120

Then the deployment points at compute: azureml:batch-cluster and sets resources.instance_count for each job's node request.

Low-priority and Spot compute

Batch (unlike online) can target a cluster with tier: low_priority . Online deployments have no low-priority option. After 31 March 2026 Azure Batch allocates those nodes as Spot virtual machines : surplus capacity, variable Spot price, capacity eviction , no service-level agreement. Skills measured still name low-priority compute; operate it as Spot.

Batch deployments reschedule at mini-batch granularity . Completed mini-batches are kept. There is no checkpoint inside a mini-batch, so eviction redoes the in-flight batch of files.

Constraints you should recite:

The deployment is bound to the cluster; you cannot pick dedicated versus Spot per job

If the entire cluster is preempted, or a single-node cluster loses its node, the job is canceled and must be resubmitted

Frequent eviction plus startup time can cost more than dedicated VMs, especially when the Spot rate is close to pay-as-you-go

Low-priority cores use a separate quota from dedicated cores
You are billed for powered-on nodes, including nodes evicted before they finish work

Use Spot for flexible overnight scoring, not for a hard SLA.

Security

Auth is Microsoft Entra — a user principal or a service principal / managed identity. There is no key mode on batch. Jobs run as the invoker , so authorization follows whoever called invoke . Private networking is supported; a workspace managed virtual network needs the extra batch-endpoint configuration Microsoft documents. That is why the comparison table in section 8.1 flags additional configuration for batch on managed networks.

Exam scenario

Risk operations drops 50,000 parquet files into workspaceblobstore/paths/nightly/ . Scoring takes about 40 seconds per file and does not need a synchronous reply. Create a compute cluster with min_instances: 0 , deploy the registered MLflow model to a batch endpoint (no scoring script), set mini_batch_size and instance_count for file-level parallelism, and write predictions.csv with append_row . Invoke the endpoint; poll the job; read blob output. If a challenger pipeline is ready, add a pipeline component deployment , invoke it by name, then switch defaults.deployment_name .

Common trap

Putting traffic: blue=90, green=10 on a batch endpoint. Batch has no traffic split and no mirror. Another trap: expecting batch run() to parse JSON like online run(raw_data) , or assuming a managed online deployment can scale to zero because the cluster min is 0. Online deployments are not clusters. A third trap: using a low-priority cluster for a latency-sensitive online service — online does not support low-priority VMs. A fourth: deploying AutoML's online scoring script as the batch driver.