Study guide
11.1 Serverless API Endpoints and Managed Compute for Foundation Models
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models
Key Takeaways Explain This Topic AI
Serverless API is the preferred Microsoft Foundry path for Foundry Models sold by Azure and selected partner models. Standard SKUs bill per token; provisioned SKUs reserve PTUs; Batch is about 50 percent cheaper with a 24-hour target and no real-time SLA.
Managed compute deploys weights onto dedicated GPU or CPU SKUs billed hourly. Many Hugging Face and custom-weight models require it. Classic hub-based projects consume Azure virtual-machine quota; new Foundry managed compute (preview) uses a separate Foundry accelerator quota.
Do not confuse Foundry Serverless API or Foundry managed compute with Azure Machine Learning managed online endpoints. Domain 2 endpoints host registered sklearn, PyTorch, MLflow, or Triton artifacts on instance_type and instance_count virtual machines.
Global deployment types may process prompts in any Azure region. Data Zone types keep processing in the US, EU, or APAC zone. Standard and Regional types pin processing to the deployment region. Data at rest still stays in the designated Azure geography.
Developer deployments exist only to evaluate fine-tuned models: 24-hour lifetime, no SLA, no data-residency guarantee. Instant access (preview) calls some models by name without a deployment and is not a production contract.
Last updated: August 2026
Serverless API Endpoints and Managed Compute for Foundation Models
Quick Answer: Microsoft Foundry puts a catalog foundation model into production with Serverless API (preferred for Foundry Models sold by Azure and selected partner models; pay-per-token Standard , plus Global, Data Zone, Batch, and Developer types, or reserved provisioned throughput units ) or managed compute (dedicated GPU or CPU SKUs billed hourly). These are not Azure Machine Learning managed online endpoints from Domain 2.
Domain 3 of Exam AI-300 asks you to deploy foundation models by using serverless API endpoints and managed compute options . Chapter 10 stood up the Foundry resource, project, identity, and network. This section is the first production surface those projects hit: a live inference API over a catalog model.
Microsoft Foundry is the current brand. Older material still says Azure AI Studio or Azure AI Foundry; the exam measures the Foundry catalog and its two hosting paths. The catalog holds more than 10,000 models and publishes roughly 50 new ones each month. It splits commercially into two buckets, and that split drives support, billing, and which deployment option you get.
Foundry Models sold by Azure. Microsoft hosts and sells them under Microsoft Product Terms. Every Azure OpenAI model sits here, plus selected models from providers such as Meta, Mistral, Cohere, and DeepSeek that Microsoft integrates and supports. Billing uses first-party Azure meters. Enterprise service-level agreements (SLAs) and Microsoft support apply. Some of these models also offer fungible provisioned throughput , so the same PTU pool can back more than one supported model.
Foundry Models from partners and community. This is most of the catalog: Hugging Face hub models, many open weights, and partner families such as Anthropic Claude. The model provider chooses how you deploy and sets license terms. Hugging Face support is the Hugging Face forum; Anthropic in Foundry still routes operational help through Microsoft Support. Azure Marketplace meters often apply for partner offers.
You are responsible for lawful use, for reading the model card and provider terms, for picking a model that fits the use case, and for applying content safety so the workload matches Microsoft's Acceptable Use Policy. Foundry does not pick the model for you.
Two deployment options, not four
The exam mixes Domain 2 and Domain 3 on purpose. Keep the product map small.
Feature Serverless API (Foundry Models) Managed compute (Foundry) AML managed online endpoint (Domain 2)
Typical models Azure OpenAI, selected models sold by Azure, Anthropic Claude, some Meta, Mistral, and Cohere SKUs Hugging Face and other open, partner, industry, or custom weights Registered sklearn, PyTorch, MLflow, or Triton artifacts
Hosting Microsoft-hosted inference API; no virtual machines in your subscription Dedicated GPU or CPU (accelerator) SKUs that Foundry or Azure Machine Learning manage Dedicated virtual-machine instances you size with instance_type and instance_count
Billing Pay-per-token Standard, PTUs hourly or with reservations, Batch at about 50 percent of Standard Hourly SKU (classic: virtual-machine core hours; new Foundry managed compute preview: hourly per accelerator) Hourly virtual machines while instances run; cannot scale to zero
Quota Tokens per minute on Standard, or PTU quota per subscription, region, and deployment type Classic: Azure virtual-machine quota. New Foundry managed compute: Foundry accelerator quota, not Azure virtual-machine quota Azure Machine Learning virtual-machine quota for the SKU
Content filters Built-in Azure AI Content Safety on language models (not embeddings or time-series) Call Azure AI Content Safety separately; preview managed compute has no built-in filters You implement them
Project shape Foundry resource (full catalog) or Azure OpenAI resource (Azure OpenAI models only) Classic: hub-based project. New Foundry managed compute: Foundry project endpoint Azure Machine Learning workspace
Serverless API is the preferred deployment option in Foundry. Use it whenever the model card lists it. Use managed compute when the card offers only that path, or when you must host custom weights on dedicated accelerators.
Instant access (preview) lets you call some supported models by name without creating a Serverless API or managed compute deployment. Treat it as a try-it path, not a production contract. The exam still expects you to know how to create a deployment .
If you create the deployment from an Azure OpenAI resource , the catalog shows only Azure OpenAI models. Upgrade to a Foundry resource when you need the rest of Foundry Models sold by Azure. Deployments are Microsoft.CognitiveServices/accounts/deployments resources; Azure Policy can deny a SKU name such as GlobalStandard if the organization forbids global processing.
Serverless API: types, data residency, and billing
A Serverless API deployment provisions an inference endpoint. Microsoft hosts the weights. You send prompts and pay for usage. Authentication is API keys or Microsoft Entra ID (recommended, keyless). Private networking follows the Foundry resource public network access flag; inbound traffic can use a private endpoint on the hub or Foundry resource. Hubs that received a private endpoint or serverless deployments before 11 July 2024 may need a new private endpoint and a new serverless deployment before networking settings apply.
Inside Serverless API you pick a deployment type (SKU). The type controls where prompts are processed and how you pay :
Global Standard ( GlobalStandard ). Pay-per-token. Prompts may be processed in any Azure region where the model is hosted. Highest default quota, newest models first, usually the lowest price. Start here unless compliance forbids it.
Data Zone Standard ( DataZoneStandard ). Pay-per-token. Processing stays inside a Microsoft data zone: United States , European Union (the Azure EU Data Boundary, which can include EFTA countries such as Norway and Switzerland), or Asia Pacific . Microsoft can add regions to a zone without prior notice.
Standard ( Standard ). Pay-per-token, processing pinned to the deployment's Azure region . Lower default quota; last to receive new models; no guaranteed availability date.
Global Provisioned , Data Zone Provisioned , and Regional Provisioned . Reserved provisioned throughput units (PTUs) billed hourly. Same residency story as the matching Standard SKU. Section 11.4.
Global Batch and Data Zone Batch . Asynchronous jobs, separate enqueued-token quota, about 50 percent cheaper than the matching Standard SKU, 24-hour target turnaround, no real-time SLA.
Developer ( DeveloperTier ). Fine-tuned evaluation only . Fixed 24-hour lifetime , then automatic delete. No SLA and no data-residency guarantee. Not production. Section 11.3.
Data at rest stays in the designated Azure geography for every type. Inferencing data (prompts and completions) is what Global, Data Zone, and Regional types relocate. New models become available in a fixed order: Global, then Data Zone, then single-region. Do not block a launch on Regional Standard if Global Standard already meets residency.
Content safety for Serverless language models runs synchronously as the service processes prompts. You can disable filters at deploy time or later on the deployment details page. Embedding models and time-series models do not get built-in filters. If you call a model through a path other than the Azure OpenAI SDK with the Azure OpenAI v1 APIs, filters may stay off unless you add Azure AI Content Safety yourself. Filters are billed separately.
Create Serverless deployments in the Foundry portal, with Azure CLI, or with Bicep (Chapter 10). The control-plane caller needs a role such as Cognitive Services Contributor on the Foundry resource.
Managed compute: dedicated SKUs, hourly bills
Managed compute deploys model weights onto dedicated accelerators . You need this when the catalog lists only managed compute (typical for many Hugging Face models), when you bring custom weights , or when you need a specific serving runtime or GPU family.
Classic hub-based path. Use a hub-based project in Foundry portal (classic). Models arrive through Azure Machine Learning registries (weights, containers, evaluation and fine-tune pipelines, sample datasets). You deploy to Azure Machine Learning managed compute and call a REST API. Billing is virtual-machine core hours . You need virtual-machine quota for the product that runs the model. Some models offer a temporary shared quota for testing. Content safety is a separate Azure AI Content Safety call, billed on its own meter.
New Foundry managed compute (preview). A managed GPU platform-as-a-service on the unified Foundry project endpoint . Foundry picks GPUs from families such as A100 (80 GB), H100 (80 GB), and MI300X (192 GB) and runs Microsoft-curated vLLM, SGLang, or NVIDIA NIM containers. You size in model-centric terms rather than picking a virtual-machine SKU. Billing is hourly per accelerator SKU . Quota is granted per accelerator SKU per region through Foundry and is not Azure virtual-machine quota. Auto-scale and an idle timeout that scales to zero exist in preview. Microsoft documents this preview as without an SLA and not recommended for production . Content filtering is not available in the public preview.
If the exam names a Hugging Face model and a hub-based project, pick managed compute . If it names gpt-4.1 , gpt-5.2 , Claude, or another model sold by Azure with a Serverless SKU, pick Serverless API .
Exam scenario
A European bank wants a chat copilot on gpt-4.1 with Microsoft support, an Azure SLA, and processing that never leaves the EU Data Boundary. Traffic is bursty and still low. Create a Serverless API deployment of type Data Zone Standard in an EU data-zone region. Do not stand up A100s. Do not deploy the catalog model to an Azure Machine Learning managed online endpoint as if it were a registered MLflow artifact.
Common trap
Treating a Serverless API deployment as a Domain 2 managed online endpoint . Serverless has no instance_type , no instance_count , no Azure Machine Learning traffic-split map, and no virtual-machine bill. Another trap: forcing a Hugging Face catalog model onto Global Standard when the model card offers only managed compute. A third: using Developer or instant access as the production SKU. A fourth: assuming Global Standard keeps prompts in the resource region — it does not; only Data Zone or Regional types constrain processing.