Study guide
16.3 Monitor, Optimize, and Promote Fine-Tuned Models
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
16.3 Monitor, Optimize, and Promote Fine-Tuned Models
Key Takeaways Explain This Topic AI
Train in non-production. Watch train_loss versus full_valid_loss and token accuracy. If they diverge, cut epochs or the learning-rate multiplier and consider an earlier checkpoint; a finished job keeps the three most recent deployable versions.
Foundry Users may train; Foundry Owner (or deployments/write) is required to deploy. Evaluate quality and safety against the base model on a held-out set before any production SKU.
Developer deployments (SKU DeveloperTier) are for candidate evaluation only: pay-per-token, no hourly hosting fee, no SLA, no data-residency guarantee, and a fixed 24-hour lifetime after which the deployment is deleted. The customized model remains.
Promote manually to Standard (regional processing) or provisioned throughput (PTU) for production. Pin the fine-tuned model ID. Automatic deployment after training is not a production gate.
Deployed custom models except Developer incur hourly hosting even at zero traffic. Unused Standard or Global custom deployments older than 15 days with 15 days of no calls are deleted. Roll back by routing to the base or prior deployment.
Last updated: August 2026
Monitor, Optimize, and Promote Fine-Tuned Models
Quick Answer: Train in non-production . Watch train_loss versus full_valid_loss and token accuracy; deploy a checkpoint if the last epoch overfits. Evaluate with quality and safety evaluators, then host on a Developer deployment ( 24-hour life, no SLA , no data-residency , no hourly hosting fee) for cheap candidate tests. Promote manually to Standard (regional) or provisioned throughput (PTU) in production, pin the version , monitor quality and hourly hosting plus tokens , and roll back to the base model . Promotion is not automatic.
The last Domain 5 fine-tuning bullet is monitor and optimize fine-tuned model performance and manage a fine-tuned model from development through production . Training success is not the same as production readiness. Foundry will not promote a job for you when train_loss looks pretty.
Train in non-production, then read the curves
Submit the job from a dev Foundry project or a non-prod Azure OpenAI resource. Use a training type that matches residency (section 16.1). After submit, jobs may queue . Training takes minutes to hours depending on the base model and dataset size.
Monitor these series on the job’s Monitor pivot or in the attached results.csv :
train_loss : loss on the training batch. Each step is one forward and backward pass. Should trend down.
full_valid_loss : validation loss at the end of each epoch . Should trend down if you provided a validation file.
train_mean_token_accuracy and full_valid_mean_token_accuracy : share of tokens predicted correctly. Should trend up when training is healthy.
If training loss keeps falling while validation loss rises , you are overfitting . Microsoft’s documented response is fewer epochs or a smaller learning-rate multiplier — not “add 10,000 more synthetic rows and hope.” You can pause a running OpenAI job after at least one step while it is in Running ; pause creates a deployable checkpoint after safety evaluations. Pause is not documented as a universal managed-compute feature.
Each epoch writes a checkpoint ( ftchkpt-... ). A checkpoint is a fully functional model: you can deploy it or use it as the base for a later fine-tune. When the job finishes, the three most recent versions are available to deploy. The final epoch is the fine-tuned model; the previous two epochs remain as checkpoints. If epoch 3 overfits, deploy epoch 2. Checkpoints can be copied across resources and subscriptions with the REST API.
results.csv columns include step , train_loss , train_mean_token_accuracy , valid_loss , validation_mean_token_accuracy , full_valid_loss , and full_valid_mean_token_accuracy . The portal plots loss and token accuracy with training and validation on the same chart when you supplied a validation file.
Loss is not product quality. A model can have beautiful curves and still fail groundedness, schema validity, or safety. Always run Foundry evaluators on a held-out set (section 16.2).
Evaluate quality and safety before any production SKU
Use the same evaluation discipline as Domain 4, pointed at the candidate and at the base model on the same dataset:
Quality: groundedness, relevance, coherence, fluency, task adherence, and tool-call correctness.
Risk and safety: hate and fairness, self-harm, violence, jailbreak, protected material.
If the fine-tune wins schema accuracy but loses a safety evaluator, you do not promote. You fix data or prompts and train again. Foundry Users may train; Foundry Owner (or a custom role with Microsoft.CognitiveServices/accounts/deployments/write ) deploys. Role names recently changed from Azure AI User / Owner; IDs did not.
At inference, the system message must match training . Microsoft documents that a different system message is a reason the custom model will not behave as expected. Pin the system message in the same Git repo as the JSONL.
Continuous fine-tuning (OpenAI models only) is the path when production traces show new failure modes: take the previous base.ft-{jobid} as the new base, train on new disjoint JSONL, eval again, then promote a new pinned deployment. Do not silently overwrite the production deployment name with an unevaluated job.
Developer deployments for 24-hour evaluation
Do not pay Standard hourly hosting just to run an eval suite.
Developer inference deployments ( DeveloperTier in the comparison table, developertier in ARM examples, --sku-name "Developer" in Azure CLI) exist only to evaluate fine-tuned candidates:
- Pay-per-token only — no hourly hosting fee
- No availability SLA
- No data-residency guarantee (processing can leave the geography)
- Fixed 24-hour lifetime , then auto-deleted regardless of traffic
- Not for production, not for latency-sensitive tests that need an SLA
Listed for GPT-4.1, GPT-4.1-mini, GPT-4.1-nano, and o4-mini from all Azure OpenAI regions in Microsoft’s current tables (confirm the model list on the deployment-types page; it moves)
Deleting the Developer deployment does not delete the customized model. Redeploy at any time. If even your test environment requires residency or an SLA, use Standard for that test — not Developer.
Control-plane deploy uses the Azure Resource Manager deployments API (or az cognitiveservices account deployment create ), not the data-plane Chat Completions API. Example SKU payload: "sku": {"name": "developertier", "capacity": 50} .
Promote to Standard or PTU — manually
Fine-tuned models store at rest at no extra hosting cost until you deploy. Each deployed custom model except Developer incurs hourly hosting whether or not anyone calls it . Token charges apply on top when you infer.
Inference type SKU Processing / residency Billing Production?
Developer DeveloperTier None guaranteed Tokens only; gone after 24 hours No
Standard Standard Processing in the deployment region Hourly hosting plus tokens Yes, regional compliance, bursty traffic
Global Standard GlobalStandard Custom weights may be stored outside the geography; processing in any Azure region Often cheaper; hourly plus tokens Yes when residency allows
Regional provisioned (PTU) ProvisionedManaged Regional Reserved provisioned throughput units Yes, latency-sensitive high volume
Microsoft documents Global Standard fine-tune hosting as preview in the deploy guide, with regional Standard available in listed regions (for example East US 2, North Central US, Sweden Central — check the current table ). Regional PTU for some fine-tunes is listed for GPT-4o and GPT-4o-mini in North Central US and Sweden Central, and GPT-4.1 PTU in Sweden Central. Fine-tunes share the same regional PTU quota as base models: if you already purchased regional PTUs, you can host the custom model on that capacity in supported regions.
Automatic deployment after a successful training job is optional and OpenAI models only . It still requires Owner. It deploys according to the type you picked in the wizard. It is not a production promotion gate and it does not wait for quality or safety evaluators. There is no platform feature that watches eval scores and flips production traffic. You promote: eval pass → new Standard or PTU deployment with a pinned model ID → application or gateway routes traffic → previous deployment stays up for rollback.
Pin the version. Fine-tuned model IDs look like gpt-4.1-2025-04-14.ft-b044a9d3cf9c4228b5d393567f693b83 . You may instead deploy a checkpoint ID ( ftchkpt-... ). Put that ID in Bicep or Azure CLI. Do not point production at “whatever the latest job produced.”
Cross-region and cross-subscription deploy is supported when the destination region supports fine-tuning and the identity that mints the token can read the source account and write the destination. The portal cannot do cross-region deploy; use SDK or REST. Cross-tenant deploy needs two tokens and the x-ms-authorization-auxiliary header.
Prompt caching is available on select fine-tuned models and can cut latency and cost when the prefix of the prompt is identical. It is an inference optimization, not a reason to skip eval.
Monitor in production, then roll back
Once the candidate is on Standard or PTU:
Quality: sample production logs into Foundry continuous evaluation (Chapter 14). Watch groundedness, relevance, schema-parse fail rate, and tool-error rate versus the baseline you recorded at promotion.
Cost: hourly hosting (custom models bill even at zero queries per second) plus token usage. A forgotten Standard fine-tune in an unused subscription is a standing charge. Developer is the exception: tokens only, then it vanishes at 24 hours.
Inactivity (Standard / Global custom deployments): if the deployment is older than 15 days and no chat completions or Responses API calls arrive for a continuous 15-day period, Azure deletes the deployment . The customized model remains and can be redeployed. Developer deployments die at 24 hours even if busy .
Rollback: keep the base model (and the previous fine-tune) deployed or instantly redeployable. Point the app to the base deployment name. Rollback is a routing change , not a platform “undo fine-tune” button. You do not retrain to roll back.
Cleanup: delete the deployment before you delete the model. Foundry will not delete a fine-tuned model that still has a deployment. Delete unused training, validation, and results.csv files when policy requires.
Exam scenario: You fine-tuned gpt-4.1-mini in a dev project with Global training. A Developer deployment beats the base model on schema validity and passes safety evaluators on the held-out gold set. Finance requires in-region inference. You deploy Standard in Sweden Central, set the API model parameter to that deployment name, keep the base gpt-4.1-mini deployment, and alert on hosting hours plus parse-fail rate. A week later parse-fail spikes; you route traffic back to the base deployment in minutes without deleting the custom model.
Common trap: Enabling automatic deployment to Standard in production from the training wizard, skipping held-out eval, and discovering a safety regression only after users complain. Automatic deploy fires after successful training , not after successful evaluation. A second trap: leaving a Standard fine-tune deployed in an unused environment — you pay hourly until you delete it or it ages out after 15 days of inactivity. A third trap: treating Developer as a weekend demo environment; it will disappear at 24 hours even if the demo is still running.
Promotion checklist
Work this list in order. None of the boxes is optional for production:
Trained in non-production; training type matches residency policy.
Validation loss is not diverging from training loss; checkpoint chosen if the last epoch overfits.
Quality and safety evaluators passed versus the base model on a held-out gold set the generator never saw.
System message at inference matches training.
Developer 24-hour eval complete (or Standard test if residency is required even in test).
Production SKU is Standard or PTU — never Developer.
Fine-tuned model ID or checkpoint ID is pinned in IaC and app configuration.
Base model or prior fine-tune remains available for rollback.
Hosting-cost, token, latency, and quality monitors are on.
No one assumes Foundry will promote the job because training succeeded.
If a box is unchecked, the model stays a candidate. Candidates live on Developer or in storage. Production traffic stays on the pinned previous deployment.