Study guide
8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
Key Takeaways Explain This Topic AI
Test online deployments with the inference HTTP server and a local Docker endpoint (--local / local=True) before cloud deploy. Studio and ARM cannot deploy locally. Invoke a zero-traffic cloud deployment by name or with the azureml-model-deployment header.
Container logs come from the inference server (score.py print and logging) and the storage-initializer (model and code download). Azure Monitor and Application Insights provide metrics; HTTP 424 is a model/container error, 429 is pending-request or rate limit.
Progressive online rollout: create green at 0 percent live, optionally mirror up to 50 percent to one shadow deployment, then shift live traffic (for example 90/10) and finally 0/100. A deployment cannot receive live and mirrored traffic at the same time.
Rollback is shifting traffic back to the previous deployment. Do not delete the old deployment until the new one is healthy and taking 100 percent. Kubernetes online endpoints do not support mirroring.
Batch has no traffic split or mirror: test with --deployment-name, then switch defaults.deployment_name. Live online traffic percentages must total 0 or 100.
Last updated: August 2026
Test, Troubleshoot, Progressive Rollout, and Safe Rollback
Quick Answer: Prove a new green deployment locally and with zero live traffic , then mirror (shadow) production requests, then shift live traffic (for example 90/10), then 0/100. Rollback is a traffic shift back to blue. Do not delete the old deployment until green is healthy. Debug with local endpoints , the inference HTTP server , container logs (inference server and storage-initializer), Azure Monitor , and Application Insights .
Exam AI-300 Domain 2 clusters test and troubleshoot endpoints with progressive rollout and safe rollback because they are one operate loop. Shipping a new model version is not az ml online-deployment create --all-traffic in production.
Test before any live traffic
Work inward from the cheapest failure:
Azure Machine Learning inference HTTP server ( azureml-inference-server-http ) — run score.py locally without Docker . Syntax errors, missing imports, and init() crashes show up here first, isolated from container configuration.
Local endpoint — CLI --local or SDK local=True builds a Docker image that mimics the cloud. You need Docker Engine. Limitations: one deployment per endpoint , no traffic rules , no authentication , no probe settings ; models and Conda files must be local (download registered assets first). Studio and Azure Resource Manager templates cannot deploy locally. Visual Studio Code dev containers can attach a debugger inside that Docker image (preview).
Cloud deployment at 0 percent traffic — create green without --all-traffic . Invoke it with az ml online-endpoint invoke --deployment-name green or HTTP header azureml-model-deployment: green . Studio's Test tab requires a deployment name; unlike CLI/SDK, studio will not follow the traffic map on the Test tab.
--all-traffic on create is a development convenience that points 100 percent at the new deployment. Do not use it in production rollouts. If you invoke the endpoint without a deployment name, routing follows the live traffic map.
Batch testing is different: invoke with --deployment-name against the non-default deployment. There is no local batch endpoint and no traffic mirror. Confirm blob outputs and job logs, then switch defaults.deployment_name .
Logs, metrics, and Application Insights
You cannot SSH to the managed virtual machine. You can pull two container streams:
Inference server logs — print and logging from score.py ( az ml online-deployment get-logs / SDK get_logs ). Use a real logging level such as INFO or the messages never appear.
Storage-initializer logs — --container storage-initializer (SDK container_type="storage-initializer" ). Tells you whether model and code downloaded before the inference server starts.
If init() throws, you often get ResourceNotReady and empty inference logs — check storage-initializer, Conda imports, and probe timeouts. Increase probe failure thresholds or pick a larger SKU if the container is healthy but slow to start. ImageBuildFailure lives in workspace storage build logs (unpinned pip dependencies, Azure Container Registry authorization, or missing image-build compute on a private virtual network). OutOfQuota is CPU, disk, memory, role-assignment limits, endpoint-count, or regional VM capacity — remember the 20 percent upgrade reservation from section 8.1. SecretsInjectionError means the endpoint identity cannot read the referenced workspace connection or key vault, or the secret name is wrong.
Azure Monitor charts availability, latency, and request rate per deployment , which is why blue-green comparison is possible. Enable Application Insights diagnostics on the deployment (studio toggle; still documented as supported, including a legacy path) to inspect user-container events. Bandwidth throttling (default 5 MBps per endpoint) shows up as delay trailers ( ms-azureml-bandwidth-request-delay-ms ). Mirroring counts against that cap — that is why mirror traffic is capped.
Useful HTTP codes when you score managed online endpoints:
- Code Meaning
- 200 Model succeeded inside latency bounds
- 401 / 403 Missing, expired, or wrong-audience token, or missing score/action
- 404 No deployment with positive traffic weight
- 408 Exceeded request_timeout_ms
- 424 Container returned non-200 ( model error ); inspect ms-azureml-model-error-statuscode
429 Too many pending requests ( 2 * max_concurrent_requests_per_instance * instance_count ) or platform rate limit
500 Platform failure
Treat 424 as a problem in score.py , the model, or probes — not as a generic Azure outage. Send x-ms-client-request-id (alphanumeric, hyphen, underscore; truncated at 40 characters) for your traces. x-request-id is reserved and overwritten by the service; attach it to a support ticket.
v2 online endpoints do not natively support Cross-Origin Resource Sharing (CORS) . Browser apps need a gateway or Azure Functions in front. That is an operate fact, not a reason to pick batch.
Progressive rollout on online endpoints
Microsoft's documented blue-green path:
Endpoint exists; blue has 100 percent live traffic.
Scale blue ( instance_count or autoscale) if load requires it before you add risk.
Create green with 0 percent live traffic.
Invoke green by name (CLI, SDK, header, or studio Test).
Mirror a percentage of live traffic to green (shadow / dark launch).
Shift live traffic, for example blue=90, green=10 .
Shift green=100, blue=0 when metrics look good.
Only then delete blue.
Mirror rules that show up on items:
- Maximum 50 percent mirror
- One shadow deployment per endpoint
- A deployment receives either live or mirrored traffic, not both
- Clients still receive blue's responses ; green's output is discarded but metrics and logs are recorded
- Invokes that name a deployment are not mirrored
- Kubernetes online endpoints do not support mirroring
- CLI older than 2.4.0 or SDK older than 1.0.0 can drop the mirror setting on update
- Live percentages must total 0 or 100
- CLI sketches: az ml online-endpoint update --name fraud-online --mirror-traffic "green=10"
- az ml online-endpoint update --name fraud-online --mirror-traffic "green=0"
- az ml online-endpoint update --name fraud-online --traffic "blue=90 green=10"
- az ml online-endpoint update --name fraud-online --traffic "blue=0 green=100"
Rollback is the inverse traffic command, not a delete: az ml online-endpoint update --name fraud-online --traffic "blue=100 green=0"
The scoring URI and credentials stay the same. Deleting green, deleting blue, or deleting the endpoint is not rollback. Studio refuses to delete a deployment that still has live traffic — set that deployment to 0 percent first. After green has taken 100 percent long enough that you trust it, az ml online-deployment delete --name blue removes unused VMs. Deleting the endpoint deletes every deployment under it.
Batch "rollout" is switching defaults.deployment_name after a named invoke. There is no 90/10 and no mirror. Keep the previous deployment until the new default's jobs look clean.
Putting the operate loop together
A durable MLOps habit for online endpoints:
- Register model version N+1 (Chapter 7) rather than overwriting files in place
- Deploy it as a new named deployment , not by mutating blue in place
- Compare deployments in Azure Monitor (latency, error rate, 424s) before shifting live traffic
- Keep blue running through the canary so rollback is one traffic update
- Only after green is the sole live deployment do you delete blue to stop paying for its VMs
That is why the endpoint/deployment split in section 8.1 exists. If you had only one object, every model bump would change the URL or require a destructive replace.
Exam scenario
blue serves 100 percent of checkout scoring on fraud-xgb:3 . Version 4 is registered. Create green at 0 percent, run the inference HTTP server and a local Docker endpoint against score.py , then cloud-invoke with azureml-model-deployment: green . Mirror 10 percent; compare latency and 424 rates in Monitor. Move live traffic to 90/10, then 0/100. If p99 doubles, set traffic back to blue=100 before deleting anything. After a week of green at 100 percent, delete blue. Do not apply this traffic map to a batch endpoint — switch that product's default instead.
Common trap
Deleting blue as soon as green is created, or assuming mirror traffic changes what customers see . Mirror is observation only. Another trap: applying online traffic split to a batch endpoint, or enabling mirror on Kubernetes . A third: treating HTTP 424 as a gateway outage (500) instead of a model/container failure. A fourth: using --all-traffic on the first production create of green, which skips the canary entirely.