13.2 ALM for Custom AI Models, Fine-Tuning Artifacts & Deployment Slots

Microsoft Certified: Agentic AI Business Solutions Architect Expert AB-100

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Requirements Analysis & Grounding Data

1.1 Assessing Agents for Task Automation, Analytics & Decision-Making 1.2 Grounding Data Evaluation: Accuracy, Relevance, Timeliness, Cleanliness & Availability 1.3 Structuring Enterprise Business Data for AI System Interoperability

2 Chapter 2: AI Strategy, Cloud Adoption Framework & CoE

2.1 Cloud Adoption Framework (CAF) for Azure AI Strategy & Roadmap 2.2 Multi-Agent Architecture: Microsoft 365 Copilot, Copilot Studio & Foundry 2.3 Microsoft AI Center of Excellence (CoE) & Solution Governance 2.4 Developing Use Cases for Prebuilt Agents & the Microsoft Agent Catalog 2.5 Solution Rules & Constraints for Copilot Studio, Microsoft Foundry & Foundry Tools

3 Chapter 3: Model Selection, Prompt Engineering & Dynamics AI

3.1 Model Selection Strategy, Custom Models & Small Language Models (SLMs) 3.2 Enterprise Prompt Engineering Guidelines & Prompt Libraries 3.3 Cross-App AI Strategy for Multi-App Dynamics 365 Solutions

4 Chapter 4: ROI, TCO, Build vs Buy & Model Routing

4.1 ROI Criteria & Total Cost of Ownership (TCO) for AI Solutions 4.2 Comprehensive Financial ROI Analysis for Business Processes 4.3 Build vs Buy vs Extend Architectural Decision Framework 4.4 Dynamic Model Router Architecture & Cost-Performance Optimization

5 Chapter 5: Dynamics 365 AI & Contact Center Design

5.1 Business Terms & Copilot Customization in Customer Experience and Service 5.2 Connectors & Data Flow Design for Copilot in Dynamics 365 Sales 5.3 Agent Integration with Dynamics 365 Contact Center Channels

6 Chapter 6: Agent Archetypes & Copilot Studio Core

6.1 Designing Task Agents, Autonomous Agents & Prompt-and-Response Agents 6.2 Copilot Studio Topic Design, Trigger Phrases & Graceful Fallbacks 6.3 Orchestration Selection: Standard NLP vs CLU vs Generative AI Orchestration

7 Chapter 7: Copilot Studio Advanced Actions & Power Platform

7.1 Designing Agent Flows & Prompt Actions in Copilot Studio 7.2 Power Apps Canvas App Integration, Generative Pages & Agent Feeds 7.3 Power Platform Well-Architected Framework for AI Workloads 7.4 Proposing Microsoft Power Platform AI Features & the AI Hub

8 Chapter 8: Agent Extensibility, MCP & Computer Use

8.1 Microsoft Foundry Custom Models & Foundry Tools Selection 8.2 Agent Extensibility via Model Context Protocol (MCP) in Copilot Studio 8.3 Computer Use Automation & Web UI Interactions in Copilot Studio

9 Chapter 9: Advanced Behaviors & Prebuilt App Orchestration

9.1 Agent Behaviors: Multi-Step Reasoning, Voice Mode & M365 Agent Optimization 9.2 Orchestrating Prebuilt AI in Dynamics 365 F&O and M365 Copilot for Sales/Service 9.3 Orchestrating Prebuilt Copilot AI Features in Dynamics 365 Customer Experience & Service 9.4 Proposing Microsoft 365 Agents for Business Scenarios 9.5 Adding Knowledge Sources to In-App Help & Guidance in Dynamics 365 Finance & Supply Chain

10 Chapter 10: Telemetry, Monitoring & Agent Tuning

10.1 Monitoring Architectures: Azure Monitor, Application Insights & Studio Analytics 10.2 Telemetry Data Interpretation, Performance Metrics & Latency Tuning 10.3 User Feedback Analysis, Backlog Prioritization & Continuous Improvement

11 Chapter 11: Testing Strategy & Model Validation

11.1 Agent Testing Frameworks, Quality Metrics & Groundedness Benchmarks 11.2 Custom AI Model Validation, Safety Criteria & Regression Testing 11.3 Copilot-Assisted Test Case Generation & Cross-Dynamics End-to-End Testing 11.4 Validating Effective Copilot Prompt Practices

12 Chapter 12: ALM for Grounding Data & Copilot Studio Agents

12.1 ALM for Grounding Data, Vector Indexes & Knowledge Synchronization 12.2 ALM for Copilot Studio Agents, Connectors & Solution Packaging 12.3 Environment Strategy & Automated Deployment Pipelines for Studio Agents

13 Chapter 13: ALM for Foundry Agents & Dynamics 365 AI

13.1 ALM for Microsoft Foundry Agents Service & Model Registries 13.2 ALM for Custom AI Models, Fine-Tuning Artifacts & Deployment Slots 13.3 ALM for Dynamics 365 AI in Finance, Supply Chain & Customer Engagement

14 Chapter 14: Agent Identity, Security & Access Controls

14.1 Agent Identity Architecture, Microsoft Entra ID & Authentication Flows 14.2 Access Controls, Role-Based Security & Permissions on Grounding Data 14.3 Secure Data Processing & Grounding Data Isolation

15 Chapter 15: Model Security & Threat Mitigation

15.1 Model Security Architecture & Defense Against Vulnerabilities 15.2 Mitigating Prompt Manipulation, Jailbreaks & Indirect Prompt Injections 15.3 Azure AI Content Safety, Prompt Shields & Guardrails

16 Chapter 16: Responsible AI, Governance, Residency & Audit

16.1 Microsoft Responsible AI Standard: Principles, Assessment & Implementation 16.2 Data Residency, Sovereign Boundaries & Cross-Border Compliance 16.3 Comprehensive Audit Trails, Model Decision Logging & Microsoft Purview 16.4 Designing Governance for Agents: Environment Strategy, Data Policies & Agent Inventory

13.2 ALM for Custom AI Models, Fine-Tuning Artifacts & Deployment Slots

Key Takeaways Explain This Topic AI

ALM for fine-tuned models mandates cryptographic dataset lineage tracking (SHA-256 hashes), hyperparameter logging, and registering immutable output weights in MLflow-compliant Azure AI Foundry Model Registries.

Zero-downtime model deployments require Managed Online Endpoints utilizing traffic splitting across deployment slots, enabling gradual Canary rollouts (e.g., 90/10 -> 50/50 -> 100/0) or instant Blue/Green cutovers.

Provisioned Throughput Units (PTU) guarantee latency SLAs and eliminate 429 throttling for critical agent workloads, requiring automated cross-region capacity allocation and burst fallback to Pay-As-You-Go during traffic spikes.

Automated rollback triggers combine Azure Monitor alert rules on HTTP 5xx error spikes, p95 latency degradation, and automated evaluation threshold breaches to instantly revert endpoint traffic pointers to the previous stable model version.

MLflow model stage transitions (Candidate -> Staging -> Production -> Archived) govern the promotional lifecycle, ensuring custom model artifacts are never promoted without passing automated safety and performance evaluations.

Last updated: September 2026

ALM for Custom AI Models, Fine-Tuning Artifacts & Deployment Slots

Quick Answer: The lifecycle for custom fine-tuned AI models requires rigorous dataset provenance tracking and repeatable fine-tuning pipelines integrated with MLflow and Azure AI Foundry Model Registries. Deploying custom models to production demands zero-downtime patterns —specifically Managed Online Endpoints utilizing deployment slots and traffic splitting (Canary and Blue/Green). Production stability is safeguarded by Provisioned Throughput Units (PTU) capacity planning, metric-based autoscaling, and automated rollback triggers driven by Azure Monitor alerts on HTTP 5xx errors, latency anomalies, or evaluation score regressions.

While general-purpose foundation models excel at broad reasoning, enterprise business solutions frequently demand custom fine-tuned models or Small Language Models (SLMs) tailored to specific domain vocabularies, specialized accounting schemas, or structured output compliance. However, custom models introduce significant ALM complexity: training datasets evolve, hyperparameter tuning produces non-deterministic artifacts, GPU compute quotas are constrained, and serving multi-gigabyte neural network weights in production carries high latency and cost risks. Solutions architects must establish disciplined operational pipelines governing the model from training data ingestion through zero-downtime serving.

1. Custom Model Fine-Tuning Lifecycle & Dataset Lineage

A fine-tuned model artifact is only as reliable as the data used to train it. If an organization cannot reproduce the exact training corpus, hyperparameters, and environment conditions that generated a production model, it cannot satisfy regulatory compliance, explainability audits, or defect triage. +-----------------------------------------------------------------------------+

• | CUSTOM MODEL FINE-TUNING ALM WORKFLOW |
• +-----------------------------------------------------------------------------+
• | |
• | [ Step 1: Dataset Ingestion & Lineage ] |
• | - Ingest Training, Validation & Test Datasets (JSONL) |
• | - Generate Cryptographic Hashes (SHA-256) |
• | - Register as Versioned Data Assets in Azure AI Foundry |
• | | |
• | v |
• | [ Step 2: Repeatable Fine-Tuning Run (MLflow Tracking) ] |
• | - Hyperparameters: Learning Rate, Epochs, Batch Size, LoRA Rank |
• | - Continuous Logging: Training Loss, Validation Loss Curves |
• | - Output Model Artifacts: Weights, Tokenizer, Config |
• | | |
• | v |
• | [ Step 3: Automated Evaluation & Registration ] |
• | - Run Quality Evaluation (BLEU, ROUGE, Groundedness, Toxicity) |
• | - Assert Threshold Gates (Pass / Fail) |
• | - Register Model in MLflow Model Registry with Stage = 'Candidate' |
• | | |
• | v |
• | [ Step 4: Staging Deployment & Human Sign-Off ] |
• | - Deploy to Staging Slot -> Transition Tag to 'Staging' |
• | - Domain Expert Review & Red Teaming Verification |
• | - Promote to 'Production' via Automated Traffic Shifting |
• +-----------------------------------------------------------------------------+
• Dataset Lineage & Versioning Controls

In Azure AI Foundry, datasets used for fine-tuning must be registered as Versioned Data Assets rather than ad-hoc storage paths:

Cryptographic Provenance: When a dataset (e.g., financial_audit_qa_v2.jsonl ) is ingested, the pipeline computes a SHA-256 checksum over the raw payload and registers the asset in the Azure AI Foundry Data Store. If a single training example is modified, added, or removed, the checksum changes, enforcing the creation of a new semantic version (e.g., v2.1.0 ).

Immutable Split Separation: The pipeline enforces strict physical separation of Training (80%), Validation (10%), and Holdout Test (10%) datasets. Test sets are cryptographically locked to prevent data leakage and benchmark gaming.

Data Cleansing & Sanitization: Pre-processing scripts strip Personally Identifiable Information (PII), proprietary customer account numbers, and copyrighted fragments, logging sanitization audit receipts to Azure Monitor.

Fine-Tuning Run Repeatability via MLflow Tracking

During model fine-tuning (e.g., fine-tuning Phi-3.5-mini or gpt-4o-mini ), the execution run is tracked via MLflow integration. The training harness logs every operational parameter: # fine_tune_tracker.py: Tracking Run Parameters via MLflow

• import mlflow
• from azure.ai.projects import AIProjectClient
• from azure.identity import DefaultAzureCredential

mlflow.start_run(run_name="finetune-phi35-financial-v2")

• # 1. Log Training Hyperparameters
• mlflow.log_params({
• "base_model": "Phi-3.5-mini-instruct",
• "dataset_version": "v2.1.0",
• "dataset_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
• "learning_rate": 2e-5,
• "learning_rate_scheduler": "cosine",
• "batch_size": 16,
• "epochs": 4,
• "lora_rank": 16,
• "lora_alpha": 32,
• "target_modules": ["q_proj", "v_proj"]
• })

• # 2. Log Evaluation Benchmark Metrics
• mlflow.log_metrics({
• "final_train_loss": 0.142,
• "final_validation_loss": 0.168,
• "groundedness_score": 0.94,
• "json_format_compliance": 0.998,
• "jailbreak_resistance_pct": 100.0
• })

• # 3. Register Output Model Artifact
• model_uri = f"runs:/{mlflow.active_run().info.run_id}/model"
• registered_model = mlflow.register_model(
• model_uri=model_uri,
• name="phi35-financial-auditor-custom"
• )

• mlflow.end_run()
• MLflow Model Registry Lifecycle Stages

Once registered, model artifacts transition through formal governance stages:

Candidate : Model trained and registered; undergoes automated static and safety evaluation.

Staging : Model deployed to pre-production online inference deployment slots for load testing and human sign-off.

Production : Model actively serving live production inference requests via Canary or Blue/Green routing.

Archived : Deprecated model versions retained in cold storage for regulatory audit trails and historical repeatability.

2. Zero-Downtime Deployment Patterns: Online Endpoints & Traffic Splitting

In mission-critical business applications, custom AI models cannot be updated by taking the endpoint offline or performing in-place replacement. Swapping model weights in-place causes request dropouts, cold-start latency spikes, and catastrophic downtime if the new model fails on live traffic.

Enterprise solutions utilize Managed Online Endpoints in Azure AI Foundry / Azure ML, which natively support multiple Deployment Slots behind a single, static HTTP inference URL. =========================================================================

• PHASE 1: STABLE BASELINE (100% TRAFFIC TO BLUE)
• =========================================================================
• [ Client / Agent ] ----> [ Endpoint: 'https://ai.corp.com/score' ]
• |
• +---> (100%) [ Deployment: 'blue-v1' ] (Current)
• +---> (0%) [ Deployment: 'green-v2' ] (Standby)

• =========================================================================
• PHASE 2: CANARY TRAFFIC SPLIT (10% TO GREEN / 90% TO BLUE)
• =========================================================================
• [ Client / Agent ] ----> [ Endpoint: 'https://ai.corp.com/score' ]
• |
• +---> (90%) [ Deployment: 'blue-v1' ] (Monitoring)
• +---> (10%) [ Deployment: 'green-v2' ] (Canary Active)

• =========================================================================
• PHASE 3: COMPLETE CUTOVER (100% TRAFFIC TO GREEN)
• =========================================================================
• [ Client / Agent ] ----> [ Endpoint: 'https://ai.corp.com/score' ]
• |
• +---> (0%) [ Deployment: 'blue-v1' ] (Standby / Retire)
• +---> (100%) [ Deployment: 'green-v2' ] (New Production)
• Deployment Strategies for Custom Inference Endpoints

Deployment Pattern Traffic Allocation Mechanics Risk Profile Rollback Speed Compute Cost Overhead

Canary Deployment Directs a small percentage of live traffic (e.g., 5% -> 10% -> 50%) to the new deployment slot while monitoring telemetry Lowest risk; defects affect only a fraction of real users; validates real-world load Fast; set traffic to 0% on Canary slot instantly Low to moderate; scales instances in Canary slot proportionally to split

Blue/Green Deployment Deploys new version (Green) alongside active version (Blue) at full capacity; cuts 100% traffic instantly upon verification Low risk; enables extensive pre-cutover synthetic validation; instantaneous cutover Instantaneous; repoint 100% traffic back to Blue slot High; requires running two full-capacity production clusters in parallel during cutover

Shadow / Dark Launch Clones 100% of live incoming requests to the new model in parallel, but discards responses and returns only the stable model output Zero user impact; ideal for comparing latency and output distributions under identical live loads Instantaneous; shut down shadow deployment High; doubles compute consumption for the duration of the evaluation window

In-Place Recreate (Anti-Pattern) Destroys the existing model container and starts the new model container on the same compute instance Unacceptable; induces minutes of complete service outage and fails immediately if container crashes Slow; requires full rebuild of prior container Lowest; zero surplus compute instances

Azure CLI Deployment Slot Orchestration

The following Azure CLI workflow demonstrates zero-downtime Canary traffic splitting for a Managed Online Endpoint: # Step 1: Create the new deployment slot ('green-v2') behind the existing endpoint

• az ml online-deployment create \
• -name green-v2 \
• -endpoint-name endpoint-financial-auditor \
• -model phi35-financial-auditor-custom:2 \
• -instance-type Standard_NC24ads_A100_v4 \
• -instance-count 2 \
• -all-traffic false

• # Step 2: Route 10% of live production traffic to the new Canary deployment
• az ml online-endpoint update \
• -name endpoint-financial-auditor \
• -traffic "blue-v1=90 green-v2=10"

• # Step 3: Monitor telemetry during soak period (e.g., 2 hours). If stable, complete cutover
• az ml online-endpoint update \
• -name endpoint-financial-auditor \
• -traffic "blue-v1=0 green-v2=100"

• # Step 4: Scale down or retire the previous deployment slot
• az ml online-deployment delete \
• -name blue-v1 \
• -endpoint-name endpoint-financial-auditor \
• -yes
• 3. PTU Capacity Management, Autoscaling & Quota Promotion

Deploying high-throughput language models in enterprise production requires careful management of Provisioned Throughput Units (PTU) and compute quotas.

Provisioned Throughput Units (PTU) vs. Pay-As-You-Go (PAYG)

In Azure OpenAI and Azure AI Foundry, foundation models are consumed via two distinct capacity models:

Pay-As-You-Go (PAYG): Shared multi-tenant infrastructure where users pay per processed token. While cost-effective for development and variable workloads, PAYG is subject to dynamic request throttling ( HTTP 429 Too Many Requests ) during regional demand surges.

Provisioned Throughput Units (PTU): Dedicated, reserved model-processing capacity assigned to an enterprise subscription. A PTU allocation guarantees consistent throughput, eliminates 429 rate-limiting within the provisioned envelope, and provides predictable monthly billing. +-----------------------------------------------------------------------------+

• | HIGH-AVAILABILITY MULTI-REGION PTU ROUTING |
• +-----------------------------------------------------------------------------+
• |
• v
• [ Azure Front Door / APIM ]
• (Global Health Probe & Router)
• / \
• Primary Region (90%) Secondary Region (10% / Failover)
• v v
• +---------------------------+ +---------------------------+
• | East US 2 PTU Cluster | | West US 3 PTU Cluster |
• | - 100 Reserved PTU | | - 100 Reserved PTU |
• | - Bursting: Enabled | | - Bursting: Enabled |
• +---------------------------+ +---------------------------+
• | |
• +-------------------+-----------------------+
• |
• Threshold > 90% Utilization
• v
• [ Pay-As-You-Go Burst Spillover ]
• (Absorbs transient traffic spikes)
• PTU Capacity Sizing & Promotion Guidelines

Benchmarking Peak Concurrency: Calculate required PTU capacity using empirical load testing data: Required PTU = (Peak Requests Per Minute * Average Tokens Per Request) / Baseline Throughput Factor . Over-provisioning wastes cloud budget; under-provisioning induces internal queuing.

Dynamic PTU Bursting: Azure AI Foundry supports bursting beyond provisioned PTU allocations into Pay-As-You-Go capacity during transient spikes, ensuring that sudden enterprise volume surges do not drop transactions.

Multi-Region Load Balancing: For mission-critical agents, architects deploy redundant PTU allocations across paired Azure regions (e.g., East US 2 and West US 3) fronted by Azure API Management (APIM) or Azure Front Door . APIM dynamically inspects HTTP status codes and routes traffic to the secondary region if the primary PTU cluster experiences saturation or latency degradation.

Metric-Based Autoscaling Policies for Managed Endpoints

For custom models running on Managed Online Endpoints (GPU virtual machines), architects configure auto-scaling rules based on real-time hardware and inference telemetry:

Scale-Out Trigger: Increase instance count by 1 when Average Request Concurrency > 8 or GPU Memory Utilization > 85% for 3 consecutive minutes.

Scale-In Trigger: Decrease instance count by 1 when Average Request Concurrency < 2 and GPU Memory Utilization < 40% for 15 consecutive minutes (incorporating a generous cool-down window to prevent rapid scaling churn).

4. Automated Rollback Triggers & Telemetry Gating

Even with rigorous pre-deployment evaluation, subtle issues—such as edge-case prompt loops, memory leaks in custom model kernels, or upstream dependency timeouts—can manifest only under live enterprise concurrency. A production-ready ALM pipeline must automate telemetry-driven rollback . +-----------------------------------------------------------------------------+

• | AUTOMATED ROLLBACK DECISION ENGINE |
• +-----------------------------------------------------------------------------+
• | |
• | [ Telemetry Stream ] ----> [ Azure Monitor & App Insights ] |
• | - HTTP Status Codes (5xx / 429) |
• | - Latency Metrics (p95, p99) |
• | - Safety Block Rate / Content Filters |
• | | |
• | v |
• | [ Azure Monitor Alert Rule ] |
• | - Condition: 5xx Error Rate > 1.5% |
• | - Condition: p95 Latency > 2500ms |
• | - Duration: 2 consecutive evaluation minutes |
• | | |
• | v |
• | [ Action Group -> CI/CD Webhook ] |
• | | |
• | v |
• | [ Automated Remediation Script ] |
• | - az ml online-endpoint update |
• | - Set Traffic: green-v2 = 0, blue-v1 = 100 |
• | - Notify SecOps & On-Call Engineering |
• +-----------------------------------------------------------------------------+
• Automated Rollback Threshold Criteria

An automated rollback action is triggered whenever any of the following service-level objectives (SLOs) are violated during a Canary or Blue/Green deployment window:

HTTP 5xx Server Error Rate: Greater than 1.0% of total inference requests over any 2-minute evaluation window.

Inference Latency Degradation: p95 latency exceeds 2,500 milliseconds (or 2.5x the established baseline of the previous stable deployment).

Content Safety Filter Block Rate: Spike in Content Safety filter violations (> 3% of completions blocked by Prompt Shields or Toxicity filters), indicating severe adversarial vulnerability or model hallucination drift.

Downstream JSON Parse Failures: Client application reporting deserialization exceptions exceeding a 0.5% threshold.

Rollback Execution Mechanics

When Azure Monitor detects an alert breach, it invokes an Azure Monitor Action Group configured with a secure webhook targeting the Azure DevOps or GitHub Actions release pipeline. The pipeline immediately executes an atomic rollback command: # Emergency automated rollback command executed via pipeline webhook

• az ml online-endpoint update \
• -name endpoint-financial-auditor \
• -traffic "blue-v1=100 green-v2=0"

Because the previous deployment slot ( blue-v1 ) remained fully provisioned and warm in standby mode, 100% of live user traffic is redirected to the known-good model within seconds , achieving complete service restoration without human intervention.

Loading diagram... Zero-Downtime Custom Model Canary Deployment & Automated Rollback Architecture