Study guide
15.1 Model Security Architecture & Defense Against Vulnerabilities
Microsoft Certified: Agentic AI Business Solutions Architect Expert AB-100
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Requirements Analysis & Grounding Data
1.1 Assessing Agents for Task Automation, Analytics & Decision-Making 1.2 Grounding Data Evaluation: Accuracy, Relevance, Timeliness, Cleanliness & Availability 1.3 Structuring Enterprise Business Data for AI System Interoperability
2 Chapter 2: AI Strategy, Cloud Adoption Framework & CoE
2.1 Cloud Adoption Framework (CAF) for Azure AI Strategy & Roadmap 2.2 Multi-Agent Architecture: Microsoft 365 Copilot, Copilot Studio & Foundry 2.3 Microsoft AI Center of Excellence (CoE) & Solution Governance 2.4 Developing Use Cases for Prebuilt Agents & the Microsoft Agent Catalog 2.5 Solution Rules & Constraints for Copilot Studio, Microsoft Foundry & Foundry Tools
3 Chapter 3: Model Selection, Prompt Engineering & Dynamics AI
3.1 Model Selection Strategy, Custom Models & Small Language Models (SLMs) 3.2 Enterprise Prompt Engineering Guidelines & Prompt Libraries 3.3 Cross-App AI Strategy for Multi-App Dynamics 365 Solutions
4 Chapter 4: ROI, TCO, Build vs Buy & Model Routing
4.1 ROI Criteria & Total Cost of Ownership (TCO) for AI Solutions 4.2 Comprehensive Financial ROI Analysis for Business Processes 4.3 Build vs Buy vs Extend Architectural Decision Framework 4.4 Dynamic Model Router Architecture & Cost-Performance Optimization
5 Chapter 5: Dynamics 365 AI & Contact Center Design
5.1 Business Terms & Copilot Customization in Customer Experience and Service 5.2 Connectors & Data Flow Design for Copilot in Dynamics 365 Sales 5.3 Agent Integration with Dynamics 365 Contact Center Channels
6 Chapter 6: Agent Archetypes & Copilot Studio Core
6.1 Designing Task Agents, Autonomous Agents & Prompt-and-Response Agents 6.2 Copilot Studio Topic Design, Trigger Phrases & Graceful Fallbacks 6.3 Orchestration Selection: Standard NLP vs CLU vs Generative AI Orchestration
7 Chapter 7: Copilot Studio Advanced Actions & Power Platform
7.1 Designing Agent Flows & Prompt Actions in Copilot Studio 7.2 Power Apps Canvas App Integration, Generative Pages & Agent Feeds 7.3 Power Platform Well-Architected Framework for AI Workloads 7.4 Proposing Microsoft Power Platform AI Features & the AI Hub
8 Chapter 8: Agent Extensibility, MCP & Computer Use
8.1 Microsoft Foundry Custom Models & Foundry Tools Selection 8.2 Agent Extensibility via Model Context Protocol (MCP) in Copilot Studio 8.3 Computer Use Automation & Web UI Interactions in Copilot Studio
9 Chapter 9: Advanced Behaviors & Prebuilt App Orchestration
9.1 Agent Behaviors: Multi-Step Reasoning, Voice Mode & M365 Agent Optimization 9.2 Orchestrating Prebuilt AI in Dynamics 365 F&O and M365 Copilot for Sales/Service 9.3 Orchestrating Prebuilt Copilot AI Features in Dynamics 365 Customer Experience & Service 9.4 Proposing Microsoft 365 Agents for Business Scenarios 9.5 Adding Knowledge Sources to In-App Help & Guidance in Dynamics 365 Finance & Supply Chain
10 Chapter 10: Telemetry, Monitoring & Agent Tuning
10.1 Monitoring Architectures: Azure Monitor, Application Insights & Studio Analytics 10.2 Telemetry Data Interpretation, Performance Metrics & Latency Tuning 10.3 User Feedback Analysis, Backlog Prioritization & Continuous Improvement
11 Chapter 11: Testing Strategy & Model Validation
11.1 Agent Testing Frameworks, Quality Metrics & Groundedness Benchmarks 11.2 Custom AI Model Validation, Safety Criteria & Regression Testing 11.3 Copilot-Assisted Test Case Generation & Cross-Dynamics End-to-End Testing 11.4 Validating Effective Copilot Prompt Practices
12 Chapter 12: ALM for Grounding Data & Copilot Studio Agents
12.1 ALM for Grounding Data, Vector Indexes & Knowledge Synchronization 12.2 ALM for Copilot Studio Agents, Connectors & Solution Packaging 12.3 Environment Strategy & Automated Deployment Pipelines for Studio Agents
13 Chapter 13: ALM for Foundry Agents & Dynamics 365 AI
13.1 ALM for Microsoft Foundry Agents Service & Model Registries 13.2 ALM for Custom AI Models, Fine-Tuning Artifacts & Deployment Slots 13.3 ALM for Dynamics 365 AI in Finance, Supply Chain & Customer Engagement
14 Chapter 14: Agent Identity, Security & Access Controls
14.1 Agent Identity Architecture, Microsoft Entra ID & Authentication Flows 14.2 Access Controls, Role-Based Security & Permissions on Grounding Data 14.3 Secure Data Processing & Grounding Data Isolation
15 Chapter 15: Model Security & Threat Mitigation
15.1 Model Security Architecture & Defense Against Vulnerabilities 15.2 Mitigating Prompt Manipulation, Jailbreaks & Indirect Prompt Injections 15.3 Azure AI Content Safety, Prompt Shields & Guardrails
16 Chapter 16: Responsible AI, Governance, Residency & Audit
16.1 Microsoft Responsible AI Standard: Principles, Assessment & Implementation 16.2 Data Residency, Sovereign Boundaries & Cross-Border Compliance 16.3 Comprehensive Audit Trails, Model Decision Logging & Microsoft Purview 16.4 Designing Governance for Agents: Environment Strategy, Data Policies & Agent Inventory
15.1 Model Security Architecture & Defense Against Vulnerabilities
Key Takeaways Explain This Topic AI
The OWASP Top 10 for Large Language Model Applications establishes a foundational taxonomy of generative AI threats, requiring enterprise architects to map Microsoft cloud-native controls—such as Azure AI Content Safety, Microsoft Purview, and API Management—against specific risk vectors.
Mitigating Excessive Agency (LLM06) in autonomous multi-agent systems mandates the Principle of Least Privilege (PoLP), strict segregation of read-only discovery tools from state-mutating execution tools, and mandatory Human-in-the-Loop (HITL) approval gates for high-impact transactions.
Supply chain security for foundation models (LLM03) requires relying on verified registries within the Azure AI Foundry Model Catalog, scanning open-weights weights against arbitrary code execution vulnerabilities (safetensors format vs. unsafe Python pickle deserialization), and container vulnerability scanning via Microsoft Defender for Containers in Azure Container Registry (ACR).
Vector and embedding weaknesses (LLM08) and data poisoning (LLM04) are addressed by enforcing Entra ID security trimming in Azure AI Search, cryptographic content hashing, and isolating data ingestion pipelines within private virtual networks.
Denial of wallet and unbounded resource consumption (LLM10) must be controlled using Azure API Management token rate-limiting policies (TPM/RPM quotas), Provisioned Throughput Units (PTU), and strict autonomous agent turn and recursion limits.
Last updated: September 2026
Model Security Architecture & Defense Against Vulnerabilities
Quick Answer: Securing enterprise agentic AI requires a multi-layered defense-in-depth architecture that maps the OWASP Top 10 for Large Language Model Applications to Microsoft native security services. To eliminate Excessive Agency (LLM06)—the paramount risk in autonomous multi-agent systems—architects must enforce the Principle of Least Privilege on agent toolsets, strictly isolate read-only queries from state-mutating actions, and insert mandatory Human-in-the-Loop (HITL) verification gates. Furthermore, model supply chains must be safeguarded by pulling verified weights from the Azure AI Foundry Model Catalog, adopting safe serialization formats ( safetensors ), and scanning agent container images using Microsoft Defender for Containers in Azure Container Registry.
As enterprise architectures evolve from static conversational bots to autonomous multi-agent systems, the attack surface expands exponentially. Agents possess dynamic planning capabilities, maintain persistent memory across sessions, and execute functional tools that interface with enterprise databases, ERP platforms, and customer communication channels. Securing these architectures requires moving beyond traditional network perimeter defense to implement comprehensive model security architecture and automated threat mitigation.
1. Mapping the OWASP Top 10 for LLMs to Microsoft Solutions
The Open Worldwide Application Security Project (OWASP) maintains the authoritative benchmark for generative AI vulnerabilities: the OWASP Top 10 for Large Language Model Applications . For the Microsoft AB-100 exam, solutions architects must understand not only the theoretical risk of each vulnerability, but the precise Microsoft native service and architectural pattern used to mitigate it. +-----------------------------------------------------------------------------+
- | ENTERPRISE AGENTIC SYSTEM: OWASP THREAT MITIGATION TAXONOMY |
- +-----------------------------------------------------------------------------+
- | INGRESS / PROMPT LAYER: |
- | - LLM01 Prompt Injection --> Azure AI Content Safety Prompt Shields |
- | - LLM07 System Prompt Leakage --> Delimiter Encapsulation & Metaprompts |
- +-----------------------------------------------------------------------------+
- | INFERENCE & GROUNDING LAYER: |
- | - LLM04 Data & Model Poisoning --> Virtual Network Isolation & RBAC |
- | - LLM08 Vector Weaknesses --> Azure AI Search Security Trimming |
- | - LLM09 Misinformation --> Azure AI Groundedness Detection API |
- | - LLM10 Unbounded Consumption --> Azure API Management (APIM) Quotas/TPM |
- +-----------------------------------------------------------------------------+
- | EXECUTION & AGENT TOOL LAYER: |
- | - LLM02 Sensitive Info Leakage --> Microsoft Purview DLP & PII Masking |
- | - LLM03 Supply Chain Defects --> Azure AI Foundry Catalog & ACR Defender|
- | - LLM05 Improper Output Handling --> Strict Structured JSON / Pydantic Schema|
- | - LLM06 Excessive Agency --> Tool Segregation & HITL Approval Gates |
- +-----------------------------------------------------------------------------+
- Comprehensive Vulnerability & Mitigation Matrix
OWASP Identifier Vulnerability Name Threat Vector & Attack Mechanics Microsoft Native Solution & Architectural Pattern
LLM01 Prompt Injection Adversarial user input (Direct) or tainted external data (Indirect) manipulates model execution logic. Deploy Azure AI Content Safety Prompt Shields for user prompts and document spotlighting; enforce XML delimiter isolation.
LLM02 Sensitive Information Disclosure Model inadvertently reveals Personally Identifiable Information (PII), credentials, or proprietary business data. Integrate Microsoft Purview Information Protection (Sensitivity Labels, DLP), Azure AI Content Safety PII redaction filters, and Customer Lockbox.
LLM03 Supply Chain Vulnerabilities Compromised third-party base models, vulnerable Python libraries, poisoned fine-tuning weights, or malicious plugins. Source models exclusively through Azure AI Foundry Model Catalog ; scan containers in Azure Container Registry (ACR) with Microsoft Defender for Containers .
LLM04 Data and Model Poisoning Malicious data introduced into training, fine-tuning, or vector grounding datasets corrupts model reasoning or inserts backdoors. Secure training data stores in Azure Data Lake Storage Gen2 with Managed Identities and Azure Private Link ; enforce cryptographic hashing ( SHA-256 ) on ingested grounding files.
LLM05 Improper Output Handling Downstream systems blindly accept raw LLM string generation, enabling Cross-Site Scripting (XSS), SQL injection, or remote shell execution. Enforce OpenAI Structured Outputs (JSON Schema) ; sanitize and validate all LLM output on the consumer side before passing to backend APIs or UI renderers.
LLM06 Excessive Agency Agents granted excessive permissions, autonomous loop authority, or multi-tool access perform unauthorized or destructive actions. Enforce the Principle of Least Privilege (PoLP) ; segregate read from write tools; mandate Human-in-the-Loop (HITL) approval before executing state mutations.
LLM07 System Prompt Leakage Attackers use evasion prompts to trick the LLM into disclosing confidential metaprompts, system instructions, or internal schemas. Harden metaprompts with strict refusal boundaries; deploy egress regex/canary token detectors; enforce Prompt Shields on prompt extraction attempts.
LLM08 Vector and Embedding Weaknesses Poisoned vector embeddings skew similarity searches, or unauthenticated users retrieve unauthorized document chunks. Implement Security Trimming (ACL filtering) in Azure AI Search using Entra ID user identity tokens; isolate vector indexes via private endpoints.
LLM09 Misinformation (Hallucination) Models generate factually false, ungrounded, or misleading statements presented with high confidence. Enable the Azure AI Groundedness Detection API ; configure low temperature (0.0–0.2); mandate explicit citations for all generated assertions.
LLM10 Unbounded Consumption Attackers execute high-frequency or compute-intensive queries, causing Denial of Service (DoS) or catastrophic financial "Denial of Wallet". Place Azure API Management (APIM) in front of Azure OpenAI; enforce Tokens-Per-Minute (TPM) and Requests-Per-Minute (RPM) rate limits; utilize Provisioned Throughput Units (PTU) .
2. Mitigating Excessive Agency in Autonomous Multi-Agent Systems
In autonomous multi-agent architectures (e.g., orchestrators built with Semantic Kernel, AutoGen, or Microsoft Copilot Studio), Excessive Agency (LLM06) represents the highest systemic threat. Excessive agency arises when an agent is granted:
Excessive Functionality: Access to tools that can alter critical business state without justification.
Excessive Permissions: Running connectors under broad service account privileges rather than the delegated context of the invoking user.
Excessive Autonomy: Executing multi-step planning loops, recursive tool chaining, and automated financial or operational actions without intermediate verification. [ End User Request ]
- |
- v
- [ Multi-Agent Orchestrator (Copilot Studio / Semantic Kernel) ]
- |
- +-------------------------------+-------------------------------+
- | |
- v (Read-Only Tool) v (State-Mutating Tool)
- +-------------------------------+ +-------------------------------+
- | Query Customer Balance | | Issue $5,000 Refund |
- +-------------------------------+ +-------------------------------+
- | - Granted by Default | | - Principle of Least Privilege|
- | - Delegated Entra ID Token | | - Requires Explicit Approval |
- | - Execution Proceed Instantly | +-------------------------------+
- | |
- v v
- [ Return Account Summary ] +-------------------------------+
- | HUMAN-IN-THE-LOOP (HITL) |
- | APPROVAL GATE |
- +-------------------------------+
- | Adaptive Card sent to Manager |
- | via Microsoft Teams / Outlook |
- +-------------------------------+
- / \
- [Approved] / \ [Rejected]
- v v
- [Execute Refund] [Halt Action]
- Architectural Principles for Eliminating Excessive Agency
- A. Principle of Least Privilege (PoLP) for Agent Toolsets
Agents must never be provisioned with generic "super-user" connectors. Instead:
Granular Scoping: Do not grant an agent a full SAP or Dataverse connector if it only requires access to the CustomerInquiries table. Use scoped API roles and row-level security.
Delegated User Context (On-Behalf-Of Flow): Where feasible, configure tools to run under delegated user credentials ( On-Behalf-Of OAuth2 tokens) rather than static application service principals. If User A cannot issue a refund in the ERP UI, the agent acting on behalf of User A must fail with an HTTP 403 Forbidden.
Read-Only vs. Read-Write Tool Segregation: Separate read-only discovery tools ( search_knowledge_base , get_order_status , check_inventory ) from state-mutating execution tools ( update_order_status , transfer_funds , delete_record ). Different subagents in a multi-agent team should be assigned distinct roles; a "Research Agent" must possess zero write tools.
B. Mandatory Human-in-the-Loop (HITL) Confirmation Gates
State-mutating operations that exceed defined risk thresholds must trigger a synchronous or asynchronous Human-in-the-Loop (HITL) verification step before the transaction commit occurs:
Copilot Studio & Power Automate: An agent topic identifies the intent to execute a state mutation (e.g., modifying payroll records). Rather than calling the ERP connector directly, the topic triggers a Power Automate flow that dispatches an Approvals Adaptive Card to a manager's Microsoft Teams client.
State Suspension & Resume: The agent's conversation state is paused in Dataverse until the manager clicks "Approve" or "Reject". Upon approval, the flow resumes and executes the mutation under the approving manager's identity audit trail.
Explicit User Confirmation: For lower-risk transactions, the agent must present an explicit confirmation card summarizing the planned parameters (target account, transfer amount, recipient) and require positive user button interaction before calling the tool.
C. Autonomous Loop Governance & Execution Budgets
Autonomous reasoning loops (e.g., ReAct patterns where an LLM plans, acts, observes, and repeats) are vulnerable to infinite execution loops caused by confusing tool responses or adversarial inputs.
Maximum Turn Limits: Enforce strict execution budgets. In Copilot Studio and Semantic Kernel, configure maximum reasoning steps (e.g., max_iterations = 5 ). If the agent fails to reach a terminal state within 5 turns, execution aborts and routes to the system fallback topic.
Token & Cost Ceilings: Bound total prompt and completion token expenditure per user session to prevent algorithmic resource exhaustion.
3. Safeguarding Model Supply Chains, Weights & Containers
Deploying enterprise agent solutions often involves integrating foundation models, open-weights domain models (e.g., Mistral, Llama 3), and custom containerized agent runtimes. This introduces severe supply chain vulnerabilities (OWASP LLM03). [ External Open Source Ecosystem ]
- Hugging Face / Public Registries
- Risk: Arbitrary Code Execution (pickle)
- Risk: Tampered Weights / Backdoors
- |
- v (Untrusted Ingestion)
- +-----------------------------------------------------------------------------+
- | AZURE TRUST BOUNDARY & DEFENSE |
- +-----------------------------------------------------------------------------+
- | 1. FOUNDRY MODEL CATALOG |
- | - Microsoft-curated & scanned foundation models |
- | - Deployed as Serverless APIs or Managed Compute |
- | - Verified SafeTensors serialization (zero pickle deserialization) |
- | |
- | 2. AZURE CONTAINER REGISTRY (ACR) |
- | - Private registry for agent microservices & sidecars |
- | - Microsoft Defender for Containers continuous vulnerability scan |
- | - Enforce image signing with Azure Key Vault & Notation |
- | |
- | 3. NETWORK PERIMETER ISOLATION |
- | - Virtual Network (VNet) injection & Azure Private Link |
- | - Disallow public endpoint access (`publicNetworkAccess = Disabled`) |
- +-----------------------------------------------------------------------------+
- The SafeTensors Imperative vs. Serialized Python Pickles
In the open-source machine learning ecosystem, models have historically been saved using Python's pickle library (or PyTorch .pt / .bin files, which utilize pickle under the hood).
The Pickle Threat: Python pickle is a general-purpose object serialization format capable of executing arbitrary bytecode upon deserialization. An attacker hosting a fine-tuned model on a public repository can embed malicious Python payloads inside the model file. When an engineer executes torch.load('weights.pt') , the embedded exploit runs with the full privileges of the host machine, establishing reverse shells or exfiltrating credentials.
The SafeTensors Standard: Enterprise solutions must mandate the SafeTensors format ( .safetensors ). SafeTensors is an open-source, memory-mapped format developed by Hugging Face that stores only raw tensor buffers and JSON metadata. It is mathematically incapable of executing code during loading, neutralizing deserialization vulnerabilities.
Azure AI Foundry Model Catalog Governance
To mitigate supply chain risks, enterprise architects should avoid downloading unvetted models from arbitrary internet repositories. The Azure AI Foundry Model Catalog provides a secure, governed registry:
Curated Foundation Models: Microsoft tests, validates, and packages models from OpenAI, Meta, Mistral AI, Cohere, and Microsoft Research.
Vulnerability Scanning: Catalog models undergo automated security screening for malicious code, backdoors, and licensing compliance.
Deployment Modalities: Models can be deployed as MaaS (Model as a Service) serverless API endpoints with 99.9% SLAs and automatic patch management, completely eliminating host OS and container management overhead.
Container Security with Microsoft Defender for Containers
When deploying custom agent code, Semantic Kernel Python runtimes, or open-weights models on Azure Kubernetes Service (AKS) or Azure Container Apps (ACA):
Private Hosting: Store all base and application container images in a private Azure Container Registry (ACR) instance with public network access disabled.
Defender for Containers: Enable Microsoft Defender for Containers on the ACR registry. Defender continuously scans every image pushed to ACR against the Common Vulnerabilities and Exposures (CVE) database, flagging out-of-date OS packages, vulnerable Python dependencies, and embedded secrets.
Content Trust and Image Signing: Utilize Notation and Azure Key Vault to cryptographically sign container images. Configure AKS or Container Apps admission controllers to reject any image lacking a valid enterprise signature.
- 4. Real-World Architectural Case Scenario: Autonomous Procurement Agent Compromise
- The Incident
A global manufacturing enterprise deployed an autonomous procurement agent using Semantic Kernel and Copilot Studio. The agent was granted broad database read-write access to create purchase orders (POs) and dispatch vendor payments. During a simulated red-team engagement, an adversary submitted a sequence of ambiguous invoice inquiries paired with adversarial goal hijacking. The agent became trapped in an unconstrained reasoning loop, autonomously chain-invoked the payment tool 14 times, and generated $280,000 in fraudulent vendor disbursements before execution timed out.
Root Cause Analysis (RCA)
Violation of Least Privilege: The procurement tool connector executed under a single enterprise-wide service principal with unrestricted write access to the general ledger.
Absence of State-Mutation Approval Gates: No Human-in-the-Loop verification gate existed for disbursements exceeding standard discretionary thresholds.
Unbounded Execution Recursion: The agent orchestrator was configured without a hard ceiling on planning iterations, enabling the agent to execute an infinite tool-calling loop.
Architectural Remediation Pattern
The lead enterprise architect redesigned the system:
Tool Privilege Segregation: The discovery tool ( lookup_vendor_invoice ) was isolated from the execution tool ( commit_purchase_order ). The execution tool's service principal permissions were revoked and replaced with Entra ID On-Behalf-Of user delegation.
Mandatory HITL Approval: Configured a Power Automate approval workflow. Any PO creation exceeding $5,000 dispatches an Adaptive Card to the department director in Microsoft Teams; the agent suspends execution until positive cryptographic approval is logged.
Execution Budgets: Implemented a hard limit of max_iterations = 4 in the Semantic Kernel execution context and placed Azure API Management in front of Azure OpenAI with a 50,000 TPM quota.
5. Architectural Exam Tips & Implementation Pitfalls
Important
AB-100 Exam Tip: Excessive Agency (LLM06) vs. Prompt Injection (LLM01) Remember that prompt injection is the exploit vector , whereas Excessive Agency is the vulnerability . If an attacker uses prompt injection to force an agent to delete a database, the vulnerability that allowed the catastrophic impact is Excessive Agency (granting the agent delete authority without verification). Address Excessive Agency through architectural tool segregation, least-privilege scoping, and HITL approval gates.
Tip
AB-100 Exam Tip: SafeTensors vs. PyTorch Pickles When an exam question involves deploying open-weights models securely to Azure compute, always choose SafeTensors ( .safetensors ). PyTorch .pt or .bin files rely on Python pickle , which enables arbitrary remote code execution during deserialization. SafeTensors strictly stores tensor arrays and metadata, preventing code execution.
Warning
Frontend Validation Anti-Pattern: Never rely on frontend regex or client-side prompt sanitation to prevent Excessive Agency or prompt injection. Attackers bypass client interfaces by interacting directly with backend APIs. Enforce all security guardrails at the API gateway (Azure API Management and Azure AI Content Safety) and at the tool execution boundary.