Study guide
10.2 Telemetry Data Interpretation, Performance Metrics & Latency Tuning
Microsoft Certified: Agentic AI Business Solutions Architect Expert AB-100
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Requirements Analysis & Grounding Data
1.1 Assessing Agents for Task Automation, Analytics & Decision-Making 1.2 Grounding Data Evaluation: Accuracy, Relevance, Timeliness, Cleanliness & Availability 1.3 Structuring Enterprise Business Data for AI System Interoperability
2 Chapter 2: AI Strategy, Cloud Adoption Framework & CoE
2.1 Cloud Adoption Framework (CAF) for Azure AI Strategy & Roadmap 2.2 Multi-Agent Architecture: Microsoft 365 Copilot, Copilot Studio & Foundry 2.3 Microsoft AI Center of Excellence (CoE) & Solution Governance 2.4 Developing Use Cases for Prebuilt Agents & the Microsoft Agent Catalog 2.5 Solution Rules & Constraints for Copilot Studio, Microsoft Foundry & Foundry Tools
3 Chapter 3: Model Selection, Prompt Engineering & Dynamics AI
3.1 Model Selection Strategy, Custom Models & Small Language Models (SLMs) 3.2 Enterprise Prompt Engineering Guidelines & Prompt Libraries 3.3 Cross-App AI Strategy for Multi-App Dynamics 365 Solutions
4 Chapter 4: ROI, TCO, Build vs Buy & Model Routing
4.1 ROI Criteria & Total Cost of Ownership (TCO) for AI Solutions 4.2 Comprehensive Financial ROI Analysis for Business Processes 4.3 Build vs Buy vs Extend Architectural Decision Framework 4.4 Dynamic Model Router Architecture & Cost-Performance Optimization
5 Chapter 5: Dynamics 365 AI & Contact Center Design
5.1 Business Terms & Copilot Customization in Customer Experience and Service 5.2 Connectors & Data Flow Design for Copilot in Dynamics 365 Sales 5.3 Agent Integration with Dynamics 365 Contact Center Channels
6 Chapter 6: Agent Archetypes & Copilot Studio Core
6.1 Designing Task Agents, Autonomous Agents & Prompt-and-Response Agents 6.2 Copilot Studio Topic Design, Trigger Phrases & Graceful Fallbacks 6.3 Orchestration Selection: Standard NLP vs CLU vs Generative AI Orchestration
7 Chapter 7: Copilot Studio Advanced Actions & Power Platform
7.1 Designing Agent Flows & Prompt Actions in Copilot Studio 7.2 Power Apps Canvas App Integration, Generative Pages & Agent Feeds 7.3 Power Platform Well-Architected Framework for AI Workloads 7.4 Proposing Microsoft Power Platform AI Features & the AI Hub
8 Chapter 8: Agent Extensibility, MCP & Computer Use
8.1 Microsoft Foundry Custom Models & Foundry Tools Selection 8.2 Agent Extensibility via Model Context Protocol (MCP) in Copilot Studio 8.3 Computer Use Automation & Web UI Interactions in Copilot Studio
9 Chapter 9: Advanced Behaviors & Prebuilt App Orchestration
9.1 Agent Behaviors: Multi-Step Reasoning, Voice Mode & M365 Agent Optimization 9.2 Orchestrating Prebuilt AI in Dynamics 365 F&O and M365 Copilot for Sales/Service 9.3 Orchestrating Prebuilt Copilot AI Features in Dynamics 365 Customer Experience & Service 9.4 Proposing Microsoft 365 Agents for Business Scenarios 9.5 Adding Knowledge Sources to In-App Help & Guidance in Dynamics 365 Finance & Supply Chain
10 Chapter 10: Telemetry, Monitoring & Agent Tuning
10.1 Monitoring Architectures: Azure Monitor, Application Insights & Studio Analytics 10.2 Telemetry Data Interpretation, Performance Metrics & Latency Tuning 10.3 User Feedback Analysis, Backlog Prioritization & Continuous Improvement
11 Chapter 11: Testing Strategy & Model Validation
11.1 Agent Testing Frameworks, Quality Metrics & Groundedness Benchmarks 11.2 Custom AI Model Validation, Safety Criteria & Regression Testing 11.3 Copilot-Assisted Test Case Generation & Cross-Dynamics End-to-End Testing 11.4 Validating Effective Copilot Prompt Practices
12 Chapter 12: ALM for Grounding Data & Copilot Studio Agents
12.1 ALM for Grounding Data, Vector Indexes & Knowledge Synchronization 12.2 ALM for Copilot Studio Agents, Connectors & Solution Packaging 12.3 Environment Strategy & Automated Deployment Pipelines for Studio Agents
13 Chapter 13: ALM for Foundry Agents & Dynamics 365 AI
13.1 ALM for Microsoft Foundry Agents Service & Model Registries 13.2 ALM for Custom AI Models, Fine-Tuning Artifacts & Deployment Slots 13.3 ALM for Dynamics 365 AI in Finance, Supply Chain & Customer Engagement
14 Chapter 14: Agent Identity, Security & Access Controls
14.1 Agent Identity Architecture, Microsoft Entra ID & Authentication Flows 14.2 Access Controls, Role-Based Security & Permissions on Grounding Data 14.3 Secure Data Processing & Grounding Data Isolation
15 Chapter 15: Model Security & Threat Mitigation
15.1 Model Security Architecture & Defense Against Vulnerabilities 15.2 Mitigating Prompt Manipulation, Jailbreaks & Indirect Prompt Injections 15.3 Azure AI Content Safety, Prompt Shields & Guardrails
16 Chapter 16: Responsible AI, Governance, Residency & Audit
16.1 Microsoft Responsible AI Standard: Principles, Assessment & Implementation 16.2 Data Residency, Sovereign Boundaries & Cross-Border Compliance 16.3 Comprehensive Audit Trails, Model Decision Logging & Microsoft Purview 16.4 Designing Governance for Agents: Environment Strategy, Data Policies & Agent Inventory
10.2 Telemetry Data Interpretation, Performance Metrics & Latency Tuning
Key Takeaways Explain This Topic AI
End-to-end agent response latency is a composite metric governed by network transit, orchestration overhead, prefill Time-to-First-Token (TTFT), decoding Tokens-per-Second (TPS), backend tool execution, and content safety evaluations.
Isolating latency bottlenecks requires analyzing Application Insights dependency records to distinguish between foundation model inference delays and slow external dependencies such as unindexed Dataverse lookups or unoptimized REST connectors.
Token economics monitoring tracks prompt tokens, completion tokens, and cached tokens to establish per-session cost profiles and determine whether Provisioned Throughput Units (PTU) or Pay-As-You-Go deployment models are optimal.
Latency tuning patterns include context window pruning, system prompt minification, semantic response caching via Azure Cache for Redis / APIM, and hierarchical routing that delegates classification tasks to fast Small Language Models (Phi-3.5/Phi-4).
Production agent drift manifests as hallucination drift (grounding degradation), semantic drift (shifting user vocabulary), and topic trigger drift (intent misrouting), each requiring telemetry-driven detection and mitigation.
Last updated: September 2026
Telemetry Data Interpretation, Performance Metrics & Latency Tuning
Quick Answer: Optimizing agent performance requires decomposing end-to-end latency into distinct phases: Time-to-First-Token (TTFT / prefill), generation throughput (Tokens-per-Second / TPS), backend tool execution, and safety guardrail overhead. When telemetry reveals tool bottlenecks (e.g., slow Dataverse queries or unindexed REST endpoints), architects must optimize backend dependencies before tuning prompts. To slash model latency and token costs, solutions architects implement Azure OpenAI prompt caching, context window pruning, semantic response caching with Azure Cache for Redis, and hierarchical routing that offloads intent classification from heavy frontier models (GPT-4o) to fast Small Language Models (Phi-3.5 / Phi-4).
In generative and agentic systems, responsiveness is a primary determinant of user adoption and operational success. While a traditional database lookup returns in under 100 milliseconds, an autonomous agent executing dynamic multi-step reasoning may consume several seconds orchestrating prompts, retrieving vector chunks, querying enterprise connectors, and generating streaming tokens.
Solutions architects must interpret telemetry streams to diagnose performance bottlenecks, optimize token economics, and detect operational drift over time.
1. Core Performance Metrics in Agentic Systems
To diagnose latency anomalies, architects must move beyond generic "round-trip duration" and decompose agent response times into granular operational stages. [ End-to-End Latency ]
- +-----------------------------------------------------------------------------------------+
- | T_network | T_orchestrate | T_TTFT | T_decode | T_tool | T_safety |
- | Ingress / | Framework / | Prefill Phase: | Autoregressive | Connector/ | Guardrail|
- | Egress | Context Prep | Queue + Prompt Eval | Generation | Database | Filtering|
- +-----------------------------------------------------------------------------------------+
- 1.1 Time-to-First-Token (TTFT) vs. Tokens-per-Second (TPS)
Language model execution consists of two distinct mathematical and architectural phases:
Time-to-First-Token (TTFT) — Prefill Phase:
Definition: The elapsed time from when the model server receives the input prompt payload to when it emits the very first generated token.
Mechanics: The model processes all input prompt tokens in parallel to construct the initial key-value (KV) attention cache. TTFT is heavily influenced by the input prompt length (system prompt + conversation history + retrieved grounding context + tool schemas) and server queueing delay.
Perceptual Significance: TTFT dictates perceived responsiveness. In text chat, a TTFT under 1,500ms feels responsive if token streaming is enabled. In telephony and voice channels, TTFT must remain sub-800ms to prevent dead air and conversational overlap.
Tokens-per-Second (TPS) — Decoding Phase:
Definition: The throughput speed at which the model generates output tokens sequentially.
Mechanics: Because generation is autoregressive (each token depends on all preceding tokens), decoding cannot be parallelized across the generated sequence. TPS is dictated by model parameter size, GPU memory bandwidth, quantization level (e.g., FP16 vs INT4), and GPU cluster load.
Perceptual Significance: Human reading speed averages 4 to 6 words per second (approximately 6 to 9 tokens per second). A generation speed of 20 to 30 TPS provides a fluid, natural reading experience.
1.2 End-to-End Latency Decomposition Formula
Solutions architects model total response duration using the following decomposition formula: Total Latency = T network + T orchestration + T TTFT + ( N output tokens TPS ) + ∑ T tools + T safety \text{Total Latency} = T_{\text{network}} + T_{\text{orchestration}} + T_{\text{TTFT}} + \left(\frac{N_{\text{output tokens}}}{\text{TPS}}\right) + \sum T_{\text{tools}} + T_{\text{safety}} Total Latency = T network + T orchestration + T TTFT + ( TPS N output tokens ) + ∑ T tools + T safety
1.3 Latency Component Breakdown & Performance Budgets
Component Target SLA Primary Root Cause of Degradation Key Telemetry Metric
Network Ingress / Egress < 150 ms Geographic distance between client, Copilot Studio tenant, and Azure region. requests.duration vs dependencies.duration
Agent Orchestration < 250 ms Heavy expression evaluations, complex topic condition logic, state variable parsing. Custom event timestamps in customEvents
Model TTFT (Prefill) < 1,200 ms Massive context windows (e.g., 20k+ unpruned tokens), cold model instances, high server queueing. Azure OpenAI TimeToFirstTokenMs metric
Model Generation (TPS) > 25 tokens/s Model parameter bloat, compute saturation on standard Pay-As-You-Go deployments. Azure OpenAI TokensPerSecond metric
Backend Tools / Connectors < 1,000 ms Unindexed Dataverse tables, slow external REST APIs, sequential Power Automate loops. dependencies.duration where type == 'HTTP'
Content Safety Guardrails < 200 ms Deep multi-modal safety evaluation, custom regex and blocklist scanning. dependencies.duration for Content Safety calls
2. Token Economics & Cost Telemetry
Enterprise agents operate under strict budgetary governance. Solutions architects must track token consumption with high precision to prevent unexpected cost overruns and size Azure infrastructure appropriately.
2.1 Token Telemetry Metrics: Input, Output, and Cached
Every interaction with a foundation model emits token usage metadata recorded in Application Insights dependency records and Azure OpenAI diagnostic logs:
Input Prompt Tokens ( prompt_tokens ): The sum of tokens in the system prompt, dynamic few-shot examples, tool definitions (JSON schemas), past conversation turns, and retrieved grounding text.
Output Completion Tokens ( completion_tokens ): Tokens generated by the model in its response or tool call argument payload. Output tokens are significantly more expensive than input tokens (typically 3x to 4x higher per thousand tokens).
Cached Prompt Tokens ( cached_tokens ): Tokens reused directly from the Azure OpenAI Prompt Cache , billed at a 50% discount and processed with near-zero prefill latency. [ Total Session Cost ] =
- (Sum(Prompt Tokens - Cached Tokens) * Cost_Prompt) +
- (Sum(Cached Tokens) * Cost_Cached) +
- (Sum(Completion Tokens) * Cost_Completion) +
- (Tool Connector Costs)
- 2.2 Azure OpenAI Prompt Caching Economics
Azure OpenAI automatically enables prompt caching on supported model deployments (such as GPT-4o and GPT-4o-mini). Understanding how prompt caching operates allows architects to deliberately structure system prompts for maximum cost savings:
Cache Eligibility Threshold: The model checks for matching prompt prefixes. The prefix must be at least 1,024 tokens long.
Exact Match Requirement: Caching requires exact character-level matching from the start of the prompt. Dynamic variables (e.g., CurrentTime = 10:14 AM or UserId = 8840 ) must never be placed at the top of the system prompt. Placing dynamic variables at the beginning invalidates the prefix cache for every turn.
Architectural Placement: Structure prompts hierarchically: (1) Static System Persona & Operational Rules -> (2) Static Tool Schemas -> (3) Grounding Documentation -> (4) Dynamic Conversation History -> (5) Current User Query. +-------------------------------------------------------------+
- | 1. Static System Persona & Global Constraints (2,000 tokens) |
- | [ CACHED - 50% DISCOUNT & SUB-100MS PREFILL ] |
- +-------------------------------------------------------------+
- | 2. Static Tool JSON Schemas (1,500 tokens) |
- | [ CACHED - 50% DISCOUNT & SUB-100MS PREFILL ] |
- +-------------------------------------------------------------+
- | 3. Dynamic Conversation History (Variable 800 tokens) |
- | [ UNCACHED - Evaluated at runtime ] |
- +-------------------------------------------------------------+
- | 4. User Query & Timestamp (150 tokens) |
- | [ UNCACHED - Evaluated at runtime ] |
- +-------------------------------------------------------------+
- 2.3 Sizing Infrastructure: Provisioned Throughput Units (PTU) vs. Pay-As-You-Go
- Architectural Factor Pay-As-You-Go (Consumption) Provisioned Throughput Units (PTU)
- Billing Model Per 1,000 input/output tokens Fixed hourly/monthly reservation per PTU allocated
Throughput & Capacity Shared multi-tenant pool; subject to throttling (HTTP 429) during regional peak demand Dedicated compute capacity; guarantees deterministic throughput and zero noisy-neighbor contention
Latency Profile Variable TTFT based on regional cluster queueing Highly deterministic and consistent sub-second TTFT
Economic Crossover Cost-effective for low, sporadic, or unpredictable workloads (< 5 million tokens/day) Cost-effective for high, sustained volume (> 15-20 million tokens/day per deployed model)
Burst Behavior Hard TPM (Tokens Per Minute) and RPM (Requests Per Minute) quotas Supports short-duration bursting above provisioned limits with spillover queues
3. Diagnostic Patterns for Latency Tuning
When telemetry in Application Insights highlights degraded performance, solutions architects apply proven architectural intervention patterns.
3.1 Remediating Backend Tool Bottlenecks
Telemetry frequently reveals that the foundation model is not the bottleneck; rather, downstream tools and connectors dominate execution duration:
Sequential Connector Execution: When an agent invokes three Power Automate flows or REST actions sequentially, latency compounds ( 1.5 s + 2.0 s + 1.8 s = 5.3 s 1.5s + 2.0s + 1.8s = 5.3s 1.5 s + 2.0 s + 1.8 s = 5.3 s ).
Remediation: Refactor the agent orchestration or backend flow to execute independent data retrievals concurrently in parallel using asynchronous branches.
Unindexed Dataverse Queries: An agent querying Dataverse for a customer record by email address without an alternate key or index on the emailaddress1 column forces a full table scan.
Remediation: Create Dataverse Alternate Keys or single-column indexes on all filtered attributes.
OData Over-fetching: Connectors that retrieve entire entity records with 80+ attributes bloat payload size and processing time.
Remediation: Enforce explicit OData $select clauses, retrieving only the 3 or 4 attributes required for conversational grounding.
3.2 Streamlined Prompt Design & Context Pruning
As multi-turn conversations progress, unpruned conversational history accumulates, inflating the context window and driving up TTFT:
Sliding Window Pruning: Retain only the most recent K K K turns (e.g., last 4 turns). Drop earlier turns from the active prompt payload.
Conversational Summarization: When conversation depth exceeds 6 turns, trigger a lightweight background task to summarize turns 1 through 5 into a concise 100-word paragraph, passing the summary alongside the last turn.
System Prompt Minification: Eliminate redundant conversational filler, verbose explanatory prose, and decorative markdown tables from system instructions. System prompts should be concise, declarative, and structured.
Intermediate Tool Scrubbing: Once a tool payload has been processed and relevant parameters extracted (e.g., account_status = 'Active' ), strip the raw 200-line JSON payload from conversational history. Never resubmit obsolete raw JSON in subsequent turns.
3.3 Hierarchical Model Selection: Offloading to Small Language Models (SLMs)
Deploying a massive frontier model (such as GPT-4o) for every agent operation is an architectural anti-pattern that creates high latency and unnecessary expense.
Hierarchical Routing Architecture: Use lightweight, high-speed Small Language Models (SLMs) such as Phi-3.5 or Phi-4 for routine classification and routing, reserving frontier LLMs exclusively for complex multi-step reasoning and synthesis. [ Incoming User Query ]
- |
- v
- +-------------------------------------------------+
- | High-Speed Triage Layer: Phi-3.5 / Phi-4 |
- | - Intent Classification |
- | - PII / Redaction Check |
- | - Complexity Scoring (Latency: < 150 ms) |
- +-------------------------------------------------+
- |
- +------------------+------------------+
- | |
- v (Simple Query / FAQ) v (Complex Reasoning)
- +-------------------------+ +-------------------------+
- | Fast Path: SLM / Cache | | Deep Path: GPT-4o |
- | - Phi-4 Direct Response | | - Multi-Step ReAct Plan |
- | - Static Knowledge Base | | - Tool Invocation Loop |
- | - Latency: < 400 ms | | - Latency: 2.5s - 6.0s |
- +-------------------------+ +-------------------------+
- Capability Fast SLM Layer (Phi-3.5 / Phi-4) Frontier LLM Layer (GPT-4o)
- Parameter Scale 3.8B to 14B parameters Hundreds of billions (MoE architecture)
- Average TTFT 120 ms - 250 ms 600 ms - 1,500 ms
- Generation Throughput 60 - 100+ tokens/s 25 - 40 tokens/s
- Cost per 1M Tokens ~$0.10 - $0.30 ~$2.50 - $10.00
Optimal Tasks Intent classification, entity extraction, sentiment detection, query routing Ambiguous goal decomposition, multi-tool orchestration, complex code generation
3.4 Semantic Response Caching Architecture
In enterprise customer service and internal IT helpdesk scenarios, 30% to 50% of incoming inquiries are semantic variations of the same core questions (e.g., "How do I reset my corporate password?" vs "Where do I go to change my login password?").
Rather than invoking the language model for every repeated inquiry, architects implement Semantic Caching using Azure Cache for Redis (with RediSearch vector similarity) or Azure API Management (APIM) LLM caching policies: [ Inbound User Query ]
- |
- v
- [ Generate Query Vector Embedding ] (text-embedding-3-small, ~20ms)
- |
- v
- [ Search Vector Cache in Redis ] (Cosine Similarity Search, ~15ms)
- |
- +----+--------------------------------+
- | |
- v (Similarity >= 0.96) v (Similarity < 0.96)
- [ Cache Hit! ] [ Cache Miss ]
Return cached answer payload. Execute Model / Agent Flow.
- Latency: < 50 ms - Latency: 2,500 ms - 5,000 ms
- Token Cost: $0.00 - Token Cost: Full price
- |
- v
- Store Query Embedding &
- Response in Redis (TTL = 24h)
- 4. Detecting and Mitigating Agent Drift
An agent that performs flawlessly in pilot testing can degrade significantly in production over time. This degradation is known as drift . Telemetry must be configured to continuously monitor for three distinct forms of drift: [ Forms of Agent Drift ]
- +-----------------------------------+-----------------------------------+
- | | |
- v v v
- [ Hallucination Drift ] [ Semantic Drift ] [ Topic Trigger Drift ]
- Knowledge sources change - User phrasing changes - New topics collide with
- Retrieval chunk truncation - Industry slang evolves existing trigger phrases
- Groundedness score drops - Vector similarity drops - Generative routing misfires
- 4.1 Hallucination Drift
Symptom: The agent begins generating factually incorrect, unsubstantiated, or fabricated statements that lack grounding in enterprise documentation.
Root Cause: Enterprise source documents (e.g., HR policy manuals) are updated in SharePoint, but the Azure AI Search index fails to synchronize; or chunking boundaries bisect critical tabular data, preventing the retrieval engine from providing complete context.
Telemetry Detection: In Azure AI Foundry, configure scheduled evaluation jobs on production sample transcripts measuring the Groundedness / Faithfulness Score (measuring the proportion of claims in the generated response that can be directly inferred from the retrieved grounding context). A drop in weekly groundedness below 0.85 triggers an alert.
4.2 Semantic Drift
Symptom: Retrieval quality degrades; the agent repeatedly falls back to generic responses because it cannot find relevant knowledge chunks.
Root Cause: User vocabulary, organizational abbreviations, or customer terminology shifts away from the vocabulary embedded in source documents.
Telemetry Detection: Track the average cosine similarity score of top- K K K retrieved chunks in Application Insights dependencies records. If average similarity drops over a 30-day window, source documents must be augmented with user synonym dictionaries and acronym taxonomies.
4.3 Topic Trigger Drift
Symptom: In Copilot Studio, inquiries intended for a specific topic (e.g., Hardware Replacement ) begin misrouting to an unrelated topic (e.g., Office Relocation ).
Root Cause: Makers publish new topics whose trigger phrases semantically collide with existing topics, or generative orchestration intent classification misinterprets ambiguous multi-intent prompts.
Telemetry Detection: Review the Topic Trigger Confusion Matrix in Log Analytics by analyzing sessions where the user immediately escalates or abandons after a topic trigger. Remediate by defining explicit negative trigger phrases and configuring disambiguation nodes.
Tip
AB-100 Exam Tip: Exact-Match Caching vs. Semantic Caching Traditional HTTP caching (exact string matching on URL or body hash) is almost useless for conversational AI because natural language queries vary continuously ("reset password" vs "forgot my password"). The AB-100 exam expects architects to recognize Semantic Caching —which computes a high-speed vector embedding of the user query and performs cosine similarity matching against a Redis cache—as the correct pattern to eliminate redundant LLM calls and reduce latency from seconds to milliseconds.
Important
AB-100 Exam Tip: Structuring Prompts for Azure OpenAI Prompt Caching To achieve the 50% discount and sub-100ms TTFT from Azure OpenAI prompt caching, two architectural conditions must be met: (1) The static prompt prefix must contain at least 1,024 tokens , and (2) dynamic variables (such as timestamps, user IDs, or real-time location tags) must be placed at the very end of the prompt payload, never at the beginning. Placing dynamic variables at the top breaks the deterministic prefix match and invalidates the cache for all users.