ExamFinch.com My Study

Study guide

13.3 Monitor Agents with Application Insights

Practice this exam

Microsoft Certified: AI Agent Builder Associate AB-620

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

1.1 Current Exam Facts 1.2 Candidate Profile & Prerequisites 1.3 Skills Map & Study Strategy

2 Plan Integration, Identity & Channels

2.1 Plan Integration with Enterprise Systems 2.2 Plan Identity Strategy 2.3 Plan Channels and Deployment

3 Responsible AI, Security & Reuse

3.1 Plan Responsible AI Strategy 3.2 Security and Governance Considerations 3.3 Plan Reusable Agent Components 3.4 Design Agents for Internal or External Audiences

4 Agent Flows Fundamentals

4.1 Create an Agent Flow 4.2 Human-in-the-Loop Agent Flows 4.3 Configure Actions and Connectors in Flows

5 Agent Flow Monitoring & Resilience

5.1 Monitor Agent Flows 5.2 Input and Output Parameters 5.3 Error Handling in Agent Flows

6 Topics Configuration Core

6.1 Add Agent Flows to a Topic 6.2 Configure Agent Response Formatting 6.3 Add Tools to a Topic

7 Advanced Topic Responses

7.1 Custom Prompts for Advanced Responses 7.2 Custom Knowledge Sources for Advanced Responses 7.3 API and Send HTTP Request Responses

8 Generative Answers, Cards & Variables

8.1 Configure the Generative Answers Node 8.2 Configure Adaptive Cards 8.3 Manage Variables

9 Enterprise Knowledge Sources

9.1 Connect to Copilot Connectors 9.2 Connect to Power Platform Connectors 9.3 Connect to Azure AI Search

10 Add Tools to Agents

10.1 Configure and Monitor Computer Use 10.2 Configure MCP Tools 10.3 Tools via Existing Custom Connectors 10.4 Add REST APIs to an Agent

11 Multi-Agent Collaboration

11.1 Design Multi-Agent Solutions 11.2 Integrate a Foundry Agent 11.3 Integrate an Existing Copilot Studio Agent

12 Fabric Data Agents & A2A

12.1 Integrate a Fabric Data Agent 12.2 Multi-Agent Solutions with A2A Protocol

13 Integrate Agents with Azure

13.1 Generative Answers with Azure AI Search & Foundry 13.2 Foundry Model Catalog for Custom Prompts 13.3 Monitor Agents with Application Insights

14 Evaluate Agent Performance

14.1 Create a Test Set 14.2 Choose an Evaluation Method 14.3 Review Test Results

15 Application Lifecycle Management

15.1 Create a Solution 15.2 Add Existing Agents to a Solution 15.3 Create and Use Environment Variables 15.4 Power Platform Pipelines for Agents

13.3 Monitor Agents with Application Insights

Key Takeaways Explain This Topic AI

Connect each Copilot Studio agent to Azure Application Insights by pasting the Application Insights connection string under Settings → Advanced.

Optional switches control activity logging, conversation detail (including user identifiers and message text), sensitive activity properties, and node execution events—each with privacy implications.

Telemetry lands in Application Insights tables such as customEvents with rich customDimensions (channel, locale, designMode, text, and more) for Kusto analysis.

Filter designMode to exclude test-canvas traffic when measuring production usage; use the Copilot Studio Dashboard workbook for operational views.

Admins can block the Application Insights in Copilot Studio connector with Power Platform data policies when agent-level telemetry must be disabled by governance.

Last updated: August 2026

13.3 Monitor Agents with Application Insights

Quick Answer: In the agent Settings → Advanced → Application Insights , paste the Azure connection string , enable the logging options your privacy policy allows, then diagnose production behavior with Logs (Kusto) , customEvents/customDimensions , and the Copilot Studio Dashboard workbook.

Building Foundry-backed answers and catalog-powered prompts is incomplete without observability . AB-620’s third Azure integration skill is monitor agents by using Application Insights —agent-level telemetry into Azure Monitor for reliability, latency, topic behavior, and custom diagnostics.

Why Application Insights for agents?

Copilot Studio analytics in the product UI help makers, but enterprise operations teams standardize on Azure Monitor / Application Insights for:

  • Cross-app correlation with APIs, App Services, and Functions the agent calls
  • Kusto (KQL) queries, alerts, and workbooks
  • Longer retention and security-team access patterns
  • Exception and dependency signals when tools fail
  • Custom events authors emit from topics for business KPIs

Think of Application Insights as the production flight recorder for conversational AI—not a replacement for responsible AI evaluation test sets (Domain 3), but the runtime half of quality.

Connection configuration

Microsoft’s agent-level telemetry article describes this maker path:

Create (or reuse) an Application Insights resource in an appropriate Azure subscription and region strategy.

Copy the resource connection string (modern guidance prefers connection strings over legacy instrumentation keys alone).

In Copilot Studio, open the agent Settings page.

Select Advanced .

In Application Insights , paste the Connection string .

Optionally enable detailed logging switches (next section).

Save, then exercise the agent in test canvas and published channels.

In Azure portal → Application Insights → Logs , confirm events arrive.

Telemetry is emitted when users interact—including during testing in Copilot Studio —so empty logs usually mean wrong connection string, blocked connector policy, or queries aimed at the wrong resource/time window.

Logging options and privacy

Microsoft documents optional settings such as:

  • Setting What it does Privacy / ops note
  • Enable logging Logs details of incoming and outgoing messages and events Core on-switch for useful traces

Log conversation details Includes user ID, user name, and message text for message activities; with OpenTelemetry tracing can include tool inputs/outputs in spans High sensitivity—policy review required

Log sensitive Activity properties Includes values of properties that may be sensitive on activities Default off in many guidance docs; enable only with justification

Node execution events Emits an event each time a node in a topic executes Excellent for diagnosing topic graphs; can increase volume

Exam and real-world rule: more logging is not automatically “more mature.” Contoso’s privacy office may require Log conversation details off in production customer channels while still allowing coarse event metrics. Document the decision; do not silently enable full message capture.

What you can diagnose

  • Symptom How App Insights helps
  • Slow replies Latency trends in workbooks; correlate timestamps with tool/HTTP dependencies
  • Topic never fires Missing topic-related events; node execution gaps
  • Tool/action failures Exceptions and custom events around the failing step
  • Channel-specific bugs Filter customDimensions.channelId (for example msteams vs webchat )
  • Test noise in dashboards Exclude designMode == True traffic from the test canvas
  • Adoption metrics Distinct users over time (stronger when users are authenticated with stable IDs)
  • Custom dimensions worth memorizing

Much of Copilot Studio’s detail rides in customDimensions on customEvents :

  • Field Meaning Sample values
  • type Activity type message , event , invoke
  • channelId Channel msteams , webchat , directline , emulator
  • locale Client locale en-us , de-de
  • text Message text (when logged) user utterance
  • designMode Test canvas conversation? True / False
  • fromId / fromName Sender identity fields depends on channel auth

Important caveat: unique user counts are meaningful when identities are stable and authenticated . Anonymous channels that mint a new ID per conversation inflate “users.”

  • Example KQL patterns (conceptual)
  • Distinct users per day (last 14 days)—useful for adoption charts: let queryStartDate = ago(14d);
  • customEvents
  • | where timestamp > queryStartDate
  • | summarize uc=dcount(user_Id) by bin(timestamp, 1d)
  • | render timechart
  • Exclude test canvas traffic: customEvents
  • | extend isDesignMode = customDimensions['designMode']
  • | where isDesignMode == "False"

Start investigations with customEvents then narrow by time, channel, and event name. Learn documents that a simple table query is valid—complexity is optional.

Copilot Studio Dashboard workbook (preview)

Application Insights can show a Copilot Studio Dashboard workbook under Monitoring → Workbooks . It consolidates conversations, latency, exceptions, tool usage, and topic analytics. Teams can Edit the workbook, add KQL tiles for custom attributes, Save variants, and Share with colleagues who have at least Reader on the Application Insights resource.

Use the workbook for stand-ups; use raw Logs for deep incident response.

Environment-level telemetry and product analytics

Microsoft also documents broader telemetry stories (environment-level Application Insights preview, Power Platform export scenarios, and product analytics). For AB-620’s wording— monitor agents by using Application Insights —prioritize agent-level connection string configuration , logging toggles, custom dimensions, and diagnostic queries. Mention environment-level options as supplemental, not a substitute for knowing the agent Advanced settings path.

Governance: disabling agent-level telemetry

To prevent makers from enabling agent-level Application Insights telemetry, admins can block the Application Insights in Copilot Studio connector via a data policy in the Power Platform admin center:

Security → Data and privacy → Data policy.

Create or edit a policy; add target environments.

Find Application Insights in Copilot Studio .

Move it to Blocked .

Publish.

When enforced, makers cannot use the connector in governed environments. Exam stems that say “compliance forbids exporting utterances to Azure” often want DLP block , not “delete the agent.”

Scenario — diagnosing a slow Foundry prompt path

Contoso’s claims agent spiked to 12-second responses after a model catalog change. Ops opens Application Insights:

Workbook latency tile confirms the regression after Tuesday 16:00 UTC.

Logs filter designMode == False and channelId == "msteams" .

Node execution events show long gaps around the custom prompt tool.

Correlation with Azure metrics on the Foundry deployment shows throttling.

Team either scales the deployment or switches the high-volume prompt to a mini model (Section 13.2 tradeoffs).

Without App Insights, the team would only hear “Teams is slow” with no evidence.

Scenario — false alarm from test traffic

A dashboard shows thousands of daily “users,” but production publish is limited to a pilot group. Querying without filtering designMode counted maker test canvas sessions. After excluding design mode, real pilot usage matches expectations. Always separate lab traffic from production analytics.

Pitfalls

Pasting the wrong resource’s connection string.

Enabling full conversation text logging without privacy approval.

Alerting on test-canvas noise.

Assuming anonymous channels yield trustworthy unique user counts.

Forgetting Reader permissions when sharing workbooks.

Ignoring DLP when “settings refuse to save” telemetry.

Using App Insights alone as a substitute for formal evaluation test sets.

Success checklist

Configure Settings → Advanced → Application Insights connection string.

Explain each logging toggle and its sensitivity.

Query customEvents and extend customDimensions (especially designMode , channelId ).

Use workbooks for health overviews and KQL for incidents.

Diagnose latency/errors by correlating topic/node/tool signals.

Govern telemetry with data policies when required.

With Search+Foundry generative answers, Foundry model catalog prompts, and Application Insights monitoring fluent, you can design, power, and operate Azure-integrated Copilot Studio agents for AB-620’s Domain 2 Azure section end to end.