Study guide
14.1 Continuous Monitoring in Microsoft Foundry
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
14.1 Continuous Monitoring in Microsoft Foundry
Key Takeaways Explain This Topic AI
Microsoft Foundry observability is three pillars: evaluation, production monitoring, and OpenTelemetry tracing stored in Azure Monitor Application Insights.
Post-production monitoring combines operational metrics, continuous evaluation of sampled live traffic, scheduled evaluation on test datasets, scheduled red teaming, and Azure Monitor alerts.
Continuous evaluation is not an Azure Machine Learning tabular data-drift monitor. Foundry scores GenAI quality and safety over time; Chapter 9 monitors compare feature distributions.
Sample production traffic. AI-assisted judges are consumption-billed. The SDK continuous rule default max_hourly_runs is 100; skipped runs usually mean the hourly cap, not a broken agent.
Continuous evaluation rules require the project managed identity to have the Foundry User role (formerly Azure AI User). Empty dashboards usually mean no traffic, a bad time range, or ingestion delay.
Last updated: August 2026
Continuous Monitoring in Microsoft Foundry
Quick Answer: Continuous monitoring in Microsoft Foundry samples production agent traffic , runs quality and safety evaluators as responses complete or on a schedule , and charts the scores next to latency, tokens, and errors on the Agent Monitoring Dashboard . Telemetry lives in the project's Azure Monitor Application Insights resource. This is not an Azure Machine Learning tabular data-drift monitor.
Domain 4 asks you to examine continuous monitoring in Foundry . Chapter 13 covered pre-production evaluation: golden datasets, groundedness, relevance, coherence, fluency, and risk/safety evaluators. This chapter is what you attach after the agent is live. Microsoft's observability concept page (updated 2026-07-31) names three pillars that work together: evaluation , monitoring , and tracing . You already built evaluators. Monitoring is how those evaluators, plus operational metrics, stay attached to real users . (Microsoft Foundry is the current brand for what many materials still call Azure AI Studio or Azure AI Foundry.)
Why Foundry monitoring is not an Azure Machine Learning model monitor
Chapter 9 taught Azure Machine Learning production monitors: a scheduled Spark job that compares tabular production model inputs (and optionally predictions) against a reference distribution. Signals are data drift , prediction drift , and data quality . A threshold breach can email you or start a retraining pipeline .
Generative AI rarely fails that way first. A retrieval-augmented generation (RAG) agent can keep the same embedding space and still start hallucinating , citing stale chunks , or emitting unsafe completions because a prompt version changed, a tool started returning empty JSON, or a model deployment rolled forward. Foundry continuous monitoring therefore scores outputs (and traces) for quality and safety over time , not feature histograms. If an exam item mentions population stability or feature distributions , that is Chapter 9. If it mentions sampled production traffic , groundedness on live traces , or the Agent Monitoring Dashboard , that is this section.
- Signal Azure Machine Learning (Chapter 9) Microsoft Foundry (this chapter)
- Primary object Tabular features and predictions from an endpoint Agent runs, prompts, completions, tool calls
- Typical job Scheduled Spark model monitor Continuous or scheduled evaluation rules plus dashboards
- Drift meaning Input or prediction distribution shift Quality/safety score shift on live or golden traffic
- Store modelDataCollector in workspace blob Application Insights (OpenTelemetry traces and metrics)
Built-in response Email / Event Grid, then retrain Azure Monitor alerts on latency, tokens, eval scores, red-team findings
The post-production loop
Microsoft's post-production list is the exam checklist. After deployment you:
Measure operational metrics (token usage, latency, run success rate, errors).
Run continuous evaluation : quality and safety on production traffic at a sampled rate .
Run scheduled evaluation : the same evaluators against a test dataset on a cron so you detect system drift even if today's users ask easy questions.
Run scheduled red teaming : adversarial scans for leakage and prohibited actions.
Fire Azure Monitor alerts when outputs miss quality thresholds or produce harmful content.
Those five are complementary. Continuous evaluation tells you what users are experiencing now . Scheduled evaluation on a held-out golden set tells you whether the system (prompt, index, tools, model) drifted even if the traffic mix changed. Red teaming probes safety that ordinary users might never trigger. Operational metrics catch timeouts and 429s that evaluators never see because the run never completed.
Continuous evaluation versus scheduled evaluation
The Monitor tab on a Foundry agent (New Foundry portal: Build → agent → Monitor ) is the operator view. The gear icon opens Monitor settings . Two recurring evaluation types matter:
Continuous evaluation samples live traffic as it occurs . In the SDK this is an evaluation rule with event type RESPONSE_COMPLETED , an EvaluationRuleFilter on agent_name , and a ContinuousEvaluationRuleAction that points at an eval definition. max_hourly_runs caps how many sampled runs are judged per hour. Microsoft documents a default of 100 and skipped runs when the cap is hit.
Scheduled evaluation uses a recurrence trigger (example: daily at 09:00) and a trace or dataset data source. Use it for benchmarks and drift against a fixed set.
The portal wizard also lets you pick Live traffic versus a Dataset , a turn level , and Random or Intelligent sampling , plus a maximum number of traces . Classic Foundry continuous evaluation exposed samplingPercent (0–100) and maxRequestRate with a system limit of 1,000 requests per hour . Do not memorize a single percent as "the official sampling rate." The exam cares that you sample rather than score every production call, and that you cap hourly judge traffic.
To create continuous rules, assign the project managed identity the Foundry User role (formerly named Azure AI User; role IDs are unchanged). Without that assignment, rule creation can look fine in your user context and then silently skip .
Sampling rate versus the cost of judge models
AI-assisted evaluators (groundedness, relevance, violence, protected material, and most custom large language model judges) call another model . Observability features such as risk/safety evaluations and agents playground evaluations are consumption-billed on the Foundry Observability meter. Playground evaluations are on by default ; turn them off in the playground metrics panel if you are iterating and do not want to pay for every try.
Sampling is how you stay solvent:
100 percent sampling on a busy agent plus five LLM judges can cost more than the production model.
Too low a rate hides a bad prompt that only hits a small slice of traffic.
Prefer higher sampling on safety evaluators than on expensive quality judges if budget is tight.
Intelligent sampling (portal) tries to spend the hourly budget on interesting traces rather than duplicates.
When runs are skipped , raise max_hourly_runs or lower sampling. Raising the cap without watching the judge deployment's tokens-per-minute limit just moves the 429s to the evaluator.
Dashboard metrics you must be able to read
Microsoft's Agent Monitoring Dashboard definitions (docs dated 2026-08-08):
Token usage : counts for the selected window. Spikes often mean verbose prompts or long completions .
Latency : run response time. Values above about 10 seconds warrant a look at throttling , complex tool calls , or network .
Run success rate : completed runs. Below about 95 percent is an investigation trigger, not a product pass/fail baked into every SKU.
Evaluation metrics : scores from sampled outputs; interpretation is evaluator-specific (Chapter 13).
Red teaming results : failed scans are security work, not "the model is slow."
Empty charts usually mean no recent traffic , a time range that excludes data, or ingestion delay — generate traffic and wait a few minutes before assuming the connection is broken. Authorization errors mean missing Log Analytics Reader (and Privileged Monitoring Data Reader if tables are protected ).
You can also monitor custom agents that do not run in Foundry: register them in the control plane, emit OpenTelemetry generative-AI semantic conventions , and send traces to the same Application Insights instance. Continuous evaluation then has a data plane to sample.
Scenario
A benefits chatbot goes live Monday. Groundedness on the Chapter 13 golden set was 0.86. By Thursday the Monitor chart shows groundedness on sampled live traffic at 0.61 while the nightly scheduled golden-set eval is still 0.85. The agent is not "drifting" in the tabular sense. Users started asking about a policy PDF that is not in the index . Continuous evaluation caught the production mix ; scheduled evaluation proved the prompt and model still pass the old tests. The fix is retrieval coverage , not a Spark data-drift job and not an immediate model swap.
Common trap
Do not turn on 100 percent continuous evaluation with a large judge ensemble "for safety" and then blame the production deployment for 429s and latency . Judge tokens share quota and cost . Sample. Cap hourly runs. Keep scheduled golden-set evals for regression. Do not use an Azure Machine Learning data collector plus model monitor as a substitute for Foundry continuous evaluation — it will never score groundedness .
Operator checklist
Connect Application Insights before you expect charts or traces.
Use continuous eval for live mix; scheduled eval for golden-set drift; red team on a schedule for adversarial coverage.
Sample; set max_hourly_runs ; watch judge spend.
Grant Foundry User to the project managed identity .
Alert on eval score, latency, tokens, and red-team findings — not only on HTTP 500s.