Study guide
13.4 Built-in and Custom Automated Evaluation Workflows
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
13.4 Built-in and Custom Automated Evaluation Workflows
Key Takeaways Explain This Topic AI
Domain 4 ends this cluster with automated evaluation workflows using built-in and custom metrics. Built-in evaluators live in the project evaluator catalog (builtin.*). Custom evaluators are code-based (grade() returns 0.0–1.0), prompt-based (LLM judge with ordinal, continuous, or binary result), or endpoint-based (your HTTP scorer).
Cloud evaluation in Microsoft Foundry (evals.create then evals.runs.create, or the portal wizard) is the default for scale and CI. Local azure-ai-evaluation evaluate() is for classic/dev-box spotting and can log to the project. Cloud supports dataset, model/agent targets, traces, synthetic, and red team sources.
microsoft/ai-agent-evals GitHub Action (v3-beta) runs catalog evaluators against Foundry agents from a data file. Authenticate with Azure Login (OIDC). Do not run the full suite on every commit. Fail the job when scores miss your thresholds.
Code-based custom evaluators sandbox at under 256 KB, two minutes, no network. Prompt-based custom judges need deployment_name and threshold. Endpoint-based scorers need a project connection and a 30-second JSON contract.
This chapter is eval design and the pre-production/CI run. Continuous evaluation of sampled production traffic and the monitoring dashboard are Chapter 14 — do not treat a one-shot cloud run as production observability.
Last updated: August 2026
Built-in and Custom Automated Evaluation Workflows
Quick Answer: Use built-in catalog evaluators plus optional custom code , prompt , or endpoint scorers. Run cloud evaluation in Microsoft Foundry for scale and CI ; use the local Azure AI Evaluation SDK evaluate() for classic/dev loops. Gate a GitHub Action ( microsoft/ai-agent-evals ) on thresholds. Continuous evaluation of live traffic is Chapter 14 .
The last Domain 4 bullet in this cluster is set up automated evaluation workflows by using built-in and custom evaluation metrics . You now have a mapped dataset, quality metrics, and safety metrics. Automation is how those runs happen on every release candidate without a human clicking the portal.
Built-in catalog versus custom scorers
Built-in evaluators are builtin.coherence , builtin.groundedness , builtin.violence , and the rest of the catalog. They already know their inputs. You add them as testing_criteria of type azure_ai_evaluator with data_mapping and, when required, initialization_parameters .
Custom evaluators exist when the catalog cannot express your rule: JSON schema validity, "must cite a policy ID," brand tone, disclaimer present. Microsoft documents three types:
Type How it scores Best for Contract
Code-based Python grade(sample, item) -> float in 0.0–1.0 Format, length, keyword, regex, deterministic checks Sandbox: under 256 KB, 2 min, no network , 2 GB RAM; packages such as numpy, pandas, jsonschema, rouge-score
Prompt-based Judge prompt with {{response}} (and other fields) Tone, rubric, semantic rules the catalog lacks JSON {result, reason} ; ordinal, continuous, or binary; needs deployment_name and threshold
Endpoint-based POST to your HTTP API Proprietary models, network, logic over the sandbox limits JSON with score , status , optional passed ; 30 s timeout; ApiKey or Entra ID connection
Register custom evaluators in the project evaluator catalog (portal Evaluation → Evaluator catalog → Custom evaluator , or project_client.beta.evaluators.create_version ). Then reference them in a run the same way as builtin.* . Code-based APIs still require a deployment_name in init parameters even though grade() does not call GPT — the orchestration schema expects it. If grade() throws or times out, that item scores 0.0 and is marked an error; write defensive try/except .
Do not write a custom judge that reimplements groundedness poorly. Use built-in groundedness, then a code evaluator that asserts "policy_id" appears in the JSON.
Cloud evaluation versus local SDK
Cloud evaluation is the current Foundry path: openai_client.evals.create (schema + testing criteria) then evals.runs.create (data source). Results live on the project. The portal wizard is the same pipeline with clicks: target (agent / model / dataset / traces), scope (turns vs conversations), data, mapping, criteria, submit. Status values include In Progress, Completed, Partial, Failed. Use cloud when you want scale, RBAC, CI, model/agent targets , synthetic or red-team generation, or App Insights traces . Role on the project: Foundry User (the renamed Azure AI User).
Local evaluation ( pip install azure-ai-evaluation , evaluate(data=..., evaluators=..., evaluator_config=column_mapping) ) runs on your machine or a pipeline agent. It still needs JSONL and column mapping. You can pass azure_ai_project to log the studio URL. Local is the classic SDK loop and still appears in hubs/prompt-flow shops. AI-300 expects you to know both : cloud for Foundry GenAIOps, local for a dev-box or an air-gapped debug. Cloud is what Microsoft tells you to use for most pre-deployment and CI scenarios so you are not babysitting compute.
Data sources you should recognize for automation:
Versioned JSONL/CSV ( file_id ) — golden set in CI.
Target completions — hit azure_ai_agent or azure_ai_model with {{item.query}} .
Agent response IDs — file_content only.
Traces — production-adjacent; more Chapter 14.
Synthetic / red team — bootstrap or adversarial, not the only gate.
GitHub Action as a quality gate
Microsoft publishes microsoft/ai-agent-evals (documented as v3-beta) to evaluate Foundry agents in GitHub Actions. Inputs: azure-ai-project-endpoint , deployment-name (judge), data-path (JSON with name , evaluators , data rows), agent-ids as name:version . Optional baseline-agent-id for pairwise tests with confidence intervals. Authenticate with azure/login and OIDC ( id-token: write ). The action invokes the agent, runs catalog (and custom) evaluators, and writes a summary report.
Treat the job as a gate : fail the workflow if groundedness mean is below 3, if violence defect rate exceeds policy, or if the pairwise test says the new agent is worse than baseline. That is "automated evaluation workflow" in exam language.
Microsoft’s own tip: do not run evaluation on every commit — judge tokens and agent calls cost money. Typical triggers: pull request to main , a workflow_dispatch before a release, or a nightly on the golden set. Path filters so a README change does not spend GPT.
You can also call the cloud SDK from a workflow step (Azure CLI login, then Python evals.runs.create ) if you need dataset evaluation without the agent-evals action. Same rule: threshold → exit code .
What this chapter is not
Continuous evaluation samples production agent responses on a schedule and feeds the monitoring dashboard. That is Chapter 14 (observability: continuous monitoring, latency, tokens, tracing). A cloud run against claims-eval/3 in CI is pre-deployment . Mixing them on the exam — "we turned on continuous eval so we can skip the golden set" — is the trap.
A complete automated loop for this chapter looks like: Git tracks the JSONL golden set and custom grade() source → PR or release workflow logs into Azure → cloud evaluation or ai-agent-evals runs built-in quality + safety + custom schema check → job fails on threshold → you do not deploy the agent version. After it is in production, Chapter 14 watches live scores.
Exam scenario
Contoso versions agent claims-triage:7 in Git. The workflow on pull request to main checks out the repo, azure/login@v2 with federated credentials, and runs microsoft/ai-agent-evals@v3-beta with agent-ids: claims-triage:7 , data file listing builtin.groundedness , builtin.relevance , builtin.violence , builtin.hate_unfairness , and custom policy_json_valid . Pass thresholds: groundedness ≥ 3, violence defect rate 0 on the adversarial slice, custom code score ≥ 0.99. The previous version is baseline-agent-id . A prompt-only change that lifts fluency but drops groundedness fails the Action . Nobody clicks Submit in the portal on Friday night as the release process.
Common trap
Calling the playground "CI." A second trap is a custom prompt evaluator that duplicates groundedness with a vaguer rubric. A third is a code evaluator that calls out to the public internet (sandbox has no network ) or exceeds 2 minutes. A fourth is running ai-agent-evals on every push of a markdown fix. A fifth is treating a successful local notebook evaluate() on 8 rows as the production gate, with no thresholds in GitHub. A sixth is configuring continuous production sampling and skipping this chapter’s pre-deploy run.