Study guide
13.1 Test Datasets and Data Mapping for Evaluation
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
13.1 Test Datasets and Data Mapping for Evaluation
Key Takeaways Explain This Topic AI
Exam AI-300 Domain 4 (10–15%) starts with create test datasets and data mapping for comprehensive model evaluation. A row is not 'some CSV of chats' — it is query, optional context, optional ground_truth, and response, mapped to the fields each evaluator requires.
Microsoft Foundry portal field mapping and SDK data_mapping use {{item.field}} for dataset columns and {{sample.output_text}} (or {{sample.output_items}}) when a model or agent target generates the response at run time. If a required field is Unassigned, that evaluator fails.
Upload JSONL (turn-level or conversation messages) or CSV (turn-level only) as a versioned project dataset. Column names do not have to be query; map {{item.question}} when the file uses question.
Coverage beats raw size: intents, languages, empty-retrieval and adversarial rows, plus a frozen golden set and a sampled production set. Judge-model cost scales with rows × evaluators, so size the batch to quota, not to 'as many as the lake has.'
For RAG, concatenate retrieved chunks into one context string with a separator such as a blank line. Groundedness cannot score a response against documents it never sees.
Last updated: August 2026
Test Datasets and Data Mapping for Evaluation
Quick Answer: A Microsoft Foundry evaluation dataset is a versioned JSONL or CSV of query , optional context , optional ground_truth , and response . Data mapping ( {{item.field}} in the SDK, field mapping in the portal) binds those columns to each evaluator. Cover intents , languages , and adversarial rows. Keep a frozen golden set and a production sample . Size the set against judge-model cost , not against how many logs you can dump.
Domain 4 of Exam AI-300 — Implement generative AI quality assurance and observability (10–15%) — opens with create test datasets and data mapping for comprehensive model evaluation . Domain 3 already versioned prompts and agents. This chapter is how you prove a change is better before it ships. Chapter 14 covers continuous monitoring of live traffic. This section is the offline design of the test set those evaluators will score.
Why an evaluation dataset is not production logs
Playground eyeballing of three happy-path questions is not evaluation. Neither is pasting last week's Application Insights dump into a spreadsheet and calling it a test. An evaluation dataset is a controlled sample with a schema the evaluators understand, labels (or at least known intents), and a version you can rerun after a prompt or model change.
Microsoft Foundry accepts:
Existing dataset — you already have queries and, often, responses. Score them as a dataset evaluation without calling the model again.
Model or agent target — you have queries (and maybe context / ground truth). Foundry generates responses at evaluation time, then scores them. Map the live output with {{sample.output_text}} .
Synthetic queries (preview) — you have almost nothing; the service generates queries from a prompt and optional seed file, then hits a target.
Traces / existing conversations — later in the flywheel; still need mapping so query and response come out of the span or message array.
JSONL is the workhorse: one JSON object per line, turn-level rows or a messages array for conversation-level runs. CSV is turn-level only . Upload with project_client.datasets.upload_file (name + version) so CI reuses file_id instead of attaching a one-off blob. Inline file_content is for tiny experiments and for agent response-ID evaluation, which does not accept file_id .
The four columns evaluators actually consume
Built-in evaluators do not guess your business names. They look for query , response , context , and ground_truth (plus agent extras such as tool_calls and tool_definitions , and conversation messages ).
Column What it is Who needs it
query The user (or test) prompt Relevance, coherence, safety, similarity, retrieval; required in portal single-turn mapping
response The model or agent output Almost every quality and safety evaluator
context Retrieved grounding text (chunks concatenated) Groundedness, Groundedness Pro, retrieval, ungrounded attributes
ground_truth Human or expert expected answer Similarity, F1, BLEU/GLEU/ROUGE/METEOR, response completeness
A query is the stimulus. A response is what you are scoring. Context is the retrieved evidence the RAG or tool path supplied — not the entire knowledge base and not the system prompt. Ground truth is the answer a reviewer would accept, used for overlap and completeness metrics, not as a substitute for context.
For multi-chunk retrieval, Microsoft documents concatenating chunks into one string with a separator such as a blank line between passages. Do not leave context as an array of objects unless you have written a custom evaluator that knows that shape. For conversation-level runs the portal maps messages (chat format) and, for agents, tool_definitions .
Data mapping: teaching the evaluator which column is which
The portal Evaluation wizard (Step 5 when you use existing data) auto-maps columns that already match the expected names. If a field shows Unassigned , you pick the dataset column from a dropdown. Required fields are marked with an asterisk. Evaluators fail if required fields stay unassigned. That is the exam trap in UI clothing.
The cloud SDK is explicit: always include data_mapping with the inputs each evaluator lists. Syntax:
{{item.field_name}} — a column on the dataset row (or item schema).
{{sample.output_text}} — text the model or agent target just generated.
{{sample.output_items}} — structured agent output including tool calls.
If your JSONL uses question and answer because that is how the product team exports tickets, you do not rename the lake. You map query to {{item.question}} and response to {{item.answer}} . Local classic SDK evaluate() uses a similar idea as column_mapping with ${data.queries} style placeholders. New cloud runs use the {{item.*}} / {{sample.*}} pair. Either way, mapping is the contract , not the filename.
When the run generates responses, do not map response to a stale {{item.response}} column that is empty or from last week's model. Map it to {{sample.output_text}} and set include_sample_schema on the data source config so the sample fields exist.
Coverage: intents, languages, and adversarial rows
A 2,000-row file of near-duplicate "what is the refund window?" questions is not comprehensive. Design coverage on purpose:
Intents / skills — every tool, every policy chapter, every "I don't know" path, and the top support tickets. Tag a column such as intent for slicing even if evaluators ignore it.
Languages and locales — if the agent serves en-US and es-MX , both belong in the set. Coherence and fluency currently document English-language scoring; do not pretend an English judge is a multilingual certification.
Retrieval extremes — empty context, contradictory chunks, and the correct chunk buried at position k. Groundedness on empty context should not invent policy.
Adversarial / red-team — jailbreak phrasing, indirect injection hidden in a retrieved document, copyrighted-lyric bait, and self-harm probes. These rows exist so risk and safety evaluators have something to fail. Keep them in a restricted dataset version; they are not playground demos.
Schema / tool rows — missing plan_code , malformed JSON, and "call the refund tool" cases if you will score tool accuracy later.
Synthetic generation (portal Synthetic data, or azure_ai_synthetic_data_gen_preview ) is a bootstrap when you have no queries, not a replacement for a reviewed golden set. Seed it with a document or prompt that describes your domain. Red-team evaluation ( azure_ai_red_team ) generates adversarial prompts on purpose; that is a safety dataset, not a fluency dataset.
Golden set versus production sample, and size versus cost
A golden set is a frozen, human-reviewed batch — often 50–200 rows for an associate-scale product, more if you have labeled qrels for document retrieval. You change it only in a pull request. Rerunning the same golden set after a prompt variant is how Chapter 12's comparison becomes a number instead of a vibe.
A production sample is a time-boxed draw from real queries (or traces). It catches drift the golden set never imagined. It is not the golden set: labels may be missing, PII may be present, and you must sample rather than score the entire lake. Intelligent sampling of traces (Chapter 14) is the production-side cousin; here you still curate a file.
Size versus cost: AI-assisted quality evaluators call a judge deployment once per row per metric. Microsoft's portal troubleshooting says the judge counts against Azure OpenAI quota, that you should shrink the set or use a cheaper judge such as gpt-4.1-mini / gpt-5-mini when quota binds, and that GitHub's evaluation action should not run on every commit. A 5,000-row set × four LLM judges is a bill and a timeout, not a badge. Start with a few dozen rows to validate mapping, then scale the golden set to the intents you must cover.
Version the file ( eval-claims-golden/3 ). Record which agent version and prompt file the responses came from. Dataset evaluation of precomputed outputs is cheaper than regenerating against a target every run; regenerate when the question is "does this agent version still pass?"
Exam scenario
Contoso Claims has a Foundry agent over an Azure AI Search index. The product manager emails a 40,000-row export with columns ticket_id , member_question , kb_snippets , agent_reply , and adjuster_answer . You upload JSONL version claims-eval/1 . In the portal you map query → member_question , context → kb_snippets (chunks already joined), response → agent_reply , ground_truth → adjuster_answer . You add 30 adversarial rows and 20 Spanish intents in claims-eval/2 . You freeze 120 reviewed rows as the golden set and sample 80 production tickets per week into a separate version. You do not map groundedness until kb_snippets is populated.
Common trap
Leaving query or response Unassigned because the CSV header says question / output , then blaming Foundry when every evaluator errors. A second trap is scoring groundedness without context (or with the system prompt stuffed in as context). A third is treating the golden set as a dump of production logs — PII, no intents, no adversarial cases, and a cost bomb for the judge model. A fourth is mapping {{item.response}} on a target run that is supposed to generate {{sample.output_text}} .