Study guide
12.2 Create Prompt Variants and Compare Performance
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
12.2 Create Prompt Variants and Compare Performance
Key Takeaways Explain This Topic AI
In prompt flow (retiring April 20, 2027; Microsoft points new work to Microsoft Agent Framework), a variant is a version of an LLM tool node with different prompt text or connection settings such as temperature. In Foundry, the same idea is a new agent version. Change one factor per variant or version; do not edit production and hope you remember the old wording.
Microsoft’s tuning guide requires a controlled experiment: change one variable at a time (system wording, few-shot examples, or temperature — not all three), and when a flow has variants on multiple nodes, batch-run one node’s variants while other nodes stay on their default.
Single-row playground eyeballing is not comparison. Submit a batch run against a representative evaluation dataset and score with metrics. Classification samples use accuracy; RAG and agents use Domain 4 quality evaluators such as groundedness (1–5), relevance, coherence, and fluency.
Compare variants offline on evaluation datasets (and optional limited A/B on non-production traffic) before promoting a winner. Set the winning variant as the node default, or ship the winning Foundry agent version. Do not decide with production users as the first test set.
Foundry evaluation runs (and, in existing flows, prompt flow's Visualize outputs view) let you inspect per-row predictions beside metrics. A higher fluency score with a drop in groundedness is a regression, not a win.
Last updated: October 2026
Create Prompt Variants and Compare Performance
Quick Answer: A variant is one LLM-node (or agent-instruction) candidate. Change one factor at a time — system text, few-shot examples, or temperature — then batch-run every candidate on the same evaluation dataset and score groundedness , relevance , coherence , fluency , or task accuracy. Promote the winner. Do not A/B on production traffic first.
The second Domain 3 bullet is create prompt variants and compare performance across different prompts . The outline names no tool. Microsoft's prompt-flow documentation describes variants in the most detail, but prompt flow retires on April 20, 2027 and is not recommended for new development, so new Foundry work applies the same idea to agent versions and evaluation runs.
What a variant is
In prompt flow, a variant is a specific version of a tool node with distinct settings. Variants are supported on the LLM tool . A new variant can change prompt content , connection settings (temperature, model deployment), or both. The first node you see is variant_0 and is the default . You Show variants , Clone to variant_1 , edit, and Hide variants when you are done. All variants stay folded; the default is what a normal run uses.
Microsoft’s news-summary example makes the factorial obvious:
- Variant Prompt Connection
- Variant 0 Summary: {{input sentences}} Temperature = 1
- Variant 1 Summary: {{input sentences}} Temperature = 0.7
- Variant 2 What is the main point of this article? {{input sentences}} Temperature = 1
- Variant 3 What is the main point of this article? {{input sentences}} Temperature = 0.7
That table is a teaching grid, not a recommended experiment. If you ship all four at once you cannot tell whether wording or temperature moved the metric. The controlled experiment rule is: change one thing.
The Web Classification sample in the tuning guide is the exam-shaped walkthrough. Starting from variant_0 on classify_with_llm , they create variant_1 with temperature 1 → 0 (classification wants precision) and variant_2 with temperature 0 plus few-shot examples . On summarize_text_content they clone a variant that asks for 300 words instead of 100 . They do not flip summary length, few-shot, and temperature in a single clone.
On Foundry agents the same discipline applies with files: v1_instructions.txt , v2_instructions.txt (few-shot added), v3_instructions.txt (grounding/citation clause added). create_version() mints agent:1 , agent:2 , agent:3 . You still change one factor per version if you want a defensible comparison.
Compare with evaluators, not vibes
Microsoft’s own warning: running variants on a few rows and judging with the naked eye cannot reflect real-world diversity, and the output is not measurable . The fix is a Foundry evaluation run (or, in an existing prompt flow, a batch run from the Evaluate wizard).
Prompt-flow batch evaluation (existing flows only):
Prepare a dataset that represents production (for classification, URLs plus ground-truth labels).
Select one node whose variants you will sweep. Other LLM nodes stay on their default variant so you do not confound two knobs.
Map columns: ground truth from the dataset, prediction from a flow output.
Pick an evaluation method. Classification uses Classification Accuracy Evaluation . RAG and generation use quality evaluators.
After the run, Visualize outputs to see metrics per variant and the per-row predictions.
Set the winner as the node default (or deploy that agent version).
Domain 4 — covered in Chapter 13 — is the metric catalog you attach now:
Groundedness — is the answer supported by retrieved context? Model-based groundedness scores 1–5 . Groundedness Pro (preview) is a binary pass/fail from Azure AI Content Safety and does not need a judge deployment.
Relevance — does the answer address the query?
Coherence — logical flow.
Fluency — readable language.
Retrieval / Document Retrieval — did the retriever surface the right chunks (a retrieval problem, not a wording problem)?
Risk and safety evaluators (hate, sexual, violence, self-harm, protected materials, indirect attack / XPIA ) — a “friendlier” variant that fails safety is not a promotion candidate.
Agent evaluators such as tool call accuracy , task adherence , and intent resolution when the prompt’s job is to drive tools.
Microsoft’s combining guidance for RAG is Retrieval + Groundedness + Relevance + Content Safety . A variant that wins fluency and loses groundedness is a regression. Do not average unrelated metrics into one vanity score unless you documented a rubric with weights.
A/B testing in this chapter means: score variant A and variant B on the same held-out evaluation dataset (and, later, a small non-production or canary slice). Domain 5’s RAG A/B frameworks are for retrieval strategy. Here you are A/B-ing prompts . You do not start by sending 50 percent of members a rewritten system prompt because someone liked it in the playground.
When a flow has variants on two nodes, you cannot run both sweeps in one batch. Microsoft documents that each run selects one LLM node with variants; other nodes use defaults. You sweep classify_with_llm , lock the winner, then sweep summarize_text_content . That is the same one-factor rule at graph scale.
Exam scenario
You maintain a prompt-flow chat flow that drafts appeal letters from retrieved policy text. variant_0 is a short system prompt at temperature 0.7. Legal wants firmer citations. You clone variant_1 that only adds “quote the retrieved clause and append [doc_id] ” — same temperature, same few-shot. You batch-run 400 labeled appeals. Groundedness mean moves 3.1 → 4.4; relevance is flat; fluency dips 0.2. Safety evaluators still pass. You set variant_1 as default, commit the Jinja2 file, and open a pull request with the eval HTML attached. You do not also drop temperature to 0 in that same clone, because then you could not tell which change helped.
Common trap
The trap is declaring a winner from five playground chats. A sibling trap is changing system text, few-shot, and temperature in one variant, then attributing the lift to “better prompting.” A third is promoting a variant that raised fluency while groundedness and XPIA scores got worse. A fourth is running production traffic as the first A/B because “the eval set is small” — grow the eval set (Chapter 13) instead of using members as guinea pigs.
Variants are controlled experiments with numbers. The next section puts those files in Git so the winner is the commit you can roll back.