Study guide
15.4 RAG Relevance Metrics and A/B Testing
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
15.4 RAG Relevance Metrics and A/B Testing
Key Takeaways Explain This Topic AI
Evaluate retrieval separately from generation. A fluent answer can still be ungrounded if the index returned the wrong chunks.
With labeled qrels, Microsoft Foundry’s Document Retrieval evaluator reports Fidelity, nDCG, XDCG, Max Relevance, and Holes for parameter sweeps (k, chunk size, hybrid versus vector, semantic ranker on or off).
Without labels, the Retrieval evaluator (LLM-judge, 1–5) scores whether retrieved context could answer the query. After generation, use Groundedness (precision versus context) and Relevance (answer versus query).
A/B one retrieval parameter at a time, hold the generator, prompt, and temperature constant, and score a frozen labeled eval set—not vibes from the playground.
Do not A/B retrieval changes on 100 percent of production traffic without a holdout. Keep a control index or query template and a rollback git tag.
Last updated: August 2026
Loading diagram... Separate retrieval evaluation from generation, then A/B one retrieval lever
RAG Relevance Metrics and A/B Testing
Quick Answer: Split RAG evaluation into process (did Azure AI Search retrieve the right chunks?) and system (did the Microsoft Foundry model answer from those chunks?). Use Document Retrieval metrics ( nDCG , Fidelity , Holes ) on labeled qrels to tune k, chunking, and hybrid settings. Then score groundedness and relevance . A/B one retrieval parameter , hold the generator constant, and never ship a new index to all production traffic without a holdout .
Domain 5 closes RAG with evaluate and improve RAG with relevance metrics and A/B testing frameworks . Domain 4 already taught quality evaluators; this section is about using them as a retrieval tuning loop , not as a one-off demo.
Two layers: retrieval versus generation
If the retriever misses the policy section, the chat model will either abstain, guess, or quote a nearby but wrong paragraph. Groundedness can look decent if the model faithfully repeats the wrong chunk; relevance to the user still fails. Conversely, a perfect retriever plus a sloppy prompt yields ungrounded prose. Microsoft Foundry therefore ships process evaluators (retrieval quality) and system evaluators (final answer quality).
Evaluator Layer You need What it tells you
Document Retrieval Process retrieval_ground_truth (qrels with labels) + retrieved_documents Fidelity , nDCG , XDCG , Max Relevance , Holes
Retrieval Process query + context (no labels) LLM-judge 1–5 whether chunks could answer the query
Groundedness System response + context (query recommended) Precision: no fabricated claims beyond context (1–5, default pass ≥ 3)
- Groundedness Pro (preview) System query, response, context Strict boolean via Azure AI Content Safety
- Relevance System query + response Does the answer address the question? (1–5)
- Response Completeness (preview) System ground_truth + response Recall versus the expected answer
Recall@k (fraction of relevant documents appearing in the top k) and nDCG@k (normalized discounted cumulative gain—are the best documents ranked highest?) are the classic IR names you should still be able to explain. Foundry’s Document Retrieval evaluator is how you compute a bundle of those ideas without hand-rolling ranking code. Holes counts retrieved IDs that lack a judgment; a high hole ratio means your qrels do not cover what the index is returning, so nDCG is untrustworthy until you label more.
Groundedness is the precision of the answer against retrieved context. Response completeness is the recall of the answer against ground truth. Do not collapse them into a single “accuracy” number on the exam.
Build a labeled eval set
You cannot A/B chunk sizes in the playground. You need:
A frozen list of real queries (production traces, support tickets, plus adversarial SKUs and paraphrases).
For each query, qrels : document or chunk IDs with a relevance label (Foundry’s document retrieval sample uses labels on a scale such as 1–5; configure ground_truth_label_min / max to match).
Optional answer ground truth for completeness and for spot-checking generation.
A mapping file that records which index, embedding deployment, k, searchMode, and semantic configuration produced each run.
Keep this dataset in Git next to the skillset. When the corpus changes, refresh labels; do not keep scoring against dead document IDs (those become Holes ).
Microsoft documents a parameter sweep : generate retrieval results for candidate settings (vector vs hybrid vs hybrid+semantic, several k values, chunk sizes), then run builtin.document_retrieval to see which setting wins on nDCG and Fidelity. That sweep is the scientific version of “tweak k in Search Explorer until it looks good.”
A/B testing rules for RAG
A retrieval A/B is an experiment with one independent variable :
Hold constant: generator model and version, prompt/instructions, temperature, safety filters, and the eval query set.
Change one retrieval lever: chunk size, overlap, embedding model (full re-index), hybrid on/off, semantic ranker on/off, k, vector threshold, searchMode, maxTextRecallSize .
Score both legs with Document Retrieval (and Retrieval if you lack full qrels), then run generation once per retrieved context and score groundedness/relevance.
Promote only if retrieval metrics improve and answer-level groundedness does not regress. A higher nDCG that stuffs 40 chunks into the prompt can still wreck groundedness via distractors.
Roll forward with a holdout. Send 5–20 percent of production traffic (or a canary index alias) to the candidate. Keep the control index and a git tag. If groundedness or incident rate worsens, flip the alias back.
Do not :
A/B the embedding model and the prompt and the GPT deployment in the same week and call the winner “hybrid search.”
Use production users as unlabeled judges without logging retrieved IDs—you will not know whether retrieval or generation broke.
Tune on the same 20 queries you overfit in Search Explorer; keep a held-out query split.
Flip 100 percent of traffic to a rebuilt index on Friday afternoon with no control.
Exam scenario: nDCG@10 is 0.41 on vector-only k = 10. You stand up index-B with the same 3-small embeddings but hybrid + semantic ranker, k = 50, top = 8 chunks to the model. On the frozen 150-query qrel set, Document Retrieval shows nDCG@10 = 0.63 and fewer Holes. Groundedness stays at a mean of 4.1 because you still send only eight chunks. You route 10 percent of the Foundry agent’s live queries to index-B for a week, watch Foundry quality monitors, then switch the alias. You did not change GPT-4o-mini in the same experiment.
Common trap: Declaring victory because the playground answer “sounds better” after enabling semantic ranker, with no qrels and with a new system prompt in the same session. A second trap: A/B testing on production with both variants writing to the same unversioned index, so you cannot roll back. A third: optimizing only fluency (a Domain 4 quality metric) while retrieval Fidelity is near zero—the model is eloquently guessing.
Operationalize the loop in GitHub Actions the same way you evaluate prompts (Chapter 12): on a pull request that changes the skillset, chunk parameters, or query template, run the eval dataset, fail the PR if nDCG or groundedness drops past a gate, and require a human review of Holes. That is GenAIOps for RAG, and it is what Exam AI-300 is testing when it pairs relevance metrics with A/B testing frameworks.