Study guide
4.2 Retrieval-Augmented Generation (RAG)
Microsoft Certified: AI Transformation Leader Exam AB-731
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
0.1 Current AB-731 Exam Facts 0.2 Audience Profile and the July 22, 2026 Blueprint 0.3 Independent Study Path for AB-731
2 Generative AI Foundations
1.1 Generative AI Versus Other Types of AI 1.2 Select a Generative AI Solution for a Business Need 1.3 Pretrained Versus Fine-Tuned Models
3 Cost, Risks, and Business Value
2.1 Tokens and ROI Cost Drivers 2.2 Fabrications, Reliability, and Bias 2.3 Scalability, Automation, and When Generative AI Adds Value
4 Prompt Engineering
3.1 Impact of Prompt Engineering 3.2 Prompt Engineering Techniques
5 Grounding, RAG, and Data
4.1 Business Requirements for Grounding Solutions 4.2 Retrieval-Augmented Generation (RAG) 4.3 Data Type, Quality, and Representative Datasets
6 Secure AI and Machine Learning
5.1 Importance of Secure AI 5.2 Application, Data, and Authentication Security for AI 5.3 When Machine Learning Adds Value 5.4 Lifecycle of a Machine Learning Solution
7 Microsoft 365 Copilot and Microsoft Copilot
6.1 Map Business Processes and Use Cases to Copilot 6.2 Differences in Capabilities Between Versions of Copilot 6.3 Microsoft 365 Copilot Chat on Web and Mobile 6.4 Copilot in Microsoft 365 Apps
8 Copilot Studio, Microsoft Graph, and Integrated AI
7.1 Microsoft Copilot Studio Capabilities 7.2 Microsoft Graph Capabilities 7.3 Integrated Microsoft AI, Risk Mitigation, and Safety
9 Agents, Extensibility, and Stack Mapping
8.1 Map Processes to Microsoft AI Apps and Services 8.2 When to Use Researcher or Analyst in Copilot 8.3 Build, Buy, or Extend Copilot
10 Microsoft Foundry and Foundry Tools
9.1 Map Business Processes to Foundry Tools 9.2 Azure Vision, Azure AI Search, and Microsoft Foundry 9.3 Match an AI Model to a Business Need 9.4 Scalability and Security Benefits of Foundry
11 Responsible AI and Governance
10.1 Importance of Responsible AI 10.2 Governance Principles for AI Use 10.3 Establish an AI Council 10.4 Microsoft Responsible AI Standards
12 Adoption Teams, Barriers, and Champions
11.1 Establish an Adoption Team 11.2 Common Barriers to AI Adoption 11.3 Establish an AI Champions Program
13 Impacts, Copilot Licensing, and Foundry Cost Models
12.1 Impacts to Data, Security, Privacy, and Cost 12.2 Copilot License Types 12.3 Foundry Tools Subscription Models
4.2 Retrieval-Augmented Generation (RAG)
Key Takeaways Explain This Topic AI
Retrieval-augmented generation retrieves relevant chunks from an enterprise store, then generates an answer with those chunks in the prompt context.
RAG reduces fabrications versus a closed model because current private facts are supplied at inference time; it does not rewrite the foundation model weights.
Azure AI Search is the common retrieval layer Microsoft Foundry connects to for keyword, vector, hybrid, and, where enabled, agentic retrieval.
Leaders own the data estate and the indexes; AB-731 does not expect them to write embeddings code.
Bad retrieval still yields bad answers, and access control must apply to retrieved documents or RAG becomes a leak.
Last updated: September 2026
How RAG is used for AI solutions
Retrieval-augmented generation (RAG) is a three-step pattern. When a user asks a question, the application retrieves relevant chunks (passages) from an enterprise store, augments the prompt with those chunks, and then generates an answer with the chunks in context. Microsoft Foundry documents this retrieve-then-augment-then-generate flow. The model is still a generator. The difference is that it is generating while looking at your passages, not only at what it memorized in pretraining.
A closed model — pretrained weights only — cannot reliably answer "What is our current parental-leave maximum?" That sentence was never in the public training mix, or it is older than your last policy change. RAG does not rewrite the foundation model. It hands the model the current paragraph from the HR handbook at inference time. That is why fabrications drop: the model has a specific text to stay close to, and you can demand citations to that text.
Fabrications do not go to zero. Microsoft's RAG limitations are blunt. If retrieval returns irrelevant or incomplete passages, the model can still produce incomplete or inaccurate answers despite grounding. Leaders should hear that sentence before they fund a RAG program. Grounding is a quality multiplier for good retrieval. It is not an insurance policy for a messy corpus.
Azure AI Search as the common retrieval layer
Azure AI Search is Microsoft's fully managed retrieval service for classic search and for RAG. You will meet it again in the Foundry Tools chapter. For this skill, remember the leader view: Search hosts indexes — and, in agentic patterns, knowledge sources and knowledge bases — that store chunked enterprise content for keyword, vector, hybrid, and multimodal queries.
Classic search is an index-first, single-request pattern: one query, ranked documents, no model planning during retrieval. Agentic retrieval , also called agentic RAG, uses a model to break a complex question into focused subqueries, run them in parallel, rerank, and return structured grounding data with citations. Some of that surface is generally available. Richer synthesis and related features may still be preview. AB-731 does not ask you to write the pipeline. It asks you to know that Azure AI Search is the common retrieval layer Microsoft Foundry connects to.
Foundry's RAG workflow for a custom app is: prepare and chunk private documents, create an Azure AI Search index, connect the Foundry project to that index, generate with retrieved context, then test and evaluate. Foundry IQ is the managed knowledge layer that turns enterprise content into permission-aware knowledge bases for agents in the Microsoft Foundry portal. Treat that as a product name you will map later. The business object you own now is the data estate feeding the index.
Azure AI Search can pull from supported platforms such as Azure Blob Storage, Azure Cosmos DB, Microsoft SharePoint, and Microsoft OneLake, or accept pushed JSON. It can enrich content at index time with chunking and embeddings. Hybrid search combines full-text precision with vector recall. Those capabilities matter because leaders choose whether the retrieval layer can actually find the paragraph operations will trust.
RAG piece What it is Leader owns
Enterprise store Systems of record: files, tickets, wikis, warehouses Which sources are in scope and who certifies them
- Chunks Passages small enough to retrieve and cite Whether tables, policies, and versions survive chunking
- Index Azure AI Search structure optimized for retrieval Refresh cadence, isolation, and security trimming
Augmented prompt User question plus retrieved grounding data Instructions to stay on retrieved content and cite it
Generation Model produces the answer Evaluation of groundedness before production
Loading diagram... RAG retrieve, augment, and generate with permissioned enterprise chunks
Leader view: data estate and indexes, not embeddings code
Embeddings are numeric representations used for vector similarity search. Someone on the engineering team will choose an embedding model and a chunk size. That is not the Transformation Leader's deliverable. Your deliverables are:
Which systems of record are in scope, and which are excluded.
Who owns quality, access, and the definition of a trusted document.
Whether a shared index is acceptable, or whether isolation requires separate indexes — an HR handbook and a product maintenance manual are a classic split.
How often the index must refresh so inference is not run on stale copies.
Which document-level access control or security filters apply at retrieval time.
How you will measure retrieval quality and groundedness before go-live.
Microsoft Foundry includes groundedness , relevance , and retrieval evaluators so teams can score whether answers stay inside retrieved context. Groundedness asks whether the response fabricates beyond the provided context. Retrieval asks whether the chunks were even relevant to the question. Leaders ask for those scores on representative questions. They do not configure the software development kit.
Fine-tuning is the wrong reflex when the need is fresh private knowledge. Microsoft Foundry guidance is to use RAG when you need answers grounded in private or frequently changing data, and to use fine-tuning when you need to change model behavior, style, or task performance. Buying a larger model also does not replace an index.
Limitations you must budget
Bad retrieval yields bad answers. Chunking that splits a table in half, an index that omits the policy appendix, or a hybrid query that ranks the wrong product line will still produce a fluent wrong answer.
Access control must apply to retrieved documents. Azure AI Search supports document-level access control with security filters, Microsoft Entra ID for production — API keys are for development — and patterns that inherit user permissions from SharePoint or from blob access control lists. If the index is a flattened copy without security trimming, RAG becomes an accidental data-exfiltration engine.
Treat retrieved content as untrusted input. A document can hide a prompt-injection payload. System messages and application logic need to reduce that risk.
Retrieval, embeddings, and extra prompt tokens add cost and latency on top of the model call.
Hallucination despite grounding remains possible. Require citations and instructions to stick to retrieved content.
In practice: A chief financial officer funds a contract question-and-answer bot. Legal discovers the index ingested a shared folder that included a counterparty draft marked Highly Confidential, and that intern accounts could retrieve it. The RAG pattern was correct. The permission design was not. The CFO's correction is to stop go-live until security filters match the source access lists, not to buy a larger model.