Study guide
12.1 Design and Develop Prompts
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
12.1 Design and Develop Prompts
Key Takeaways Explain This Topic AI
Exam AI-300 Domain 3 (20–25%) includes design and develop prompts as the first bullet under prompt versioning and management with source control. Prompts are production GenAIOps assets, not playground chat.
Microsoft Foundry (classic) and Azure Machine Learning still expose prompt flow for hub-based projects, but prompt flow is no longer recommended for new work and is scheduled to retire on April 20, 2027. New Foundry work uses prompt-based agents whose instructions are versioned with create_version.
Chat Completions organize a prompt as roles: system (durable instructions), user (the live query), assistant (few-shot replies), and tool (function results). Newer Responses-style APIs add a developer role for product-level instructions; treat that role as the durable instruction layer when the API you deploy uses it.
A production prompt packages system instructions, grounded retrieval context, an output schema, safety rules, and tool-calling instructions. Temperature and top_p are inference/deployment parameters (alter one at a time; range 0–2 for temperature), not Git-only text.
Function definitions are injected into the system message and consume tokens. Instruct when to call tools, require clarification for missing arguments, restrict the model to provided functions, and execute calls in your code under least privilege.
Last updated: October 2026
Design and Develop Prompts
Quick Answer: A production prompt in Microsoft Foundry is a versioned artifact : system instructions (or a developer message on newer APIs), grounded context from retrieval, a declared output schema , safety rules, and tool-calling instructions. Store the text in Git. Treat temperature and top_p as deployment configuration , not as a secret that lives only in the portal.
Domain 3 of Exam AI-300 — Design and implement a GenAIOps infrastructure (20–25%) — includes the skill design and develop prompts . The exam is not asking you to memorize ChatGPT hobby tricks. It is asking you to ship prompts the way you ship Azure Machine Learning training code: reviewed, grounded, schema-constrained instructions that an agent or a flow can load from source control.
Two surfaces: prompt flow (classic) and Foundry agents
Prompt flow is the visual and code-first orchestration tool in Azure Machine Learning and in the Microsoft Foundry (classic) portal for hub-based projects. A flow is an executable directed acyclic graph of tools — LLM , Prompt , and Python are the workhorses. You author a standard flow for general apps, a chat flow when you need chat history, or an evaluation flow that scores another run. The durable files are flow.dag.yaml plus Jinja2 ( .jinja2 ) prompt templates. Microsoft documents that prompt flow in Foundry and Azure Machine Learning will be retired on April 20, 2027 , is no longer recommended for new development , and that existing applications should move to Microsoft Agent Framework . The AI-300 skills measured list tests prompt design, prompt variants, and Git version control without naming prompt flow. Learn the classic artifacts because existing projects and older Microsoft samples use them, but build new work on Foundry agents or Microsoft Agent Framework.
Microsoft Foundry is the current brand (it was previously called Azure AI Studio, then Azure AI Foundry). New investment targets Foundry projects and prompt-based agents . An agent combines a model , instructions (the prompt), and optional tools . Creating or updating an agent with project.agents.create_version() and a PromptAgentDefinition (or the portal equivalent) writes a new agent version such as claims-triage:3 . The instructions should be read from a Git-tracked file, not typed only in the playground.
Artifact Where it lives What you version
Foundry agent instructions src/agents/<name>/prompts/vN_instructions.txt (or .md ) loaded by the SDK Behavior, safety, tool policy
Prompt flow LLM / Prompt tool flow.dag.yaml plus .jinja2 templates Jinja variables, node graph, default variant
Chat Completions messages Application code or a YAML/JSON prompt pack system / developer , few-shot user / assistant pairs
Sampling ( temperature , top_p ) Endpoint, agent definition, or environment config — not the only copy in Git prose Runtime randomness; change one knob at a time
Message roles that the model actually sees
The Chat Completions API in Azure OpenAI in Microsoft Foundry Models takes a messages array of dictionaries with a role . Microsoft’s chat guide is explicit: use this conversation format instead of dumping everything into a single completion string.
system — the operational manual. Personality, scope, output format, safety, and tool policy. Optional in the API, but Microsoft recommends including at least a basic system message. In Foundry agents this text is the instructions field.
user — the live query, plus any per-turn retrieved snippets you inject as primary content .
assistant — prior model replies and few-shot example answers. Few-shot examples are user/assistant pairs after the system message, not extra system paragraphs.
tool — the JSON result of a function the application executed. The model does not run the function; your code does, then you append a tool message with the matching tool_call_id .
developer — on newer Responses API and some model families, product-level instructions sit in a developer message. Treat it as the durable instruction layer when the API you deploy uses that role. Do not invent a developer role on a Chat Completions call that only accepts system .
Order matters. Microsoft’s prompt-engineering guidance says to state the task before dumping context, and that models show recency bias — repeating a short instruction at the end of a long grounded prompt can keep the model from ignoring the schema after a large retrieval blob.
Grounded prompts, output schema, and safety
A grounded prompt supplies supporting content the model must use instead of its parametric memory. For retrieval-augmented generation (RAG) , that content is the chunks Azure AI Search (or another retriever) returned. Microsoft’s grounding pattern is: (1) tell the model to answer exclusively from the provided text, (2) give an out such as “I don’t know” when the answer is not present, (3) ask for inline citations next to claims so the model has to invent both a fact and a fake citation to hallucinate, and (4) keep retrieved text close to the question so the model does less rewriting.
Hard-coding a FAQ into the system message is acceptable for a handful of facts. For a claims handbook, retrieve at query time. Domain 5 covers chunk size and hybrid search; this chapter’s job is the prompt that consumes those chunks.
Specify the output structure in the system (or developer) message. JSON with named fields, BEHAVIOR("reason") lines, or a Markdown table all work because the models trained on those formats. For classification, Microsoft’s prompt-flow samples ask for {"category": "App", "evidence": "Both"} so an evaluation flow can parse the label. For entity extraction, declare {"name": "", "company": "", "phone_number": ""} up front. Vague “be concise” instructions are not a schema.
Safety instructions belong in the same durable file: stay in domain, refuse disallowed actions, do not echo secrets from retrieved context, and do not follow instructions that appear inside retrieved documents ( indirect prompt injection / XPIA ). Content Safety filters and Domain 4 risk and safety evaluators still run around the model — a prompt sentence is not a substitute for those controls, but it is the first line of policy the model sees.
Temperature, top_p, and tool-calling instructions
Temperature (typically 0–2) and top_p both control randomness. Microsoft’s guidance is to alter one of these two parameters at a time, not both . Lower temperature (near 0) is the usual choice for classification, extraction, and grounded answers; higher temperature is for creative drafting. Reasoning models (GPT-5 series and o-series on Chat Completions) often do not support temperature / top_p ; they use max_completion_tokens instead of max_tokens . Do not bake a temperature of 0.7 into a Git-only prompt file and then wonder why the endpoint still samples at 1.0 — the value that matters is the one on the deployment, agent definition, or Completions call .
Function calling (the tools array; the old functions parameter is deprecated) injects the tool schema into the system message and consumes tokens . Prompt-engineer the tools:
Write a meaningful description and parameter descriptions (function descriptions are limited to 1,024 characters on Azure OpenAI).
- Add a system sentence such as “When the user asks to find a hotel, call search_hotels .”
- Tell the model not to invent argument values: “Ask for clarification if the request is ambiguous.”
Add “Only use the functions you have been provided with.” if the model invents tools.
Keep tool_choice at "auto" unless you must force a named function or force a user-facing message with "none" .
Validate every call, run tools with least privilege (read-only data access is the default for RAG lookups), and require a human confirm for write actions.
Exam scenario
Contoso Health is building a member-chat agent in a Foundry project. The product owner pastes a long “be a friendly nurse” paragraph into the playground, sets temperature to 1 in the UI, and is happy with two demo questions. You, the MLOps engineer, move the instructions into prompts/v1_instructions.txt , add: answer only from retrieved policy chunks; return JSON { "answer": "", "citations": [] } ; never diagnose; call lookup_policy when the user names a plan code; ask for the plan code if it is missing. You set temperature 0.2 on the agent/deployment config , not in the Markdown. You load the file in create_version() . That is prompt design for AI-300.
Common trap
The trap is treating the playground as the prompt. A second trap is stuffing retrieved documents into the user turn with no “answer only from context / I don’t know / cite sources” system rule — the model then freely mixes memory with RAG. A third trap is putting API keys, member PHI, or a temperature you cannot find later inside the instruction file. A fourth is designing chain-of-thought “show your hidden reasoning” prompts for reasoning models; Microsoft documents that extracting hidden reasoning outside supported summary parameters can violate acceptable use.
Design the prompt as a Foundry or Azure Machine Learning artifact with roles, grounding, schema, safety, and tool policy. The next section measures whether a change actually improved it.