Study guide
16.2 Create and Manage Synthetic Data for Fine-Tuning
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
16.2 Create and Manage Synthetic Data for Fine-Tuning
Key Takeaways Explain This Topic AI
Microsoft Foundry data generation (preview) expands scarce labeled sets into JSONL Chat Completions files. Do not treat preview output as production-ready without review.
Simple Q&A needs one PDF, Markdown, or text file under 20 MB. Tool use needs one OpenAPI 3.0.x or 3.1.x JSON file under 20 MB. You request 50–1,000 samples and may enable an 80/20 train-validation split.
Never put personally identifiable information or secrets in the seed file. Generators can leak PII, amplify seed bias, and cement the generator model’s own factual or tool-call errors.
Human-review a stratified sample, deduplicate exact and near-duplicate JSONL lines, and keep training, validation, and a held-out gold evaluation set disjoint.
Seed quality beats volume. Mix privacy-cleared real traces with synthetic rows so the model sees messy user language, not only generator-clean turns.
Last updated: August 2026
Create and Manage Synthetic Data for Fine-Tuning
Quick Answer: Use Microsoft Foundry data generation (preview) to expand a scarce labeled set into JSONL Chat Completions files. Generators take one reference file (a document for Simple Q&A, an OpenAPI JSON file for Tool use), emit 50–1,000 samples, and can split 80/20 train/validation. Never put personally identifiable information (PII) in the seed. Human-review , deduplicate , and never train on the evaluation set . Synthetic data can leak PII, amplify bias, and teach the model its own errors .
Domain 5 also measures create and manage synthetic data for fine-tuning . When you have 40 excellent gold conversations and the job wants hundreds, you do not scrape Slack into JSONL. You grow the gold set with a generator, then a human (or a stronger model plus a human) edits the output. Preview features change; the exam still expects the operating rules: seed hygiene, review, split discipline, and risk.
Why synthetic data exists
Microsoft lists four reasons Foundry generates synthetic training data:
Real labeled conversations are scarce .
You can preserve privacy if you generate from a cleaned policy or API specification instead of production transcripts that still contain member IDs.
You can encode business-specific structure (your JSON schema, your tools, your confirmation policy).
Generation is cheaper than paying annotators for every row.
It is still preview . Formats, limits, and the portal wizard can change. Always validate outputs before a production fine-tune. Synthetic data is a force multiplier for a good seed , not a substitute for a seed.
Foundry generators
In the portal, open Data > Synthetic Data Generation and select Generate data . The wizard asks for a task type, a reference file, a sample count, a generator model, and an optional split.
Generator Seed file What it produces Extra parameter
Simple Q&A One PDF, Markdown, or plain text file, under 20 MB Chat JSONL question–answer pairs Long answer (analytical reasoning) versus Short answer (factual brevity)
Tool use One OpenAPI 3.0.x or 3.1.x JSON (Swagger) file, under 20 MB Multi-turn conversations with tool_calls None in this preview
You also choose:
Sample count between 50 and 1,000 (you cannot ask for 10 rows or 50,000 in this wizard).
The model that writes the synthetic turns. If that model is not deployed, Foundry creates a deployment — which requires Foundry Owner (or equivalent deployment rights). Foundry User can run generation when a generator model already exists.
Optional 80/20 train-validation split of the generated dataset.
Optional suffix on the job name.
Output is JSONL ready for Foundry fine-tuning. Use Use this dataset in the preview pane or download the files for post-processing (PII scan, dedup, mixing with gold traces). Cost metrics appear on the completed job.
Supported regions for generation at the time of Microsoft’s July 2026 article include eastus2 , eastus , westus , northcentralus , southcentralus , swedencentral , germanywestcentral , francecentral , uksouth , uaenorth , japaneast , and australiaeast . Confirm the current Learn page before you design a landing zone; preview region lists move.
Minimum role to open the wizard: Foundry User or Foundry Project Manager on the Foundry resource. Deploying the generator model is a separate, higher bar.
Seed-file hygiene
The preview accepts one reference file. If you have six policy PDFs, concatenate them into one file before upload. Keep formatting clean:
Strip navigation chrome, cookie banners, “click here,” and repeated headers that are not subject matter.
Put the target domain in the file. A generator cannot invent your claims SOP from a blank page.
Do not include PII , account numbers, secrets, or production credentials. A generator will happily rewrite employee identifiers into hundreds of plausible fakes — or, worse, copy the real ones into assistant answers.
For tool use, the OpenAPI specification is the curriculum. Incomplete paths, missing required properties, or hallucinated operation IDs in the spec become incorrect tool calls in every synthetic dialogue.
Simple Q&A from a tax statute produces system / user / assistant rows with a generic helpful-assistant system message unless you replace it. After download, overwrite that system message with the production prompt you will train and serve (section 16.1). Tool-use output already includes a long policy system message plus a tools array; still review it against your real agent policy.
Risks: PII, bias, and self-taught errors
Synthetic data is not free quality. Three failure modes show up on AI-300-style scenarios and in production postmortems.
PII leakage. If the seed PDF is a poorly redacted export of case notes, generated Q&A can restate names, member IDs, and diagnoses. Run PII detection (Azure AI Language or an equivalent Presidio-style scan) on both the seed and the output . Drop or rewrite hits. Fine-tuning on leaked PII can surface that PII at inference, which is a privacy incident, not a model-quality ticket.
Bias amplification. If 90 percent of seed examples show one demographic, one dialect, one claim outcome, or one tool path, the generator will over-produce that pattern. The fine-tuned model then treats the skew as policy. Balance the seed before you scale to 1,000 rows. After generation, check intent and demographic (or product-line) histograms, not only BLEU-like overlap.
Teaching the model its own mistakes. A generator that invents a statute citation, a fake flight number, or a tool argument the API will reject will produce hundreds of confident, wrong assistant turns. Fine-tuning cements those errors into LoRA weights. Always sample-review. For tool-use data, parse tool_calls and validate arguments against the OpenAPI schema or a sandbox API. Delete or repair rows that would book, cancel, or pay without the confirmation policy you require.
Distribution shift. Synthetic Q&A is cleaner than angry customers. Mix real , privacy-cleared production traces with synthetic rows so the model sees interruptions, typos, and “that is not what I asked.” A model trained only on generator-clean turns often fails the first messy production utterance.
Human review, deduplication, and the evaluation split
Treat generation as a factory , not a publish button.
Review workflow:
Spot-check a stratified sample — every intent, every tool, every document section — not just the first 20 lines of the JSONL file.
Score samples with the same quality evaluators you will use later (groundedness, relevance, task adherence) and risk and safety evaluators (hate, self-harm, violence, jailbreak, protected material). Chapter 13 is the evaluator catalog; this chapter is about not poisoning the training set those evaluators will judge.
Have a domain expert accept or reject. Send rejects back as few-shot repairs or delete them. Do not “fix” a wrong citation by generating 200 more rows from the same dirty seed.
Deduplicate:
Exact-line JSONL duplicates waste epochs and overweight that conversation.
Near-duplicates (same user question, lightly paraphrased) collapse diversity. Hash normalized user turns and cap copies.
If you generated from the same PDF twice, do not concatenate both dumps blindly.
Never train on the evaluation set.
If you enable the portal’s 80/20 split, keep that validation file out of training_file . Validation loss is a training-time curve, not a product gate.
Hold out a third gold set that the generator never saw: real tickets, real tool traces, real schema failures. Validation loss on a synthetic-only holdout is optimistic.
Evaluation datasets you built in Domain 4 (groundedness and relevance mapping) must not be copied into JSONL training rows. Training on the test is how you ship a model that “passes eval” and fails users.
Seed quality beats volume . Fifty expert-edited dialogues plus 200 reviewed synthetic expansions beat 1,000 unreviewed generator dumps. Microsoft’s own fine-tuning article warns that a large unpruned internal dump can hurt relative to the base model. The same warning applies to synthetic volume.
Exam scenario: You have 60 gold tool-calling traces for a booking agent and a valid OpenAPI spec. You generate 800 more rows, enable 80/20, and fine-tune. Offline eval on the synthetic validation split looks excellent. Production users report the agent calling cancel_reservation without confirmation. Root cause: the generator omitted the “summarize then confirm” policy that existed only in the 60 gold traces, and you never mixed gold with synthetic. Fix: prepend the gold traces, put the confirmation policy in the system message on every row, regenerate, human-review confirmation turns, validate tool arguments against the spec, and evaluate on a held-out real transcript set.
Common trap: Training on the same generated file you later upload as the “eval set” in Foundry Evaluations. You are measuring memorization of synthetic phrasing, not generalization. Split once , pin file IDs in Git, and refuse jobs whose training_file identity equals the eval mapping file.
Operate the dataset as an MLOps asset
Synthetic data is a versioned artifact, the same way a component or an environment is:
Version seed files, generator job IDs, generator model name, sample count, long-versus-short setting, and split ratios in Git or a data catalog.
Prefer local upload or the SDK when Blob import would require public network access on the storage account. Microsoft requires public access for the Blob import path; many enterprises cannot allow that.
After human edits, re-run JSONL validation (UTF-8 BOM, chat schema, 512 MB cap from section 16.1).
Record what must never be generated : PII fields, secrets, internal hostnames, competitor names if policy forbids them.
When the OpenAPI spec or the policy PDF changes, regenerate — do not keep training on last quarter’s synthetic dump.
Review gates before a fine-tune job may consume the files:
Seed has no PII or secrets.
Generator region is allowed for the workload.
Sample count is in the 50–1,000 window.
Human stratified review is complete and recorded.
Exact and near-duplicate JSONL lines are removed.
Training, validation, and held-out gold eval files are disjoint.
Safety evaluators have scored a sample of assistant turns.
Production system message is present on every training row.
Gold traces are mixed in if they exist.
If any gate fails, you do not “make up for it” with more epochs. You fix the data. Section 16.3 will not save a model that learned the generator’s hallucinations.