Study guide
4.3 Custom Text Classification and Custom NER
Azure AI Engineer Associate AI-102
Track your progress as you read
Finished sections get a check mark.
1 Introduction
About the Azure AI Engineer Associate (AI-102) Exam Study Strategies and Exam Tips for the AI-102
2 Domain 1: Plan and Manage an Azure AI Solution (20-25%)
1.1 Azure AI Services Overview and Selection 1.2 Resource Provisioning and Configuration 1.3 Security and Access Management 1.4 Responsible AI Principles and Implementation 1.5 Azure AI Solution Architecture Patterns
3 Content Safety and Moderation (within Plan and Manage, Domain 1)
2.1 Azure AI Content Safety Overview 2.2 Prompt Shields and Adversarial Attack Detection 2.3 Text and Image Moderation Implementation 2.4 Content Filtering in Azure OpenAI Service
4 Domain 4: Implement Computer Vision Solutions (10-15%)
3.1 Azure AI Vision — Image Analysis 4.0 3.2 Azure AI Face Service 3.3 Azure AI Custom Vision 3.4 Optical Character Recognition (OCR) 3.5 Video Analysis and the Retired Spatial Analysis
5 Domain 5: Implement Natural Language Processing Solutions (15-20%)
4.1 Azure AI Language — Text Analytics Features 4.2 Conversational Language Understanding (CLU) 4.3 Custom Text Classification and Custom NER 4.4 Azure AI Speech Services 4.5 Azure AI Translator Service 4.6 Question Answering and Conversational AI
6 Domain 6: Implement Knowledge Mining and Information Extraction Solutions (15-20%)
5.1 Azure AI Search — Fundamentals and Architecture 5.2 AI Enrichment with Skillsets 5.3 Azure AI Document Intelligence
7 Domain 2: Implement Generative AI Solutions (15-20%)
6.1 Azure OpenAI Service — Models and Deployment 6.2 Chat Completions API and Prompt Engineering 6.3 Embeddings and Vector Search for RAG 6.4 Fine-Tuning and Model Customization 6.5 Image Generation (gpt-image; DALL-E 3 Retired) 6.6 RAG After On Your Data, and AI Agents
8 Domain 3: Implement an Agentic Solution (5-10%)
3.1 Agentic Solutions: Foundry Agent Service and the Microsoft Agent Framework
9 Exam Review: Cross-Domain Topics and Advanced Practice
7.1 Service Comparison and Selection Guide 7.2 SDK and REST API Patterns 7.3 Responsible AI Across All Domains 7.4 End-to-End Solution Architectures 7.5 Exam Day Quick Reference
4.3 Custom Text Classification and Custom NER
Key Takeaways Explain This Topic AI
Custom text classification assigns documents to your classes in single-label (one class) or multi-label (many classes) projects.
Custom NER extracts domain-specific entities the pre-built NER model does not cover, using labeled character spans.
Both features require .txt documents in Azure Blob Storage plus a JSON labels file; Language Studio connects to the container via managed identity or SAS.
Training supports automatic percentage split (e.g. 80/20) or manual train/test assignment; evaluation reports precision, recall, and F1 per class/entity.
Both are long-running operations: begin_single_label_classify / begin_multi_label_classify / begin_recognize_custom_entities return a poller and require a named deployment.
Last updated: June 2026
Quick Answer: Custom text classification assigns documents to your own classes; custom NER extracts your own entity types. Both need labeled .txt files in Azure Blob Storage , a JSON labels file, a train/test split, training in Language Studio or REST, and a named deployment before prediction. Evaluation reports precision, recall, and F1 per class or entity.
Single-Label vs. Multi-Label Classification
- Type Rule Example Project kind
- Single-label Exactly one class per document Ticket = Billing OR Technical CustomSingleLabelClassification
- Multi-label Zero or more classes per document Article = Politics AND Economy CustomMultiLabelClassification
Pick single-label when classes are mutually exclusive; pick multi-label when a document can legitimately belong to several. Choosing wrong is a common scenario-question trap — a news article tagged with several topics needs multi-label.
Data and Labeling Requirements
- Requirement Minimum Recommended
- Documents per class/entity 10 50+
- Storage Azure Blob (.txt, UTF-8) dedicated container
- Labels file one JSON mapping documents versioned
- Auth to storage SAS or managed identity managed identity
The labels JSON declares metadata.projectKind , the list of classes (or entities ), and per-document assignments. For custom NER each labeled span carries an exact offset and length — off-by-one offsets are the most common cause of poor recall. {
- "projectFileVersion": "2022-05-01",
- "metadata": {"projectKind": "CustomEntityRecognition",
- "projectName": "LegalExtractor", "language": "en"},
- "assets": {"projectKind": "CustomEntityRecognition",
- "entities": [{"category": "CaseNumber"}, {"category": "JudgeName"}],
- "documents": [{"location": "doc1.txt", "language": "en",
- "entities": [{"regionOffset": 45, "regionLength": 12,
- "labels": [{"category": "CaseNumber", "offset": 45, "length": 12}]}]}]}}
- Training, Splitting, and Calling poller = client.begin_single_label_classify(
- documents=["I was charged twice for my renewal"],
- project_name="TicketClassifier", deployment_name="production")
for result in poller.result():
for c in result.classifications:
print(c.category, round(c.confidence_score, 2))
Use begin_multi_label_classify for multi-label and begin_recognize_custom_entities for custom NER. All three are long-running pollers and require a deployment name — predicting against an undeployed model fails.
- Split strategy How Best for
- Automatic % service randomly holds out e.g. 20% most projects
- Manual you tag each doc Train or Test reproducible/regulated test sets
- Evaluation Metrics
- Metric Meaning Formula
- Precision Of predicted positives, how many were right TP / (TP + FP)
- Recall Of actual positives, how many were found TP / (TP + FN)
- F1 Harmonic mean balancing the two 2·P·R / (P + R)
Interpreting these is heavily tested. High precision + low recall = conservative model that misses real entities (fix: add more diverse positive examples). Low precision + high recall = over-eager model with many false positives (fix: add negative/boundary examples and clean mislabels).
Best Practices and Traps
Label consistency: the same text type must always get the same label, or the model learns noise.
Negative examples: include documents that contain none of the target entities so the model learns when not to predict.
Class balance: roughly equal counts per class; a dominant class skews accuracy upward while small classes silently fail.
Boundary precision: trim trailing punctuation/whitespace from custom-NER spans.
Common Trap: Custom features need data in Blob Storage , not uploaded directly in Language Studio — Studio merely connects to the container. And custom classification is distinct from CLU: CLU classifies short conversational utterances into intents, while custom text classification classifies whole documents into business categories.
Project Lifecycle and Storage Connection
The end-to-end lifecycle for both custom features is the same: connect a storage account, create the project and schema, label data, train with a chosen split, review metrics, deploy to a named slot, then call the runtime endpoint. The storage connection step trips up many candidates. Language Studio does not host your files; instead you grant the Language resource access to a blob container, preferably through a managed identity assigned the Storage Blob Data Contributor role, or via a shared access signature.
If that permission is missing, project creation succeeds but training fails to read documents, which surfaces as an empty or errored job rather than an obvious permission message.
A practical sizing rule is that the model is only as good as the diversity of its labeled spans and documents. Ten examples per class is a floor that produces a model you can demonstrate, not one you should ship; fifty or more per class is where accuracy stabilizes. For custom NER specifically, the quality of character offsets dominates everything else, so labeling in the Studio interface rather than hand-editing JSON is recommended because the tool computes offsets for you and respects the configured string index type.
When Custom Beats Pre-Built
The exam repeatedly contrasts custom features with pre-built ones and with generative alternatives. Choose custom NER when the entities are proprietary, such as internal part numbers, policy codes, or claim identifiers that no general model knows. Choose custom classification when categories reflect your own taxonomy, such as routing tickets into your specific support queues. If, instead, the entities are universal (people, places, dates) or the categories are generic sentiment, the pre-built models win because they need no data and no training cost. Knowing where that line sits lets you discard distractors quickly under time pressure.