Study guide
4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus
AWS Certified Machine Learning Engineer — Associate MLA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Introduction & Exam Strategy
1.1 AWS MLA-C01 Blueprint, Format & Strategy 1.2 The Machine Learning Engineering Lifecycle on AWS
2 Domain 1: Data Ingestion & Storage Architecture for ML
2.1 Ingesting Batch and Streaming Data for ML 2.2 High-Performance Storage for Training and Inference 2.3 Data Cataloging, Governance, and Querying
3 Domain 1: Feature Engineering & Data Transformation
3.1 Distributed Feature Engineering with AWS Glue & DataBrew 3.2 Interactive Data Preparation with SageMaker Data Wrangler 3.3 Feature Management with SageMaker Feature Store
4 Domain 1: Data Quality, Integrity & Ground Truth Labeling
4.1 Data Validation, Schema Drift, and Quality Rules 4.2 Handling Missing Values, Outliers & Class Imbalance 4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus
5 Domain 2: Model Selection & Framework Architecture
5.1 SageMaker Built-in Algorithms & Selection Criteria 5.2 Custom Models with Script Mode and BYOC 5.3 Foundation Models, JumpStart, and Amazon Bedrock
6 Domain 2: SageMaker Model Training & Hyperparameter Optimization
6.1 SageMaker Training Jobs & Cost Optimization 6.2 Distributed Training Strategies & Architectures 6.3 Hyperparameter Optimization with Automatic Model Tuning (AMT)
7 Domain 2: Model Evaluation, Tracking & Model Registry
7.1 ML Evaluation Metrics & Validation Strategies 7.2 Experiment Tracking with SageMaker Experiments 7.3 Model Governance with SageMaker Model Registry
8 Domain 3: SageMaker Inference Options & Endpoint Architecture
8.1 Real-Time, Serverless, and Asynchronous Inference 8.2 High-Throughput Offline Inference with Batch Transform 8.3 Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)
9 Domain 3: Deployment Strategies & Infrastructure Scaling
9.1 Production Deployment Patterns & Traffic Shifting 9.2 Endpoint Auto-Scaling & Hardware Acceleration
10 Domain 3: MLOps Pipelines & Workflow Orchestration
10.1 SageMaker Pipelines Core Architecture 10.2 Advanced Pipeline Patterns & Integration 10.3 End-to-End MLOps CI/CD & Automated Retraining
11 Domain 4: Production Model Monitoring with SageMaker Model Monitor
11.1 Data Quality & Model Quality Monitoring 11.2 Model Bias & Model Explainability Drift 11.3 Operational Observability with Amazon CloudWatch
12 Domain 4: Responsible AI, Bias Detection & Governance
12.1 Pre-Training and Post-Training Bias with SageMaker Clarify 12.2 Model Documentation & Auditing with Model Cards & Dashboard 12.3 Safety, Guardrails & Generative AI Governance
13 Domain 4: ML Security, Network Isolation & Cost Optimization
13.1 IAM Roles and Least Privilege for ML 13.2 Network Isolation, VPC Endpoints & Data Encryption 13.3 ML Cost Optimization & Resource Governance
4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus
Key Takeaways Explain This Topic AI
Amazon SageMaker Ground Truth supports 4 distinct workforce types: Public (Amazon Mechanical Turk for non-sensitive data), Private (internal staff authenticated via Amazon Cognito or OIDC), Vendor (vetted AWS Marketplace vendors), and Ground Truth Plus (turnkey managed labeling service).
Built-in labeling task modalities include Image Classification, Object Detection (Bounding Boxes), Semantic Segmentation, Text Classification, Named Entity Recognition (NER), and 3D Point Cloud Labeling, with custom workflows supported via Liquid HTML and AWS Lambda.
Ground Truth Active Learning (Automated Data Labeling) significantly reduces labeling costs by training an auto-labeling model that automatically annotates high-confidence samples while routing ambiguous, low-confidence samples to human annotators.
Annotation consolidation resolves multi-worker disagreements using probabilistic algorithms such as Dawid-Skene and expectation-maximization (EM) alongside golden calibration datasets.
Labeled outputs are generated as Augmented Manifest files (.jsonl), which can be fed directly into SageMaker Training Channels using the AugmentedManifestFile S3 data type without intermediate data conversion.
Last updated: August 2026
Data Labeling with SageMaker Ground Truth & Ground Truth Plus
Supervised machine learning algorithms require large volumes of accurately annotated training data. Generating high-quality ground truth labels is often the most time-consuming and expensive bottleneck in the ML lifecycle. Amazon SageMaker Ground Truth and Amazon SageMaker Ground Truth Plus provide managed human-in-the-loop data labeling services that streamline dataset creation across computer vision, natural language processing, audio, and 3D point cloud domains.
For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master the differences between the four workforce types, active learning and automated labeling mechanics, annotation consolidation algorithms (such as Dawid-Skene ), custom labeling workflow architectures using AWS Lambda , and ingesting Augmented Manifest files directly into SageMaker training jobs. +---------------------------------------------------------------------------------------------------+
- | SAGEMAKER GROUND TRUTH WORKFLOW SPECTRUM |
- | |
- | [Raw Unlabeled Data] ---> [Ground Truth Labeling Job] |
- | | |
- | +----------------------+----------------------+ |
- | | Active Learning (Auto-Labeling) | Workforce Selection |
- | v v |
- | [Auto-Labeling Model] [1. Public (Mechanical Turk)] |
- | - Confident >= Threshold ---> Auto-labeled [2. Private (Cognito / OIDC)] |
- | - Confident < Threshold ---> Routed to Human [3. Vendor (AWS Marketplace)] |
- | [4. Ground Truth Plus (Managed)] |
- | | |
- | v |
- | [Annotation Consolidation] |
- | (Dawid-Skene / Consensus) |
- | | |
- | v |
- | [Augmented Manifest (.jsonl)] |
- | | |
- | v |
- | [SageMaker Training Job] |
- +---------------------------------------------------------------------------------------------------+
- 1. Built-in Task Types & Custom Workflows
Amazon SageMaker Ground Truth provides out-of-the-box labeling user interfaces (worker UIs) for the most common ML tasks:
Built-in Modalities & Task Types
- Modality Task Type Worker Interface & Output Description
- Computer Vision Image Classification (Single/Multi-label) Assigns one or more categorical tags to whole images
Computer Vision Object Detection (Bounding Box) Draws rectangular 2D bounding boxes around specified classes with coordinates [top, left, height, width]
Computer Vision Semantic Segmentation Pixel-level mask segmentation to classify every pixel in an image (e.g., road, vehicle, pedestrian)
Natural Language Text Classification Categorizes paragraphs or documents (e.g., sentiment, topic classification)
Natural Language Named Entity Recognition (NER) Highlights and tags specific token spans in text (e.g., PERSON, ORGANIZATION, DATE, MEDICAL_CODE)
3D Point Cloud 3D Bounding Box & Tracking Labels 3D cuboids and tracks dynamic objects across LiDAR point cloud frames
Video Video Object Tracking / Detection Bounding boxes tracked across sequential video frames with automatic interpolation
Custom Labeling Workflows
When requirements exceed built-in task types (e.g., combined text-image QA, audio transcript annotation, or complex multi-step forms), ML engineers create Custom Workflows using two integration hooks:
Pre-Annotation Lambda: Preprocesses raw input data from S3 before it is displayed to the worker (e.g., tokenizing text, attaching reference metadata, or dynamically fetching external assets).
Custom Worker UI (Liquid HTML): Built using HTML, CSS, JavaScript, and Crowd HTML Elements with Liquid templating tags (e.g., <crowd-bounding-box src='{{ task.input.source-ref | grant_read_access }}' ...> ).
Post-Annotation Lambda (Consolidation): Executes custom consolidation algorithms over multiple worker responses to produce a single ground truth record. +---------------------------------------------------------------------------------------------------+
- | CUSTOM WORKFLOW ARCHITECTURE |
- | |
- | [Raw S3 Object] ---> [Pre-Annotation Lambda] ---> [Custom Worker HTML UI (Liquid)] |
- | | |
- | v |
- | [Final Manifest] <--- [Post-Annotation Lambda] <--- [Worker Annotations (1..N)] |
- +---------------------------------------------------------------------------------------------------+
- 2. Workforce Management & Security Governance
Choosing the correct workforce is a critical architectural and regulatory decision on the MLA-C01 exam. +---------------------------------------------------------------------------------------------------+
- | WORKFORCE SELECTION MATRIX |
- | |
- | Workforce Type Identity Provider Best For PII / Sensitive |
- | --------------------- ------------------------ -------------------------- --------------- |
- | Amazon Mechanical Turk Public AWS Crowdsource Large non-sensitive datasets STRICTLY NO |
- | Private Workforce Amazon Cognito / OIDC Internal staff, confidential ALLOWED (Secure)|
- | Vendor Workforce AWS Marketplace Vendors Specialized domain expertise ALLOWED (SLA) |
- | Ground Truth Plus Managed Turnkey AWS Team Zero-ops custom workflows ALLOWED (HIPAA) |
- +---------------------------------------------------------------------------------------------------+
- Detailed Workforce Characteristics
- 1. Amazon Mechanical Turk (Public Workforce)
A global on-demand crowdsource community of over 500,000 workers.
Advantages: Highly scalable, fastest turnaround, lowest cost per task.
Strict Constraint: NEVER use Mechanical Turk for datasets containing Personally Identifiable Information (PII), Protected Health Information (PHI), financial data, or proprietary intellectual property. Workers are anonymous public individuals.
2. Private Workforce
Composed of internal company employees, subject matter experts, or contracted contractors.
Authentication & Federation: Managed via Amazon Cognito User Pools (standalone worker portal) or federated with corporate Identity Providers (e.g., Okta, Microsoft Entra ID / Azure AD, PingFederate) using OpenID Connect (OIDC) or SAML 2.0 .
Security: Fully compliant with enterprise data protection policies. Restrictable to corporate VPC networks using AWS PrivateLink.
3. Vendor Workforce
Professional labeling service providers vetted by AWS and contracted directly through the AWS Marketplace .
Advantages: Access to specialized domain experts (e.g., radiologists for medical imaging or attorneys for legal contract analysis) backed by Service Level Agreements (SLAs) for labeling accuracy and turnaround time.
4. Amazon SageMaker Ground Truth Plus
A fully managed, turnkey data labeling service where AWS manages the entire labeling operation.
AWS ML experts build custom labeling workflows, create worker training materials, assemble and manage pre-vetted domain-expert workforces, and enforce multi-stage quality assurance.
Key Benefit: Requires zero data labeling infrastructure or UI template development from your internal engineering team.
3. Automated Data Labeling & Active Learning
Automated Data Labeling in SageMaker Ground Truth utilizes an Active Learning feedback loop to minimize the number of human annotations required, reducing labeling costs by up to 70% . +---------------------------------------------------------------------------------------------------+
- | ACTIVE LEARNING AUTOMATED LABELING LOOP |
- | |
- | [1. Raw S3 Objects] |
- | | |
- | v |
- | [2. Human Labeling Phase] ---> Human workers label initial random batch (e.g., 20%) |
- | | |
- | v |
- | [3. Auto-Labeling Model] ---> SageMaker trains an internal model on human-labeled data |
- | | |
- | v |
- | [4. Batch Inference] ---> Model predicts labels on remaining unlabeled data |
- | | |
- | +-----------------------------------+ |
- | | Confidence >= Threshold | Confidence < Threshold |
- | v v |
- | [Auto-Labeled to Manifest] [5. Routed to Human Workers] |
- | | |
- | v |
- | [6. Retrain Auto-Model] ---> (Iterative Loop Repeats) |
- +---------------------------------------------------------------------------------------------------+
- Operational Mechanics of Active Learning
Bootstrap Phase: A random sample of raw data is sent to human annotators.
Model Training: Once a sufficient volume of human annotations is accumulated (minimum requirements: at least 1,250 validation objects for image classification, 5,000 for text), Ground Truth trains a custom neural network auto-labeling model.
Inference & Scoring: The auto-labeling model scores the remaining unlabeled objects.
Confidence Partitioning:
Objects whose model prediction confidence exceeds the pre-calibrated threshold are automatically assigned labels and written directly to the output manifest ( human-annotated: 'no' ).
Objects below the confidence threshold are dispatched to human annotators ( human-annotated: 'yes' ).
Iterative Retraining: Newly human-labeled samples are merged back into the training pool to retrain the auto-labeling model across subsequent iterations, progressively raising model confidence and decreasing human workload.
4. Quality Control & Annotation Consolidation
To ensure ground truth integrity, multiple workers typically label the same object (e.g., 3 to 5 workers per image). Ground Truth uses mathematical consolidation algorithms to resolve conflicting worker inputs. +---------------------------------------------------------------------------------------------------+
- | ANNOTATION CONSOLIDATION METHODS |
- | |
- | 1. Majority Voting: |
- | - Takes the mode of worker labels. Naive approach; treats all workers as equally reliable. |
- | |
- | 2. Dawid-Skene Algorithm (Expectation-Maximization): |
- | - Jointly estimates worker confusion matrices (individual worker accuracy) and true class |
- | probabilities simultaneously. |
- | - Gives higher probabilistic weight to consistently accurate annotators. |
- | |
- | 3. Golden Datasets / Worker Calibration: |
- | - Injects known ground truth verification tasks into worker queues. |
- | - Measures worker accuracy in real time; disqualifies workers who fail calibration checks. |
- +---------------------------------------------------------------------------------------------------+
- The Dawid-Skene Algorithm
Rather than assuming every worker is equally skilled, the Dawid-Skene algorithm uses an iterative Expectation-Maximization (EM) framework:
E-step: Estimates the true class probability distribution for each object based on current worker accuracy estimates.
M-step: Updates the individual error matrix (confusion matrix) for each worker based on the estimated true labels.
This ensures that if Worker A has a 99% historical accuracy while Worker B has a 60% accuracy, Worker A's label carries significantly more mathematical weight during consolidation.
5. Output Manifests & SageMaker Training Integration
Ground Truth outputs annotations in the form of an Augmented Manifest file ( output.manifest ), formatted as a newline-delimited JSON ( .jsonl ) file stored in Amazon S3.
Structure of an Augmented Manifest File
Each line represents a single data record containing the S3 source reference, the consolidated label, and detailed metadata: {
- 'source-ref': 's3://ml-training-lake/images/sample_001.jpg',
- 'bounding-box-labels': {
- 'image_size': [{'width': 1920, 'height': 1080, 'depth': 3}],
- 'annotations': [
- {'class_id': 0, 'left': 340, 'top': 180, 'width': 220, 'height': 450}
- ]
- },
- 'bounding-box-labels-metadata': {
- 'class-map': {'0': 'pedestrian'},
- 'human-annotated': 'yes',
- 'confidence': 0.96,
- 'creation-date': '2026-08-16T12:00:00.000000',
- 'type': 'groundtruth/object-detection',
- 'job-name': 'gt-pedestrian-detection-job'
- }
- }
- Ingesting Augmented Manifests into SageMaker Training Jobs
SageMaker built-in algorithms (such as Object Detection, Image Classification, and Semantic Segmentation) and custom Script Mode estimators can consume Augmented Manifest files directly using the TrainingInput configuration with s3_data_type='AugmentedManifestFile' : import sagemaker
from sagemaker.inputs import TrainingInput
- # Configure TrainingInput channel pointing to Ground Truth Augmented Manifest
- train_channel = TrainingInput(
- s3_data='s3://ml-training-lake/ground-truth-output/gt-job/manifests/output/output.manifest',
- content_type='application/x-recordio',
- s3_data_type='AugmentedManifestFile',
- record_wrapping='RecordIO',
- attribute_names=['source-ref', 'bounding-box-labels'] # Identifies input and label keys
- )
- # Initialize Estimator
- estimator = sagemaker.estimator.Estimator(
- image_uri=sagemaker.image_uris.retrieve('object-detection', 'us-east-1'),
- role=sagemaker.get_execution_role(),
- instance_count=1,
- instance_type='ml.g5.2xlarge',
- output_path='s3://ml-training-lake/models/'
- )
- # Fit estimator using the manifest channel
- estimator.fit({'train': train_channel})
Important
Attribute Names Alignment: When passing an Augmented Manifest to a SageMaker training channel, the attribute_names parameter must list exactly two JSON keys: the source reference key (e.g., source-ref ) and the label key (e.g., bounding-box-labels ). SageMaker streams the raw objects and annotations directly without requiring manual data conversion.
Note
Amazon Augmented AI (Amazon A2I): Where Ground Truth labels raw training data, Amazon A2I inserts human review into production predictions — routing low-confidence or high-risk inferences (for example, Amazon Textract document extractions, Amazon Rekognition moderation results, or a custom model score below a confidence threshold) to human reviewers before the output is consumed downstream. Choose A2I when an exam scenario requires a human-in-the-loop review step on live model outputs rather than dataset annotation.