Study guide

3.2 Training, Evaluation, Deployment, and Monitoring Lifecycle

AWS AI Practitioner Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: AIF-C01 Orientation and Official Source Control

1.1 Credential Scope, Code, and Search Language 1.2 Exam Format, Cost, Score, and Delivery 1.3 Official Exam Guide, Skill Builder, and Practice Workflow 1.4 Target Candidate Boundaries and Out-of-Scope Tasks 1.5 Domain Weights and Study Prioritization 1.6 Exam Policy, Retake, Results, and Recertification

2 Chapter 2: AI/ML Foundations and Use-Case Fit

2.1 AI, ML, Deep Learning, and Core Terminology 2.2 Supervised, Unsupervised, and Reinforcement Learning 2.3 Data Types, Labels, Structure, and Quality 2.4 Inference Patterns: Batch, Real-Time, and Embedded AI 2.5 AI Use-Case Fit and No-AI Decisions 2.6 Classification, Regression, Clustering, Forecasting, and Recommendation 2.7 AI/ML Foundations Case Lab

3 Chapter 3: ML Lifecycle, Metrics, and Practitioner MLOps

3.1 Data Collection, EDA, Preprocessing, and Feature Concepts 3.2 Training, Evaluation, Deployment, and Monitoring Lifecycle 3.3 Model Sources: Managed APIs, Open Source, and Custom Models 3.4 Evaluation Metrics and Business Metrics 3.5 Practitioner MLOps: Repeatability, Monitoring, and Retraining 3.6 SageMaker Lifecycle Service Map 3.7 ML Lifecycle Case Lab

4 Chapter 4: Generative AI Foundations and Inference Concepts

4.1 Foundation Models, LLMs, Transformers, and Modalities 4.2 Tokens, Context Windows, Embeddings, and Vector Search 4.3 Inference Parameters: Temperature, Top-p, and Output Controls 4.4 Hallucination, Grounding, and Context Quality 4.5 GenAI Use-Case Fit and Risk Triage 4.6 Prompt Injection, Data Leakage, and User Input Risk 4.7 Generative AI Foundations Case Lab

5 Chapter 5: Prompting, Model Selection, Customization, and Evaluation

5.1 Prompt Engineering Patterns and Business Quality 5.2 Zero-Shot, Few-Shot, Templates, and Instruction Design 5.3 Model Selection: Capability, Latency, Cost, and Risk 5.4 RAG vs Fine-Tuning vs Prompting vs Custom Models 5.5 Model Evaluation, Human Review, and Red-Team Feedback 5.6 Cost, Performance, and Throughput Decision-Making 5.7 Prompting and Model Selection Case Lab

6 Chapter 6: Amazon Bedrock, RAG, Agents, and Guardrails

6.1 Bedrock Core Concepts, Model Access, and Managed FM Choice 6.2 Knowledge Bases, RAG, Data Sources, and Grounding 6.3 AI Agents, Tools, Orchestration, and Business Workflows 6.4 Guardrails, Content Filters, Denied Topics, and Sensitive Data 6.5 Bedrock Model Evaluation, Monitoring, and Human Feedback 6.6 Bedrock Cost, Latency, Throughput, and Operational Fit 6.7 Bedrock, RAG, and Agents Case Lab

7 Chapter 7: AWS Managed AI/ML Services and SageMaker Map

7.1 Managed AI Services vs Foundation Model Apps vs Custom ML 7.2 Text, Language, Search, and Document AI Services 7.3 Vision, Speech, Contact Center, and Personalization Services 7.4 Amazon Quick, Amazon Q Developer, Kiro, and Practitioner Fit 7.5 SageMaker Canvas, Studio, Clarify, Autopilot, and Data Wrangler 7.6 Data Foundation Services: S3, Glue, OpenSearch, and Quick Sight 7.7 AWS AI Service Selection Case Lab

8 Chapter 8: Responsible AI, Human Review, and Safety

8.1 Fairness, Bias, Transparency, and Explainability 8.2 Privacy, Safety, Human Review, and Accountability 8.3 Guardrails, Clarify, Human Review, and Content Safety Controls 8.4 Responsible AI Risk Registers and Governance Workflows 8.5 Monitoring, Feedback, Drift, and Incident Response 8.6 Responsible AI Case Lab

9 Chapter 9: Security, Compliance, Governance, and Cost Controls

9.1 Shared Responsibility, IAM, and Least Privilege for AI 9.2 Encryption, Secrets, Networking, and Data Privacy 9.3 Prompt Injection, Data Exfiltration, and GenAI Threat Modeling 9.4 Logging, Monitoring, CloudTrail, CloudWatch, and Config 9.5 Compliance: Artifact, Config, Macie, and Policy Evidence 9.6 AI Cost Controls, Pricing, Throughput, and Budget Governance 9.7 Security and Governance Case Lab

10 Chapter 10: Integrated AWS AI Business Scenario Labs

10.1 Customer Support GenAI Assistant Lab 10.2 Document Intelligence and Compliance Review Lab 10.3 Personalization, Forecasting, and Fraud Detection Lab 10.4 Enterprise Search, RAG, and Knowledge Management Lab 10.5 Responsible AI and Security Review Board Lab 10.6 Cost, Performance, and Operations Review Lab 10.7 Full AIF-C01 Business Simulation

11 Chapter 11: Final Review, Exam Readiness, and Recertification

11.1 Final 30-Day AIF-C01 Study Plan 11.2 Official Practice Resources and Weak-Domain Remediation 11.3 90-Minute Exam Timing, Flagging, and Guessing Workflow 11.4 Test-Day Checklist: Online or Test Center 11.5 Post-Exam Results, Retake, and Recertification Plan 11.6 AWS AI Practitioner Final Mixed Review

3.2 Training, Evaluation, Deployment, and Monitoring Lifecycle

Key Takeaways Explain This Topic AI

The ML lifecycle moves from problem framing and data preparation through training, evaluation, deployment, monitoring, and retraining.

Training teaches a model patterns from data, while evaluation checks whether those patterns generalize to data the model did not train on.

Deployment is an operational decision as much as a technical one because latency, scale, cost, security, human review, and rollback all matter.

Monitoring is required because data, user behavior, business rules, and model performance can drift after launch.

Last updated: October 2026

The lifecycle at practitioner depth

The AWS AI Practitioner target candidate is familiar with AI/ML solutions but does not necessarily build them. That means you should understand the lifecycle well enough to review plans, question risks, choose between managed services and custom work, and interpret project status. You do not need to implement training loops, tune hyperparameters, or build pipelines for this exam scope.

A typical ML lifecycle starts with problem framing, data collection, data preparation, training, evaluation, deployment, monitoring, and retraining. The order sounds linear, but real projects are iterative. Evaluation may reveal weak labels. Monitoring may reveal drift. A business review may show that a model is accurate but not useful because the decision arrives too late or costs too much. Business problem

  • > Data readiness
  • > Training or service selection
  • > Evaluation
  • > Deployment decision
  • > Monitoring
  • > Feedback and retraining

Training is the phase where an algorithm learns patterns from data. In supervised learning, the training data includes labels, such as approved or denied, fraudulent or legitimate, churned or retained, and defective or acceptable. In unsupervised learning, the system looks for structure without explicit labels, such as clusters or anomalies. Reinforcement learning involves learning through rewards, but most practitioner business scenarios focus on supervised, unsupervised, or generative AI service use.

Evaluation tests whether the trained model performs well on data not used for training. Teams commonly split data into training, validation, and test sets. Training data teaches the model. Validation data helps compare model choices during development. Test data is held back to provide a more honest final check. If a team evaluates only on training data, the result can hide overfitting, where the model memorizes the past rather than generalizing.

  • Lifecycle stage Main question Practitioner risk to check
  • Problem framing What decision or task improves? Goal is vague, low value, or better solved with rules
  • Data preparation Is the data usable and approved? Missing labels, leakage, privacy gaps, bias, poor quality
  • Training Can a model learn the needed pattern? Overfitting, underfitting, weak baseline, high cost
  • Evaluation Does it work on unseen data and key segments? Single metric hides failures or unfair outcomes
  • Deployment Can users safely consume predictions? Latency, scaling, access, rollback, human review gaps
  • Monitoring Does it keep working after launch? Drift, stale data, cost growth, user misuse
  • Retraining Should the model be refreshed or retired? No feedback loop or unclear ownership
  • Deployment is not just publishing a model

Deployment means making predictions, classifications, recommendations, or generated outputs available to users or systems. That may happen through a real-time endpoint, batch job, embedded application, managed API, dashboard, or human review queue. Amazon SageMaker AI can host model endpoints and batch transform jobs. Managed AI services such as Amazon Comprehend, Rekognition, Textract, Translate, Transcribe, Polly, and Personalize can reduce custom development when the use case matches the service.

The deployment pattern should fit the workflow. A fraud risk score during checkout may need low latency. A monthly churn prediction can run as batch scoring. Document extraction may use Textract asynchronously for large files. A support agent suggestion may require human review before a message reaches a customer. If the risk is high, the deployment should include thresholds, escalation rules, audit logging, and rollback steps.

A deployment readiness checklist should be short and specific:

Name the user or system that will consume the output.

Confirm latency and availability expectations.

Confirm IAM, encryption, network, and logging requirements.

Define human review for low-confidence or high-impact cases.

Define rollback or fallback to rules, queues, or manual processing.

Estimate inference cost and expected volume.

Confirm monitoring metrics and ownership before launch.

Monitoring closes the loop

A model that performs well at launch can degrade. Data drift occurs when input data changes, such as new customer behavior, new regions, new devices, economic shocks, or changed product catalogs. Concept drift occurs when the relationship between inputs and outcomes changes, such as a fraud pattern changing after attackers adapt. Operational drift can happen when upstream systems change field names, units, formats, or missing-value behavior.

Monitoring should include technical health and business outcomes. Technical monitoring includes latency, error rate, throughput, failed requests, endpoint capacity, and data quality. Model monitoring includes prediction distribution, feature drift, confidence, and performance against labels when labels arrive later. Business monitoring includes cost, conversion, false-positive workload, customer feedback, manual review volume, and incident reports.

AWS services support monitoring at different layers. Amazon CloudWatch can track application and service metrics, logs, and alarms. CloudTrail can record API activity for audit. SageMaker Model Monitor and SageMaker Clarify are no longer open to new customers. For new projects, AWS's replacement guidance uses open-source SageMaker AI monitoring solutions with Amazon Quick Sight dashboards and CloudWatch for drift, computes the standardized bias metrics directly with SHAP for explanations, and uses Amazon Bedrock Evaluations for foundation models. These services do not remove the need for business ownership. Someone must decide what action follows an alarm.

When to pause or choose another path

Not every AI idea should become a model. If the process requires deterministic behavior, use rules or standard automation. If data is poor or not permitted for use, fix governance and collection first. If the decision is high impact and explanations are required, include human review and explainability expectations. If the improvement is small compared with operational cost, do not overbuild.

For study, focus on lifecycle judgment. Training is not success by itself. A high offline score is not success by itself. Production launch is not the finish line. A responsible solution proves value, is monitored, has an owner, and can be changed or retired when conditions change.