Study guide

Verify the Model Is Ready for Operationalization

PMI Certified Professional in Managing AI PMI-CPMAI

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: PMI-CPMAI Exam Foundations & the CPMAI Methodology

The PMI-CPMAI Exam: Format, Eligibility, Fees & Scoring The CPMAI Methodology: Six Phases and Data-Centric Iteration CPMAI vs Waterfall, Agile, and CRISP-DM

2 Chapter 2: AI and Machine Learning Fundamentals for CPMAI

The Seven Patterns of AI Supervised, Unsupervised & Reinforcement Learning Why AI Projects Fail: Value, Data Readiness & Right-Sizing

3 Chapter 3: Domain I — Support Responsible and Trustworthy AI Efforts (15%)

Oversee the Privacy and Security Plan Manage AI/ML Transparency in Data and Algorithm Selection Conduct Bias Checks on Model, Data, and Algorithm Monitor Regulatory and Policy Compliance Manage Accountability Documentation and the Audit Trail

4 Chapter 4: Domain II — Identify Business Needs and Solutions, Part 1 (26%)

Identify the Problem to Be Solved Evaluate Initial AI Feasibility Conduct Security, Safety, and Ethics Risk Assessments Develop the AI Project Scope Statement Determine Return on Investment

5 Chapter 5: Domain II — Identify Business Needs and Solutions, Part 2

Manage Adoption and Integration Risks Draft the AI Solution Define Success Criteria and KPIs Support Business Case Creation Identify Project Resources: People, Hardware, and Contractors

6 Chapter 6: Domain III — Identify Data Needs, Part 1 (26%)

Define the Required Data Identify Data Subject-Matter Experts Identify Data Sources and Locations Coordinate the AI Workspace and Infrastructure Gather the Required Data

7 Chapter 7: Domain III — Identify Data Needs, Part 2

Check Data Privacy, Compliance, and Access Oversee Data Evaluation Determine If the Data Meets Solution Needs: The Phase 2 Go/No-Go Convey Data Understanding to Leadership

8 Chapter 8: Domain IV — Manage AI Model Development and Evaluation (16%)

Oversee AI/ML Model Technique and Algorithm Selection Oversee Model QA/QC and Configuration Management Manage AI/ML Model Training Manage Data Transformation and Data Preparation Verify Data Quality for the Data Preparation Go/No-Go Verify the Model Is Ready for Operationalization

9 Chapter 9: Domain V — Operationalize AI Solution, Part 1 (17%)

Manage Creation of the AI Solution Deployment Plan Manage the AI Solution Deployment Oversee Model Governance in Production Oversee AI Solution Metrics and Performance Monitoring

10 Chapter 10: Domain V — Operationalize AI Solution, Part 2

Prepare the Final Report and Lessons Learned Manage the AI Solution Transition Plan Oversee the AI Solution Contingency Plan

Verify the Model Is Ready for Operationalization

Key Takeaways Explain This Topic AI

CPMAI Phase 5 Model Evaluation judges the model against the success criteria fixed in Domain II Task 8 — the agreed metric and threshold, the business-impact threshold, and the baseline it must beat — not against whichever metric looks best on the day.

The test set counts only if it was genuinely untouched; a model tuned against it has no honest generalization estimate, and offline performance overstates production performance in essentially every case.

Aggregate performance routinely hides a segment where the model is worse than the process it replaces, so results must be broken out by site, region, channel, product, and any protected attribute in scope.

Deployment readiness includes latency and throughput load-tested at forecast peak volume, end-to-end integration testing, a security review, and a tested rollback path with a named authority to trigger it.

A model that clears its technical metric but has no monitoring plan and no rollback path is not ready — the project manager withholds approval, and conditional approval with a limited rollout is a legitimate outcome alongside approve, defer, and reject.

Last updated: August 2026

The Last Gate Before Production

CPMAI Phase 5, Model Evaluation , decides whether there is anything worth deploying. Phase 6, Model Operationalization, and ECO Domain V then execute the deployment; this task is the approval that precedes it. By now the model has passed the quality and control work of Domain IV Tasks 1 to 3 and the data readiness gate of Task 5. Task 6 asks a harder question: is this fit to be run by an operations team, in front of real customers, for years?

Evaluate Against the Criteria You Already Agreed

The success criteria were fixed in Domain II, Task 8 , during Business Understanding, before anyone had seen a result. This gate measures against those :

The agreed technical metric and its threshold — the one written down, not the one that looks best today.

The business-impact threshold — the operational or financial effect the project promised.

The baseline the model must beat — the current process, the existing rules engine, or a simple heuristic. A model that beats random but not the rules it replaces has produced no value at greater cost and lower transparency.

Metric shopping is the trap the exam sets. The team agreed to recall at a fixed false-positive rate, arrives with a strong area-under-curve figure instead, and explains that the original metric "was not really the right one." Your response is neither to accept the better-looking number nor to accuse anyone: require the model to be reported against the agreed criterion and the baseline, and treat any change of metric as a formal change to the success criteria with the sponsor's approval.

Two translations matter. Convert the metric into business terms at production volume — "94 percent accurate" means nothing until you say what the other 6 percent costs per year. And be explicit about the operating threshold : performance is always reported at a chosen cut-off, and where that cut-off sits is a business trade between false positives and false negatives. The process owner sets it, not the data scientist, and reported results must be the results at that setting.

Robustness and Generalization

The untouched test set. It counts only if it was genuinely held back. If it was used to compare candidates or tune a threshold, it is a second validation set and no honest generalization estimate remains.

Recent data. Report performance on the most recent period, not the average across five years. A model can look fine overall while having quietly stopped working last quarter.

Subgroups. Break results down by site, region, channel, product, customer tenure, and any protected attribute in scope. An aggregate number routinely hides a segment where the model performs worse than the process it replaces.

Degraded and shifted inputs. What happens when a field arrives empty, a feed is late, an unseen category appears, or volume doubles? Whether the system degrades gracefully or fails belongs in the runbook.

Then set expectations honestly: offline performance overstates production performance , essentially always. Prepared data is cleaner than live data, the test set shares its era with the training data, and people behave differently once they know a model is scoring them. Never approve a business case that only works at the offline number.

Deployment Readiness

Infrastructure provisioned and sized, with the cost per prediction known at forecast volume rather than at pilot volume.

Latency and throughput load-tested at production peak. A model that answers in 40 milliseconds on a workstation and four seconds behind a real queue misses the operational window it was funded to hit.

End-to-end integration testing through the upstream feed, the scoring service, the downstream system that consumes the score, and the human workflow that acts on it. Most unpleasant surprises live here, not in the model.

Security review covering access control, endpoint exposure, data in transit and at rest, adversarial or prompt-injection testing where relevant, and the provenance of third-party models and libraries.

A tested rollback path with a known execution time, a named person authorized to trigger it, and written criteria for when they should.

Documentation and Operational Procedures

Model card — intended use, explicitly out-of-scope uses, a description of the training data and its date range, evaluation results overall and by segment, and known limitations.

Monitoring plan — named metrics, thresholds, frequency, and an owner for each alert, covering input drift and, where feasible, actual outcomes rather than predictions alone.

Retraining triggers — what condition initiates retraining, who authorizes it, and how a retrained model is re-validated before it replaces the incumbent.

Runbook — the support team's document: how to tell it is working, what to do at each alert, whom to call, how to fall back to manual processing, how to roll back.

Human oversight procedures — review, override, and appeal paths wherever the model affects people, plus completed training for the staff who will use it. A tool nobody trusts has been installed, not deployed.

The Approval Decision

You assemble the evidence pack: the evaluation report against the original criteria, test-set and subgroup results, robustness testing, load and integration results, the security sign-off, the bias and fairness assessment, the model card, the monitoring plan, the runbook, and the rollback plan. Signatures come from the people who carry the consequences.

Readiness dimension Evidence required Who signs off Blocker if missing

Business success criteria Results against the Domain II Task 8 criteria at the agreed operating threshold, with baseline comparison Sponsor or business owner Without an agreed benchmark the model cannot be shown to add value

Technical performance Untouched test-set results, recent-period results, subgroup breakdown Data science lead A test set used for tuning makes every reported number optimistic

Robustness Behavior under missing fields, unseen categories, feed outage, and shifted volume Data science lead with engineering An unknown failure mode becomes a production incident

Fairness and responsible AI Error-rate comparison across segments, bias assessment, explanations suited to the audience Risk, compliance, or the ethics review body Undetected disparate performance is regulatory and reputational exposure

Infrastructure and performance Load test at forecast peak, measured latency and throughput, cost per prediction Platform or IT owner Clears the metric in the lab and misses the operational window live

Integration End-to-end test through feed, scoring service, consuming system, and human workflow IT with the process owner A score is produced that no process can act on

Security Access control, endpoint exposure, adversarial testing, model and dependency provenance Security lead An exposed endpoint or an untraceable third-party component

Documentation Model card with intended use, limitations, training-data description, dated results PM assembles; data science lead attests No record of what the model is for or where it must not be used

Monitoring Named metrics, thresholds, frequency, alert owners, drift and outcome tracking Operations owner Degradation goes unseen — the classic one-and-done deployment failure

Retraining and rollback Documented triggers and authority; a tested, timed rollback with a named trigger authority Operations owner with the sponsor No way to reverse a bad release

Support readiness Runbook, escalation path, manual fallback, completed user training Service or operations owner The support team cannot diagnose or contain an incident

You then have four ways to close the gate. You can approve outright. You can approve conditionally , releasing the model into a limited rollout — shadow operation, one region, a capped share of volume, or human review on every output — with the conditions named and a review date set. You can defer , holding approval until specific remediation is complete. Or you can reject and send the work back to an earlier phase. Nothing obliges you to pick the first one because the metric cleared.

The Exam Behavior to Internalize

A model that clears its technical metric but has no monitoring plan and no rollback path is not ready , and the project manager withholds approval. AI systems degrade as the world moves away from their training data; deploying without the ability to see that degradation or to reverse the release is the one-and-done deployment failure the entire methodology exists to prevent. Narrowing the rollout is a legitimate conditional approval — but only when the conditions include observability and reversibility. Scoping to a single region without monitoring merely makes the blind spot smaller.