Study guide
Define Success Criteria and KPIs
PMI Certified Professional in Managing AI PMI-CPMAI
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: PMI-CPMAI Exam Foundations & the CPMAI Methodology
The PMI-CPMAI Exam: Format, Eligibility, Fees & Scoring The CPMAI Methodology: Six Phases and Data-Centric Iteration CPMAI vs Waterfall, Agile, and CRISP-DM
2 Chapter 2: AI and Machine Learning Fundamentals for CPMAI
The Seven Patterns of AI Supervised, Unsupervised & Reinforcement Learning Why AI Projects Fail: Value, Data Readiness & Right-Sizing
3 Chapter 3: Domain I — Support Responsible and Trustworthy AI Efforts (15%)
Oversee the Privacy and Security Plan Manage AI/ML Transparency in Data and Algorithm Selection Conduct Bias Checks on Model, Data, and Algorithm Monitor Regulatory and Policy Compliance Manage Accountability Documentation and the Audit Trail
4 Chapter 4: Domain II — Identify Business Needs and Solutions, Part 1 (26%)
Identify the Problem to Be Solved Evaluate Initial AI Feasibility Conduct Security, Safety, and Ethics Risk Assessments Develop the AI Project Scope Statement Determine Return on Investment
5 Chapter 5: Domain II — Identify Business Needs and Solutions, Part 2
Manage Adoption and Integration Risks Draft the AI Solution Define Success Criteria and KPIs Support Business Case Creation Identify Project Resources: People, Hardware, and Contractors
6 Chapter 6: Domain III — Identify Data Needs, Part 1 (26%)
Define the Required Data Identify Data Subject-Matter Experts Identify Data Sources and Locations Coordinate the AI Workspace and Infrastructure Gather the Required Data
7 Chapter 7: Domain III — Identify Data Needs, Part 2
Check Data Privacy, Compliance, and Access Oversee Data Evaluation Determine If the Data Meets Solution Needs: The Phase 2 Go/No-Go Convey Data Understanding to Leadership
8 Chapter 8: Domain IV — Manage AI Model Development and Evaluation (16%)
Oversee AI/ML Model Technique and Algorithm Selection Oversee Model QA/QC and Configuration Management Manage AI/ML Model Training Manage Data Transformation and Data Preparation Verify Data Quality for the Data Preparation Go/No-Go Verify the Model Is Ready for Operationalization
9 Chapter 9: Domain V — Operationalize AI Solution, Part 1 (17%)
Manage Creation of the AI Solution Deployment Plan Manage the AI Solution Deployment Oversee Model Governance in Production Oversee AI Solution Metrics and Performance Monitoring
10 Chapter 10: Domain V — Operationalize AI Solution, Part 2
Prepare the Final Report and Lessons Learned Manage the AI Solution Transition Plan Oversee the AI Solution Contingency Plan
Define Success Criteria and KPIs
Key Takeaways Explain This Topic AI
Success criteria come in three layers — model performance, business impact, and adoption or experience — and the project counts as successful only when the business layer moves.
On imbalanced data accuracy misleads: on 100,000 claims of which 500 are fraudulent, a model that flags nothing scores 99.5% accuracy, so precision and recall carry the meaning.
Precision is true positives divided by everything flagged; recall is true positives divided by everything that was truly positive; which one you favor is set by the relative cost of a false positive versus a false negative.
A model must beat the documented incumbent baseline — today's rules engine or human performance measured before Phase 4 — not merely post a respectable score in isolation.
The classification threshold is a business decision owned by whoever carries each error cost and the review capacity, and success criteria must be signed before model development, never written after seeing results.
Last updated: August 2026
Three Layers, and Only One of Them Decides Success
The exam expects a three-layer metric stack. Model metrics say how well the model predicts. Business impact metrics say whether the organization is measurably better off. Adoption and experience metrics say whether people are actually using it. All three go into the success criteria, but only one of them decides: a project with excellent model metrics and no movement in the business layer has failed, however good the demo was. Define all three in CPMAI Phase 1 (Business Understanding) so the Phase 5 (Model Evaluation) go/no-go has something real to test against.
Model Metrics at Management Depth
You do not need to build a classifier, but you must be able to read one. A confusion matrix compares what the model said with what was true. A fraud model scores 100,000 claims for a quarter; 500 of them — 0.5% — are truly fraudulent. The model flags 800:
- Actually fraudulent Actually not fraudulent
- Model flagged 400 (true positives) 400 (false positives)
- Model did not flag 100 (false negatives) 99,100 (true negatives)
Precision = true positives / everything flagged = 400 / 800 = 50% . Half of what the investigators open is a waste of their day.
Recall (sensitivity) = true positives / everything truly fraudulent = 400 / 500 = 80% . Four fraud cases in five are caught.
F1 score = the harmonic mean of precision and recall = about 0.62 . One number for ranking candidate models; useless as a decision.
Accuracy = (400 + 99,100) / 100,000 = 99.5% . Here is the trap: a model that simply answers "not fraudulent" every single time also scores 99.5% accuracy and catches nothing at all. On imbalanced data, accuracy measures the imbalance, not the model — which is why a headline accuracy figure on a rare-event problem should always make you ask for the confusion matrix.
Precision and recall trade against each other. Lower the score threshold and you catch more fraud (recall rises) while opening more innocent claims (precision falls). Which way you lean is decided by the relative cost of the two errors . A missed sepsis case can kill a patient, so recall dominates and false alarms are tolerated up to the point of alarm fatigue. A false decline on an e-commerce payment loses a good customer permanently, so precision matters more. For continuous targets — demand units, remaining useful life, claim cost — the equivalents are MAE (mean absolute error), RMSE (root mean squared error, which punishes large misses harder) and MAPE (mean absolute percentage error, readable by business people but unstable near zero).
Beat the Incumbent, Not an Abstract Standard
Whatever does the job today is your baseline : a rules engine, a scoring sheet, a vendor tool, or an experienced human. Measure it before Phase 4 (Model Development) begins, on the same data and the same definitions the model will be judged on. If the current rules engine catches 65% of fraud at 40% precision, then a model at 80% recall and 50% precision is a genuine gain, and a model at 62% and 38% is an expensive way to get worse — even though 62% sounds respectable in a slide. For human baselines, measure a sample of real decisions rather than asking people how accurate they think they are. Without a documented baseline, every later comparison is a comparison to zero, and the Phase 5 go/no-go becomes a matter of opinion.
The Threshold Is a Business Decision
A classifier outputs a score, not a verdict. Where you cut that score decides how many false positives and false negatives you buy. That cut is not a data-science optimum — the F1-maximizing threshold is a mathematical convenience, not a policy. It is set by whoever owns the cost of each error type, constrained by review capacity. If the special investigations unit can work 200 referrals a day, the threshold is capacity-bound before it is anything else. The project manager's job is to convene the business owner, the risk or compliance owner, and the operations manager; then document the chosen threshold, the expected daily volumes at it, and who approved it, in the scope statement and business case.
Business Impact Metrics and Thresholds
A business metric needs four things: a number, a source system, a measurement window, and an attribution method. "Reduce unplanned downtime" is a direction, not a criterion. "Reduce unplanned downtime hours on the packaging line by 15% within two quarters of go-live, measured in the maintenance management system against the prior four quarters, using the two untreated lines as a control" is a criterion. Attribution matters because business numbers move for many reasons; a holdout group, an A/B split, or a phased rollout with a control site is what lets you claim the improvement was the model's.
Adoption, Experience, and Technical Benchmarks
Carry the Task 6 measures — coverage, override rate, time-to-decision, follow-through — into the success criteria with targets set before go-live, and add a short user-satisfaction instrument with a stated cadence. Set technical benchmarks in the same document: 95th-percentile inference latency inside the workflow's budget, availability target, and cost per thousand predictions. A model that meets its recall target but misses the latency budget has still failed the workflow.
Align Every Criterion to an Organizational Objective
Metric layer Example metric Who owns it Threshold set by whom
Model performance Recall and precision at the operating threshold Data science lead Business owner of the error costs, advised by data science
Technical performance 95th-percentile latency, availability, cost per 1,000 predictions Platform and MLOps engineering IT architecture, against the workflow's latency budget
Business impact Fraud losses avoided per quarter versus baseline Executive sponsor or process owner Executive sponsor, recorded in the business case
Adoption and experience Coverage, override rate, user satisfaction Operations manager for the affected team Change sponsor with the operations manager
Strategic alignment Contribution to the named corporate objective Portfolio or PMO lead Executive leadership
If no named executive owns the business metric you have chosen, it is not a business metric — it is a project vanity metric, and it will not defend the project at the next budget review.
The Trap: Defining Success After You See the Results
When a team finishes Phase 4 with 0.71 recall and then proposes 0.70 as the success criterion, success has been reverse-engineered from the result. Criteria set that way can never fail, teach nothing, and collapse the moment finance asks what improved. Lock model, business, technical, and adoption criteria — each with a threshold and an owner — before development starts, and treat any later change as a documented, sponsor-approved change to the business case.