14.3 Monitoring, Reviewing & Improving the Program

PECB ISO/IEC 42001 Lead Auditor AI Management

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Orientation & AI Landscape

1.1 Current Exam Facts & PECB Credential Path 1.2 AI Concepts & ISO/IEC 22989 Terminology 1.3 AI Lifecycle & Trustworthiness Characteristics

2 Chapter 2: ISO AI Family & AIMS Purpose

2.1 ISO/IEC AI Standards Family 2.2 AIMS Purpose, PDCA & Management System Thinking 2.3 Global AI Governance Landscape for Auditors

3 Chapter 3: AIMS Requirements — Context & Leadership

3.1 Annex SL High-Level Structure & Clauses Overview 3.2 Clause 4 Context of the Organization 3.3 Clause 5 Leadership & AI Policy

4 Chapter 4: AIMS Requirements — Planning

4.1 Clause 6.1 Risks, Opportunities & AI Risk Assessment 4.2 Clause 6.1.4 AI System Impact Assessment 4.3 Clause 6.2 AI Objectives & Planning to Achieve Them 4.4 Clause 6.3 Planning of Changes

5 Chapter 5: AIMS Requirements — Support & Operation

5.1 Clause 7 Support 5.2 Clause 8 Operation & Operational Planning and Control 5.3 Clause 8 AI System Lifecycle Controls in Operation

6 Chapter 6: AIMS Requirements — Evaluation & Improvement

6.1 Clause 9 Performance Evaluation 6.2 Clause 10 Improvement 6.3 Integrated AIMS–ISMS Thinking for Auditors

7 Chapter 7: Annex A Controls — Governance & Resources

7.1 Annex A Architecture: 38 Controls Across A.2–A.10 7.2 A.2 Policies Related to AI & A.3 Internal Organization 7.3 A.4 Resources for AI Systems 7.4 A.5 Assessing Impacts of AI Systems

8 Chapter 8: Annex A Controls — Lifecycle & Data

8.1 A.6 AI System Life Cycle — Design, Development, Verification 8.2 A.6 Deployment, Operation, Monitoring & Documentation 8.3 A.7 Data for AI Systems 8.4 Bias, Fairness & Data Quality Audit Focus

9 Chapter 9: Annex A Controls — Transparency, Use & Third Parties

9.1 A.8 Information for Interested Parties 9.2 A.9 Use of AI Systems 9.3 A.10 Third-Party and Customer Relationships

10 Chapter 10: Fundamental Audit Concepts (ISO 19011)

10.1 Seven Audit Principles 10.2 Audit Types, Objectives, Criteria & Scope 10.3 Evidence-Based & Risk-Based Audit Approaches 10.4 Auditor Competence, Ethics & Impartiality

11 Chapter 11: Preparing an ISO/IEC 42001 Audit

11.1 Initiating the Audit & Feasibility 11.2 Determining Audit Objectives, Scope & Criteria 11.3 Document Review & Stage 1 Readiness 11.4 Audit Plan, Team & Working Documents

12 Chapter 12: Conducting an ISO/IEC 42001 Audit

12.1 Opening Meeting & On-Site Kickoff 12.2 Collecting Evidence: Interviews, Observation, Documents 12.3 AI-Specific Evidence: Model Cards, Lineage, Logs, Impact Assessments 12.4 Sampling Strategies & Generating Findings

13 Chapter 13: Closing an ISO/IEC 42001 Audit

13.1 Classifying Nonconformities & Observations 13.2 Audit Conclusions & Closing Meeting 13.3 Writing the Audit Report 13.4 Corrective Action Follow-Up & Verification

14 Chapter 14: Managing the AIMS Audit Program

14.1 Establishing & Implementing an Audit Program 14.2 Program Resources, Risks & Opportunities 14.3 Monitoring, Reviewing & Improving the Program 14.4 Certification Cycle: Surveillance, Recertification & Multi-Standard Audits

14.3 Monitoring, Reviewing & Improving the Program

Key Takeaways Explain This Topic AI

Programme monitoring uses KPIs such as plan attainment, NC closure/verification rates, auditor performance, coverage of high-risk AI systems, and auditee feedback

Management review of the audit programme evaluates whether objectives and extent remain suitable as the AI estate and risks change

Continual improvement applies PDCA Act-phase changes: competence, methods, sampling focus, procedures, and tools

Lessons learned from AIMS audits (recurring impact-assessment gaps, third-party AI weaknesses, evidence access failures) should feed the next programme cycle

Improvement actions must be recorded and verified—informal hallway fixes are not programme management
Last updated: August 2026

14.3 Monitoring, Reviewing & Improving the Program

Establishing an AIMS audit programme is the Plan phase; running audits is Do. ISO 19011 Clause 5 also requires the programme to be monitored, reviewed, and improved . Without Check–Act, the programme drifts: schedules slip, the same Annex A weaknesses recur, and competence falls behind new generative-AI practices.

This section is about managing the programme , not about Clause 9.1 monitoring of the AIMS itself—though programme outputs feed AIMS management review.

Why programme monitoring matters for AI

AI estates change faster than many traditional management systems. A programme that was adequate when the organization had three classical ML models may be inadequate after multi-region LLM deployment. Monitoring answers:

Are we executing the audits we committed to?

Are audits producing reliable, timely insight on high-risk AI?

Are auditors performing and remaining competent?

Are findings closed and verified, or stacking up?

Is the programme still aligned with risk, scope, and stakeholder needs?

KPIs for an AIMS audit programme

Select a balanced set—delivery, quality, coverage, and learning. Avoid vanity metrics (pages of report) that ignore assurance value.

Example quarterly dashboard: plan attainment 83% (target ≥90%); high-risk AI sampled 12/15; majors overdue verification 3; audits with full competence 7/9; mean auditee feedback 4.1/5.

Exam trap: High plan attainment with zero sampling of high-risk AI systems is not success. Coverage and risk alignment matter as much as calendar compliance.

Using KPIs without weaponizing them
Investigate root causes (resource cuts, access denial, poor scoping) before blaming auditors alone

Do not reward “zero NCs” as a KPI—that creates under-reporting pressure and violates integrity/fair presentation culture

Compare like with like: Stage 1 readiness audits vs deep Stage 2 implementation audits differ
Monitoring activities (ongoing Check)

Programme managers should routinely:

Track schedule vs actual and reforecast remaining year capacity
Review NC and opportunity logs across audits for systemic themes
Sample audit reports/working papers for consistency with ISO 19011 principles and AIMS criteria
Verify that team appointments matched the competence matrix
Monitor impartiality disclosures and complaint handling
Confirm follow-up audits/verifications are scheduled, not optional

Watch external signals: major product launches, incidents, regulatory inquiries, M&A—triggers for programme adjustment

For CBs , monitoring also includes scheme-level consistency, auditor calibration, and accreditation body expectations under ISO/IEC 17021-1.

Management review of the audit programme

Periodically (e.g., annual, with interim updates), top management (internal programme) or CB governance reviews the programme. This is distinct from—but inputs to—the organization’s AIMS Clause 9.3 management review .

Typical programme review inputs
KPI results and trends
Achievement of programme objectives
Changes in AI estate size, complexity, risk, and certified/declared scope
Resource adequacy and competence gaps
Status of programme-level risks and opportunities
Feedback from auditees, clients, regulators, accreditation (as applicable)
Results of external assessments of the audit function/CB
Follow-up status of previous programme improvement actions
Typical outputs
Revised objectives and extent (more/fewer audits; new thematic AI focus)
Resource decisions (hires, training budget, tools)
Procedure updates (remote audit rules, confidentiality for model IP)
Schedule changes and special-audit criteria refinements
Competence development plans
Decisions on combined multi-standard approaches

AIMS scenario: KPIs show 95% plan attainment but only 40% of high-risk systems sampled because inventory was outdated. Management review expands extent, funds two additional AI-competent auditors, and requires monthly inventory reconciliation with product management—not merely “try harder next year.”

Continual improvement of the programme

Improvement is the Act phase: deliberate change based on evidence.

Improvement lever Example action after AIMS lessons
Competence Add generative-AI evaluation module to auditor CPD; pair junior auditors with ML experts
Methods Standardize hybrid evidence protocols for model registries and monitoring dashboards
Sampling design Shift from equal time per department to risk-weighted AI system sampling
Procedures New trigger list for special audits after major model launches or serious AI incidents
Working documents Better checklists for impact assessments, data lineage, human oversight records
Integration Pilot combined AIMS–ISMS surveillance with clear dual reporting
Technology Secure evidence vault with retention and access controls suitable for training-data extracts

Document improvement actions with owners and due dates. Verify effectiveness in the next monitoring cycle (Did high-risk coverage rise? Did report cycle time fall without quality loss?).

Lessons learned from AIMS audits

Cross-audit learning is a programme asset. Common AIMS themes that should drive improvement:

Impact assessments exist as templates but not for all high-risk systems → thematic audit + earlier Stage 1 challenge of inventory vs assessments

Third-party foundation models weakly controlled (A.10) → add supplier AI audit stream; require contract evidence packages

Monitoring metrics not linked to intended use or fairness concerns → update sampling to include metric design interviews, not only dashboard screenshots

Confidentiality incidents with model IP in working papers → tighten evidence-handling procedure and tools
Combined ISMS auditors missing AI lifecycle depth → mandate AIMS-specific competence before solo leading

NCs closed with policy rewrites only → programme rule: effectiveness verification requires operational evidence

Facilitate short lessons-learned sessions after major certification cycles or annually for internal programmes. Capture what to start/stop/continue in the programme plan.

Linking programme improvement to organizational improvement

Do not confuse the two loops:

Loop Object Owner focus
Audit programme improvement How we audit (schedule, competence, methods, KPIs) Programme manager / CB
AIMS improvement (Clause 10) How the organization manages AI Auditee management

Programme outputs (findings trends, coverage gaps) inform AIMS improvement and management review, but rewriting the auditee’s AI policy is not “improving the audit programme.” Conversely, training auditors is not a substitute for the auditee fixing nonconformities.

Practical monitoring calendar Monthly: schedule status, open NC aging, competence conflicts for upcoming audits

Quarterly: KPI dashboard, risk register refresh, inventory delta review
After each major audit: file lessons, report QA sample, auditee feedback
Annually: full management review of programme; update objectives/extent/resources
Triggered: special review after serious AI incident, accreditation finding, or mass product launch
Exam focus

Domain 7 expects you to name programme KPIs , distinguish programme management review from a single audit’s closing meeting, and show how lessons from AIMS audits change next year’s competence, sampling, and special-audit triggers. Monitoring without improvement is incomplete; improvement without data is guesswork. Together they keep the AIMS audit programme trustworthy as AI systems evolve.