Study guide
12.3 AI-Specific Evidence: Model Cards, Lineage, Logs, Impact Assessments
PECB ISO/IEC 42001 Lead Auditor AI Management
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Orientation & AI Landscape
1.1 Current Exam Facts & PECB Credential Path 1.2 AI Concepts & ISO/IEC 22989 Terminology 1.3 AI Lifecycle & Trustworthiness Characteristics
2 Chapter 2: ISO AI Family & AIMS Purpose
2.1 ISO/IEC AI Standards Family 2.2 AIMS Purpose, PDCA & Management System Thinking 2.3 Global AI Governance Landscape for Auditors
3 Chapter 3: AIMS Requirements — Context & Leadership
3.1 Annex SL High-Level Structure & Clauses Overview 3.2 Clause 4 Context of the Organization 3.3 Clause 5 Leadership & AI Policy
4 Chapter 4: AIMS Requirements — Planning
4.1 Clause 6.1 Risks, Opportunities & AI Risk Assessment 4.2 Clause 6.1.4 AI System Impact Assessment 4.3 Clause 6.2 AI Objectives & Planning to Achieve Them 4.4 Clause 6.3 Planning of Changes
5 Chapter 5: AIMS Requirements — Support & Operation
5.1 Clause 7 Support 5.2 Clause 8 Operation & Operational Planning and Control 5.3 Clause 8 AI System Lifecycle Controls in Operation
6 Chapter 6: AIMS Requirements — Evaluation & Improvement
6.1 Clause 9 Performance Evaluation 6.2 Clause 10 Improvement 6.3 Integrated AIMS–ISMS Thinking for Auditors
7 Chapter 7: Annex A Controls — Governance & Resources
7.1 Annex A Architecture: 38 Controls Across A.2–A.10 7.2 A.2 Policies Related to AI & A.3 Internal Organization 7.3 A.4 Resources for AI Systems 7.4 A.5 Assessing Impacts of AI Systems
8 Chapter 8: Annex A Controls — Lifecycle & Data
8.1 A.6 AI System Life Cycle — Design, Development, Verification 8.2 A.6 Deployment, Operation, Monitoring & Documentation 8.3 A.7 Data for AI Systems 8.4 Bias, Fairness & Data Quality Audit Focus
9 Chapter 9: Annex A Controls — Transparency, Use & Third Parties
9.1 A.8 Information for Interested Parties 9.2 A.9 Use of AI Systems 9.3 A.10 Third-Party and Customer Relationships
10 Chapter 10: Fundamental Audit Concepts (ISO 19011)
10.1 Seven Audit Principles 10.2 Audit Types, Objectives, Criteria & Scope 10.3 Evidence-Based & Risk-Based Audit Approaches 10.4 Auditor Competence, Ethics & Impartiality
11 Chapter 11: Preparing an ISO/IEC 42001 Audit
11.1 Initiating the Audit & Feasibility 11.2 Determining Audit Objectives, Scope & Criteria 11.3 Document Review & Stage 1 Readiness 11.4 Audit Plan, Team & Working Documents
12 Chapter 12: Conducting an ISO/IEC 42001 Audit
12.1 Opening Meeting & On-Site Kickoff 12.2 Collecting Evidence: Interviews, Observation, Documents 12.3 AI-Specific Evidence: Model Cards, Lineage, Logs, Impact Assessments 12.4 Sampling Strategies & Generating Findings
13 Chapter 13: Closing an ISO/IEC 42001 Audit
13.1 Classifying Nonconformities & Observations 13.2 Audit Conclusions & Closing Meeting 13.3 Writing the Audit Report 13.4 Corrective Action Follow-Up & Verification
14 Chapter 14: Managing the AIMS Audit Program
14.1 Establishing & Implementing an Audit Program 14.2 Program Resources, Risks & Opportunities 14.3 Monitoring, Reviewing & Improving the Program 14.4 Certification Cycle: Surveillance, Recertification & Multi-Standard Audits
12.3 AI-Specific Evidence: Model Cards, Lineage, Logs, Impact Assessments
Key Takeaways Explain This Topic AI
AIMS audits require distinctive artifacts—model/system cards, training data documentation, lineage, evaluation reports, impact assessments, oversight logs, deployment approvals, third-party terms, and drift/incident tickets.
Each artifact maps to ISO/IEC 42001 clause themes and Annex A control areas (e.g., impact assessment, lifecycle, data, transparency, use, third parties).
Sufficiency tests ask whether artifacts are current, approved, linked to the correct system version, and consistent with observed operations.
Missing linkage between impact assessment conclusions and release/monitoring design is a classic AIMS nonconformity pattern.
Third-party model terms and residual-risk acceptance must be evidenced when black-box components limit direct technical inspection.
Last updated: August 2026
12.3 AI-Specific Evidence: Model Cards, Lineage, Logs, Impact Assessments
Auditor focus: ISO/IEC 42001 is a management system standard, but its operational proof often sits in AI lifecycle artifacts . Learn what “good enough” looks like for each artifact type, which requirements it supports, and how to test sufficiency without becoming a full-time ML researcher.
Why AIMS evidence looks different
AIMS audits still need policies, procedures, and training records—but trustworthiness (reliability, safety, security, privacy, fairness, transparency, accountability) is proven through system-specific technical and socio-technical records . An AI policy PDF plus org chart is not enough. Names vary; map substance to criteria.
Artifact catalog and mapping
Artifact What it typically shows Primary hooks (illustrative)
Model / system card Intended use, limitations, performance summary, ethics/risk notes, contact owners Transparency (A.8), lifecycle documentation (A.6), operational control (Clause 8)
Training / evaluation data documentation Sources, rights, quality, known biases, splits, refresh rules Data for AI (A.7), risk/impact inputs (6.1 / A.5)
Data lineage graph / pipeline records Trace from sources → features → datasets → model versions A.7, change control, reproducibility under Clause 8
Evaluation reports Performance, robustness, fairness/slice metrics, safety tests V&V in A.6, objectives (6.2), monitoring design (9.1)
AI system impact assessment Impacts on individuals/groups, severity, mitigations, residual risk Clause 6.1.4 themes, A.5 controls
Human oversight logs Reviews, overrides, escalations, sampling of supervised decisions A.9 use, operational control, monitoring
Deployment approval gates Who authorized go-live/change; criteria checked A.6 deployment, Clause 8 operational planning & control
Third-party model / API terms & diligence Allowed uses, data handling, SLAs, residual risk acceptance A.10 third parties, Clause 8 external processes
Drift / incident tickets Detection, response, learning, possible model rollback Clause 9–10, A.6 monitoring/operation, improvement
Mappings are risk- and SoA-dependent . Always tie evidence to the organization’s applicable controls and documented processes—not to an invented universal checklist.
Model and system cards
Model cards (and broader system cards ) summarize purpose, evaluation, and limits. Test: existence for sampled systems (not only demos); intended use vs. real product behavior; version binding to production; honest limitations; and operational linkage—monitoring and oversight claimed on the card must exist live. A glossy marketing card without the described tests is weak control evidence and a transparency red flag.
Training data documentation and lineage
Data documentation should answer: Where did data come from? Under what rights? How labeled? How quality-checked? How often refreshed? What sensitive attributes are present or proxied?
Lineage connects raw sources to training sets, features, and model versions. Sufficiency tests:
Can the organization identify which data version trained which model version ?
Are exclusion rules (e.g., remove minors, remove expired consents) evidenced in pipeline configs or jobs—not only in a slide?
When a data defect is found, can they trace affected models and open impact/incident handling?
Missing lineage for a high-impact model is rarely a minor paperwork nit; it undermines corrective action, reproducibility, and accountability.
Evaluation reports (performance, fairness, safety)
Evaluation evidence should fit intended use and risk . A chatbot, a medical triage assist tool, and a hiring ranker need different evaluation emphasis. Auditor questions:
Were tests run on representative data , including relevant slices?
Are metrics defined and thresholds justified (not only “accuracy improved”)?
For fairness claims, are group definitions, metrics, and limitations explicit?
Did evaluation occur before production promotion, and after material changes?
Who reviewed and accepted residual risk when thresholds were missed?
Beware one-time launch reports with no re-evaluation after major data or prompt changes. Also beware metric shopping: beautiful offline scores that ignore production distribution shift.
Impact assessment reports
AI system impact assessment (aligned with Clause 6.1.4 and Annex A.5 themes) examines effects on individuals, groups, and other interested parties—beyond pure model accuracy. Sufficiency tests:
- Test Pass signals Fail signals
- Coverage In-scope high-impact systems assessed Only pilot toys assessed; production systems skipped
- Timing Before deployment / material change Retroactive paperwork after go-live
Substance Impacts, likelihood/severity thinking, affected parties, mitigations Generic template with no system-specific content
Trace-through Mitigations appear in design, oversight, monitoring, user info Mitigations listed but not implemented
Review Ownership, approval, revisit triggers Undated, unowned, never updated after incidents
Auditors should sample impact assessment → control implementation linkage: if the assessment requires human review of high-stakes denials, oversight logs must show that review occurring.
Human oversight logs and deployment approval gates
Oversight logs prove that human control is not theoretical. Look for case IDs, reviewer identity/role, decision (accept/override), rationale where required, timestamps, and quality sampling of reviews. Empty queues, 100% auto-accept with “rubber stamp” reviewers, or logs that cannot be retrieved for the sample period are warning signs.
Deployment approval gates (CAB tickets, MLOps policy checks, signed release checklists) show operational planning and control. Sample recent promotions:
Required artifacts attached (eval, impact, security/privacy sign-off as applicable)?
Approver independence proportionate to risk?
Emergency hotfixes documented with retrospective controls?
A perfect procedure with zero gated releases in the ticket system for a “highly controlled” model is a triangulation failure.
Third-party model terms and residual risk
Foundation models, APIs, and vendor platforms limit white-box inspection. Evidence shifts to:
- Contracts / DPAs / acceptable use and data retention terms
- Vendor due diligence and ongoing monitoring of provider notices
- Organization-side evaluation of outputs for the intended use
- Documented residual risk acceptance when provider opacity remains
- Exit / contingency if the provider changes model behavior or terms
“We use Vendor X, so ISO 42001 doesn’t apply” is not a valid exclusion pattern for systems the organization provides or uses under its AIMS scope. A.10 and Clause 8 external process control still apply to what the organization can and must control.
Drift/incident tickets and sufficiency tests
Production AI fails via data/concept drift, prompt abuse, retrieval corruption, and silent decay. Tickets should show detection source, AI-harm severity (not only uptime), containment (rollback, human-only mode), root cause across data/model/process/vendor, and Clause 10 learning. Claimed drift monitoring with never-opened tickets despite metric breaches is not effective.
For each critical artifact ask: relevant to criterion and system; authentic from systems of record; current to the production version; complete enough for risk level; consistent with interviews/observation; actionable when defects appear. Failures become potential NCs or need-more-evidence—criteria vs. evidence, not personal criticism.
Common NC patterns involving AI artifacts
Frequent patterns: model cards for demos only; generic impact assessments disconnected from monitoring; offline-only evaluations with no production review; lineage broken after platform migration; oversight logs shorter than stated retention; vendor models adopted via shadow SaaS with no A.10 trail. Mastering these artifacts lets you audit AIMS with ISO 19011 rigor in the language of AI delivery teams.