Study guide
12.2 Collecting Evidence: Interviews, Observation, Documents
PECB ISO/IEC 42001 Lead Auditor AI Management
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Orientation & AI Landscape
1.1 Current Exam Facts & PECB Credential Path 1.2 AI Concepts & ISO/IEC 22989 Terminology 1.3 AI Lifecycle & Trustworthiness Characteristics
2 Chapter 2: ISO AI Family & AIMS Purpose
2.1 ISO/IEC AI Standards Family 2.2 AIMS Purpose, PDCA & Management System Thinking 2.3 Global AI Governance Landscape for Auditors
3 Chapter 3: AIMS Requirements — Context & Leadership
3.1 Annex SL High-Level Structure & Clauses Overview 3.2 Clause 4 Context of the Organization 3.3 Clause 5 Leadership & AI Policy
4 Chapter 4: AIMS Requirements — Planning
4.1 Clause 6.1 Risks, Opportunities & AI Risk Assessment 4.2 Clause 6.1.4 AI System Impact Assessment 4.3 Clause 6.2 AI Objectives & Planning to Achieve Them 4.4 Clause 6.3 Planning of Changes
5 Chapter 5: AIMS Requirements — Support & Operation
5.1 Clause 7 Support 5.2 Clause 8 Operation & Operational Planning and Control 5.3 Clause 8 AI System Lifecycle Controls in Operation
6 Chapter 6: AIMS Requirements — Evaluation & Improvement
6.1 Clause 9 Performance Evaluation 6.2 Clause 10 Improvement 6.3 Integrated AIMS–ISMS Thinking for Auditors
7 Chapter 7: Annex A Controls — Governance & Resources
7.1 Annex A Architecture: 38 Controls Across A.2–A.10 7.2 A.2 Policies Related to AI & A.3 Internal Organization 7.3 A.4 Resources for AI Systems 7.4 A.5 Assessing Impacts of AI Systems
8 Chapter 8: Annex A Controls — Lifecycle & Data
8.1 A.6 AI System Life Cycle — Design, Development, Verification 8.2 A.6 Deployment, Operation, Monitoring & Documentation 8.3 A.7 Data for AI Systems 8.4 Bias, Fairness & Data Quality Audit Focus
9 Chapter 9: Annex A Controls — Transparency, Use & Third Parties
9.1 A.8 Information for Interested Parties 9.2 A.9 Use of AI Systems 9.3 A.10 Third-Party and Customer Relationships
10 Chapter 10: Fundamental Audit Concepts (ISO 19011)
10.1 Seven Audit Principles 10.2 Audit Types, Objectives, Criteria & Scope 10.3 Evidence-Based & Risk-Based Audit Approaches 10.4 Auditor Competence, Ethics & Impartiality
11 Chapter 11: Preparing an ISO/IEC 42001 Audit
11.1 Initiating the Audit & Feasibility 11.2 Determining Audit Objectives, Scope & Criteria 11.3 Document Review & Stage 1 Readiness 11.4 Audit Plan, Team & Working Documents
12 Chapter 12: Conducting an ISO/IEC 42001 Audit
12.1 Opening Meeting & On-Site Kickoff 12.2 Collecting Evidence: Interviews, Observation, Documents 12.3 AI-Specific Evidence: Model Cards, Lineage, Logs, Impact Assessments 12.4 Sampling Strategies & Generating Findings
13 Chapter 13: Closing an ISO/IEC 42001 Audit
13.1 Classifying Nonconformities & Observations 13.2 Audit Conclusions & Closing Meeting 13.3 Writing the Audit Report 13.4 Corrective Action Follow-Up & Verification
14 Chapter 14: Managing the AIMS Audit Program
14.1 Establishing & Implementing an Audit Program 14.2 Program Resources, Risks & Opportunities 14.3 Monitoring, Reviewing & Improving the Program 14.4 Certification Cycle: Surveillance, Recertification & Multi-Standard Audits
12.2 Collecting Evidence: Interviews, Observation, Documents
Key Takeaways Explain This Topic AI
Primary AIMS evidence methods are interviews, observation of AI operations, and review of documents/records—combined through triangulation.
Interview technique must fit the audience: executives for intent and resources; model owners for lifecycle controls; data engineers for lineage and quality; end users for human oversight and use reality.
Observation of monitoring dashboards, approval gates, and ops rituals tests whether written AIMS controls operate in production.
Avoid leading questions; record who said what, what was shown, and which criterion it supports so findings remain verifiable.
Cultural and technical communication gaps (jargon, hierarchy, multi-language teams) are managed with plain language, confirmations, and multi-source corroboration—not assumptions.
Last updated: August 2026
12.2 Collecting Evidence: Interviews, Observation, Documents
Auditor focus: AIMS findings stand or fall on verifiable evidence . A confident interview without records, a polished policy without operation, or a dashboard screenshot without context is incomplete. Use three primary methods together, then triangulate.
What counts as audit evidence
Per ISO 19011 , audit evidence is records, statements of fact, or other information that are relevant to the audit criteria and verifiable . Evidence is evaluated against criteria (ISO/IEC 42001 requirements, applicable Annex A controls, organizational AIMS documented information, and other agreed requirements) to produce findings .
For AI management systems, evidence often mixes:
- Management-system artifacts (policy, risk/impact assessments, objectives, internal audit, management review)
- Technical lifecycle artifacts (model cards, training data docs, evaluation reports, deployment tickets)
- Operational signals (monitoring, human overrides, incidents, drift alerts)
- People evidence (competence, awareness, role understanding, real use practices)
Sufficiency is not “more PDFs.” Sufficiency is enough independent, relevant, reliable information to support a conclusion for the sampled process or system.
Three primary methods
Method What it yields AIMS examples
Interview Understanding of intended process, roles, decisions, and awareness Executive on AI policy intent; model owner on release gates; user on override practice
Observation Whether activities and tools operate as claimed Live model registry walkthrough; monitoring dashboard; human review queue
Document/record review Formalized requirements and historical performance Impact assessments, approval records, evaluation reports, training logs, tickets
Treat any technical verification as controlled observation/document checks—never unconstrained “hacking” of production AI.
Triangulation (non-negotiable)
Triangulation means corroborating one source with another before treating a claim as strong evidence.
Example: A model owner says “every production release needs fairness evaluation.” Corroborate with:
- Documented release procedure and evaluation criteria
- Sample of recent release tickets with attached evaluation reports
- Observation of the CI/CD or registry gate that blocks unapproved promotion
If interviews say “yes,” procedures say “yes,” but the last five releases lack evaluation artifacts, the triangulated finding points to a performance/implementation gap , not conformity.
- Interview techniques by audience
- Executives and AIMS leadership
Goal: intent, resources, accountability, and how AI risk reaches governance. Use open → probe → closed questions (assurance routes, management-review inputs, policy approval dates). Listen for resources and decisions , not slogans—“we take AI ethics seriously” is not evidence; minutes that fund remediation after a fairness incident can be.
Model owners and product owners
Goal: lifecycle control reality—intended use, changes, oversight, monitoring ownership.
Ask for a story of one system end-to-end : problem framing → data → training/evaluation → deployment approval → production monitoring → incident/change. Follow the trail with “show me” requests. Challenge mismatches between intended use statements and how sales or operations actually apply the model.
Data engineers and ML platform teams
Goal: data quality, lineage, access control, reproducibility, and pipeline change control.
Prefer concrete artifacts : dataset version IDs, lineage graphs, labeling guidelines, feature store configs, pipeline run IDs linked to a model version. Avoid abstract “we use best practices.” Ask what happens when training data schema drifts or a toxic subset is discovered post-deployment.
End users and human-in-the-loop operators
Goal: whether A.9 use controls and human oversight exist in the field.
Ask how users are told they are interacting with AI (where required), how they escalate wrong outputs, whether overrides are logged, and whether production pressure overrides policy. End-user interviews often surface shadow AI tools or prompt practices that never appear in the official inventory.
Observation of AI operations and monitoring
Observation is more than a facility walkthrough. For AIMS, observe:
Model registry / deployment console — version lineage, approval status, environment promotion
Monitoring dashboards — performance, data drift, prediction drift, fairness slices (where claimed), latency/error budgets that affect safe operation
- Human oversight queues — review SLAs, sampling of reviewed cases, escalation paths
- Incident / problem tickets for AI failures — classification, containment, learning
- Change advisory or release meetings if they gate high-risk models
When observing dashboards, note what metrics exist, alert thresholds, owners, and recent alert handling —not only that a pretty UI exists. Ask to open a recent alert and follow it to ticket and action. Observation without follow-through is theater.
For remote observation, have the owner navigate while you direct views; prepared recordings may supplement but should not replace interactive sessions for critical systems.
Document and record review
Distinguish documents (say what should happen) from records (show what did happen).
Typical AIMS document/record sets:
- Scope, AI policy, roles, competence matrices
- Risk assessments, AI system impact assessments , Statements of Applicability-style control decisions if used
- Lifecycle procedures: design, V&V, release, monitoring, retirement
- Model/system cards, data sheets, evaluation reports
- Training/awareness records for AI-related roles
- Internal audit and management review outputs covering AIMS
- Third-party contracts and due diligence for supplier models/APIs
Review for currency, approval, consistency with inventory, and linkage . An impact assessment dated two years before a major use-case change is a red flag. A model card that claims monitoring the organization cannot show live is another.
- Recording evidence and avoiding leading questions
- Working notes discipline
For each meaningful item, capture:
- Date/time, location (physical/virtual), participants
- Criterion referenced (clause/control/procedure)
- What was said / shown / sampled (IDs, versions, ticket numbers)
- Your evaluation (conformity, potential NC, need more evidence)
- Cross-references to files saved in the working paper pack
Write facts , not adjectives: “Release ticket DEP-4412 for model recsys-v3.2 has no attached fairness report; procedure REL-AI-02 §4.2 requires one for high-impact ranking models” beats “sloppy release process.”
Avoid leading questions
Weak (leading) Stronger
“You always run bias tests before deploy, right?” “What evaluation steps are required before production deploy for this system?”
“I’m sure monitoring covers drift?” “Which drift signals are monitored, who owns alerts, and what is the last alert you handled?”
“You wouldn’t deploy without approval, would you?” “Walk me through the last production promotion—who approved and where is it recorded?”
Leading questions invite social compliance and pollute evidence. Use open stems, then probe, then confirm with closed questions only after the interviewee has stated a process.
Cultural and technical communication challenges
AIMS audits cross engineering, business, and language cultures. Manage jargon with plain-language definitions local to the organization; hierarchy by interviewing some practitioners without the full management chain when appropriate; production fear with read-only paths; multi-language teams with interpreters and restated confirmations; and vendor black boxes by shifting to contracts, diligence, output monitoring, and residual risk acceptance. Stay courteously skeptical—restate unclear answers instead of leading or ambushing.
Putting the methods together
For a high-impact system, a practical day pattern is: brief owner interview → pull impact assessment, model card, evaluation, and release procedure → observe registry, monitoring, and one recent release ticket → follow gaps with data/ops interviews → end-of-day triangulation note (sufficient / need more / potential finding). That cadence turns three methods into one coherent evidence story for AI-specific artifacts and formal findings.