Study guide

5.3 Explainability, Interpretability & Transparency Methods (LIME, SHAP, Datasheets)

PECB Certified ISO/IEC 42001 Lead Implementer Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: AI & AIMS Fundamental Concepts & Frameworks

1.1 Artificial Intelligence Concepts, Terminology & Taxonomy (ISO/IEC 22989 & 23053) 1.2 The ISO/IEC Standards Family for AI (42001, 23894, 5338, 38507, 42005) 1.3 Responsible AI Principles: Fairness, Transparency, Explainability, Safety & Robustness 1.4 Global AI Regulatory Landscape & Governance (EU AI Act, NIST AI RMF, Executive Orders) 1.5 The EU AI Act (Regulation 2024/1689): Risk Tiers, Compliance Obligations & Timeline 1.6 PECB Certification Framework & Lead Implementer Professional Role 1.7 Exam Format, Mechanics & PECB Credentialing Requirements

2 Chapter 2: ISO/IEC 42001 AIMS Core Requirements (Clauses 4-10)

2.1 Annex SL High-Level Structure & Harmonized Management System Principles 2.2 Clause 4: Context of the Organization & Understanding Interested Parties 2.3 Clause 5: Leadership, Commitment & Top Management Responsibilities 2.4 Clause 6 & 7: Planning & Support Infrastructure for AIMS 2.5 Clause 8, 9 & 10: Operation, Performance Evaluation & Continual Improvement Requirements

3 Chapter 3: Planning the AIMS Implementation & AI Risk Management

3.1 AIMS Project Initiation, Implementation Roadmap & Business Case 3.2 AIMS Scope Definition & Organizational Boundaries (Clause 4.3) 3.3 AI Policy Formulation, Leadership Commitment & Governance Structure 3.4 Baseline Gap Analysis & Readiness Assessment 3.5 AI Risk Assessment Methodology (ISO/IEC 23894 & Clause 6.1.2) 3.6 AI System Impact Assessment (Clause 6.1.4 & Annex A.5) 3.7 AI Risk Treatment Planning, Control Selection & Risk Acceptance 3.8 Statement of Applicability (SoA) Development & Justification

4 Chapter 4: Implementing Annex A Controls - Governance, Resources & Lifecycle

4.1 Annex A Overview & Control Architecture (38 Controls Across 9 Objectives A.2–A.10) 4.2 Policies & Internal Organization for AI (Annex A.2 & A.3) 4.3 Resources & Competence for AI Systems (Annex A.4) 4.4 AI System Life Cycle Controls: Objectives, Design & Development (Annex A.6.1–A.6.2.4) 4.5 AI System Verification, Validation, Deployment & Operation (Annex A.6.2.5–A.6.2.9) 4.6 Technical Documentation, Logging & Record-Keeping for AI Systems

5 Chapter 5: Implementing Annex A Controls - Data, Transparency, Use & Third Parties

5.1 Data Governance for AI Systems: Quality, Provenance & Preparation (Annex A.7) 5.2 Bias Mitigation & Fairness Engineering in AI Systems 5.3 Explainability, Interpretability & Transparency Methods (LIME, SHAP, Datasheets) 5.4 Information for Interested Parties & Communication (Annex A.8) 5.5 Use of AI Systems & Responsible Usage Controls (Annex A.9) 5.6 Third-Party Relationships, Vendor Risk & AI Supply Chain (Annex A.10)

6 Chapter 6: Monitoring, Performance Evaluation & Certification Audit Preparation

6.1 AIMS Performance Monitoring, Metrics & Key Performance Indicators (Clause 9.1) 6.2 Data Drift, Concept Drift & AI Model Performance Monitoring 6.3 Internal Audit of the AIMS (ISO 19011 & Clause 9.2) 6.4 Management Review & Operational Suitability Evaluation (Clause 9.3) 6.5 Nonconformity Management, Root Cause Analysis & Corrective Actions (Clause 10) 6.6 Certification Audit Preparation: Stage 1, Stage 2, Surveillance & Recertification

5.3 Explainability, Interpretability & Transparency Methods (LIME, SHAP, Datasheets)

Key Takeaways Explain This Topic AI

Interpretability refers to intrinsically understandable model architectures (e.g., decision trees), whereas explainability uses post-hoc methods for complex models (e.g., deep neural networks).

SHAP (SHapley Additive exPlanations) provides game-theoretic feature attribution grounded in Shapley values, ensuring local accuracy and consistency.

LIME (Local Interpretable Model-agnostic Explanations) builds surrogate linear models around individual predictions to explain complex model behavior locally.

Datasheets for Datasets standardizes documentation regarding dataset motivation, composition, collection process, preprocessing, and recommended uses.

Model Cards document model architecture, intended usage, evaluation metrics, fairness audits, and operational boundaries for transparency.

Last updated: July 2026

5.3 Explainability, Interpretability & Transparency Methods (LIME, SHAP, Datasheets)

ISO/IEC 42001 emphasizes that AI systems must operate transparently, allowing interested parties to understand, audit, and evaluate how automated decisions are generated. Explaining complex AI systems—particularly deep neural networks and ensemble architectures—is a core requirement for risk management, regulatory compliance (such as the GDPR right to explanation), and building stakeholder trust.

Distinguishing Interpretability, Explainability, and Transparency

In AI governance terminology, these three terms refer to distinct dimensions of system legibility:

Transparency: The macro-level availability of information regarding an AI system's design, training data, operational boundaries, governance controls, and performance metrics.

Interpretability (Intrinsic Transparency): The degree to which a human can comprehend the internal decision logic of an AI model directly from its architecture without external mathematical surrogate tools.

Explainability (Post-Hoc Transparency): The capability to generate human-understandable explanations for specific predictions made by complex, non-interpretable ("black-box") models using external diagnostic frameworks.

Intrinsic vs. Post-Hoc Explainability

Model Category Transparency Type Technical Examples Governance & Audit Implications

Intrinsic / Glass-box High natural interpretability Linear Regression, Decision Trees (shallow), Generalized Additive Models (GAMs) Easy to audit directly; decision logic is self-evident; lower predictive capacity on complex non-linear data

Post-hoc Local Explanations Explains single individual predictions LIME, SHAP, TreeSHAP, Integrated Gradients Suitable for explaining specific automated decisions to end-users and regulators

Post-hoc Global Explanations Explains overall model feature importance Global SHAP summary plots, Partial Dependence Plots (PDP), Permutation Feature Importance Essential for model validation, auditing global feature dependence, and verifying system safety

Feature Attribution Frameworks: SHAP and LIME

Post-hoc local explainability relies heavily on feature attribution methods, which assign an importance score to each input feature for a given prediction.

Feature Attribution Method Theoretical Basis Scope Strengths Operational Drawbacks

SHAP (SHapley Additive exPlanations) Game theory (Shapley values from cooperative games) Local and Global Solid mathematical foundation; guarantees efficiency, symmetry, dummy, and additivity properties Computationally expensive for high-dimensional models; requires background sample approximation

LIME (Local Interpretable Model-agnostic Explanations) Local linear surrogate modeling Local Fast computation; completely model-agnostic; intuitive local linear approximation Sampling instability; explanations can vary between identical runs due to local perturbation sampling

Integrated Gradients Axiomatic path integrals of gradients Local Computationally efficient for deep neural networks; satisfies completeness axiom Requires access to model gradients (white-box access); not applicable to decision trees

Deep-Dive: SHAP (Shapley Additive Explanations)

SHAP computes the marginal contribution of each feature across all possible feature subsets (coalitions). The Shapley value ϕ i \phi_i ϕ i ​ for feature i i i is calculated as: ϕ i ( x ) = ∑ S ⊆ F ∖ { i } ∣ S ∣ ! ( ∣ F ∣ − ∣ S ∣ − 1 ) ! ∣ F ∣ ! [ f x ( S ∪ { i } ) − f x ( S ) ] \phi_i(x) = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F| - |S| - 1)!}{|F|!} \left[ f_x(S \cup \{i\}) - f_x(S) \right] ϕ i ​ ( x ) = S ⊆ F ∖ { i } ∑ ​ ∣ F ∣ ! ∣ S ∣ ! ( ∣ F ∣ − ∣ S ∣ − 1 )! ​ [ f x ​ ( S ∪ { i }) − f x ​ ( S ) ]

This game-theoretic foundation guarantees that the sum of feature attributions equals the difference between the model output and the expected baseline prediction (Efficiency property).

Deep-Dive: LIME

LIME generates synthetic perturbations around a target instance x x x , obtains model predictions for these perturbed samples, and trains an interpretable surrogate model g ∈ G g \in G g ∈ G (such as a sparse linear model) by minimizing a loss function L ( f , g , π x ) \mathcal{L}(f, g, \pi_x) L ( f , g , π x ​ ) weighted by proximity measure π x \pi_x π x ​ , plus a complexity penalty Ω ( g ) \Omega(g) Ω ( g ) .

Standardized Artifact Documentation: Datasheets for Datasets

Structural transparency requires standardized documentation for datasets and models. Datasheets for Datasets (Gebru et al.) provides a structured questionnaire framework covering key lifecycle stages:

Motivation: Who created the dataset, why was it created, and who funded it?

Composition: What do the instances represent? Are there missing data or sensitive features?

Collection Process: How was data acquired, what sampling mechanisms were used, and was consent obtained?

Preprocessing / Cleaning: How was raw data transformed, normalized, or anonymized?

Uses: What tasks is the dataset suitable for, and what uses are explicitly discouraged?

Distribution & Maintenance: How is the dataset distributed, updated, and maintained?

Model Cards and System Transparency Artifacts

Model Cards (Mitchell et al.) complement dataset documentation by standardizing operational disclosure for trained model artifacts:

Model Details: Developer, model release date, version, architecture, and type.

Intended Use: Primary intended applications, out-of-scope use cases, and target user personas.

Factors: Demographic, environmental, or technical factors evaluated for performance disparities.

Metrics: Evaluation measures (Accuracy, F1-score, ROC-AUC, Fairness metrics) and selection rationale.

Evaluation Data: Datasets used for validation and testing, including representation details.

Ethical Considerations & Caveats: Operational limitations, sensitive edge cases, and known risks.

Governance Integration for ISO/IEC 42001 Compliance

Lead Implementers must operationalize explainability controls within the AIMS:

Mandate Datasheets for Datasets for all ingested data assets.

Require Model Cards prior to promoting models from staging to production.

Embed automated SHAP / LIME explanation generators into decision workflows requiring explainable outputs (e.g., credit underwriting, medical diagnostics).