10.3 Testing Control Design & Operating Effectiveness

Certified in Risk and Information Systems Control CRISC

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Exam Overview & Risk Foundations

1.1 About the CRISC Exam 1.2 Exam Day Strategy & Question Analysis

2 Chapter 2: Enterprise Governance & Organizational Strategy

2.1 Enterprise Governance of IT & Strategic Alignment 2.2 Organizational Structures, Roles & RACI Matrix 2.3 The Three Lines Model & Assurance Coordination 2.4 Legal, Regulatory & Contractual Compliance

3 Chapter 3: Risk Governance Frameworks & ERM

3.1 Enterprise Risk Management Frameworks 3.2 Risk Appetite, Tolerance & Capacity 3.3 IT Risk Strategy Alignment with Business Objectives

4 Chapter 4: Risk Culture, Ethics & Policy Architecture

4.1 Risk Culture, Tone at the Top & Accountability 4.2 Ethics, Integrity & ISACA Code of Professional Ethics 4.3 Risk Policy, Standards, Guidelines & Procedures Architecture

5 Chapter 5: Asset Management & Business Resilience Governance

5.1 Asset Identification, Valuation & Data Classification 5.2 Data Ownership, Custodianship & Governance Roles 5.3 Business Resilience Governance & Strategic Alignment 5.4 Data Privacy Governance & Global Mandates

6 Chapter 6: IT Risk Identification & Threat Modeling

6.1 Risk Identification Techniques 6.2 Threat Landscape Analysis, Threat Actors & Attack Vectors 6.3 Vulnerability Assessment & Exposure Analysis 6.4 Risk Scenario Development & Threat Modeling

7 Chapter 7: Risk Analysis Methodologies & Evaluation

7.1 Qualitative Risk Analysis & Risk Matrices 7.2 Quantitative Risk Analysis & FAIR Framework 7.3 Semi-Quantitative Analysis & Monte Carlo Simulation 7.4 Risk Evaluation, Ranking & Prioritization Criteria

8 Chapter 8: Risk Register, Aggregation & BIA Integration

8.1 Risk Register Architecture, Maintenance & Lifecycle 8.2 Risk Aggregation, Correlation & Interdependencies 8.3 Business Impact Analysis (BIA) Integration

9 Chapter 9: Risk Treatment Strategies & Cost-Benefit Analysis

9.1 Risk Response Options 9.2 Cost-Benefit Analysis & Return on Security Investment (ROSI) 9.3 Developing & Implementing Risk Action Plans 9.4 Managing Residual Risk & Secondary Risk

10 Chapter 10: Control Design, Selection & Assessment

10.1 Control Types & Categories 10.2 Control Design Principles & Defense-in-Depth 10.3 Testing Control Design & Operating Effectiveness 10.4 Remediation of Control Deficiencies & Exceptions

11 Chapter 11: Third-Party & Supply Chain Risk Management

11.1 Third-Party Risk Management Lifecycle & Due Diligence 11.2 Vendor Contracts, SLAs & Right-to-Audit Clauses 11.3 Supply Chain Risk Management & Software Bill of Materials (SBOM) 11.4 Cloud Service Provider Governance & Shared Responsibility

12 Chapter 12: Risk Monitoring, KRIs & Executive Reporting

12.1 Key Risk Indicators (KRIs), Thresholds & Triggers 12.2 Aligning KRIs with KPIs & Business Performance Metrics 12.3 Stakeholder Risk Reporting & Governance Communication 12.4 Continuous Risk Monitoring & Automated Telemetry

13 Chapter 13: Enterprise IT Architecture & Operations

13.1 Enterprise Architecture & IT Governance Alignment 13.2 Cloud Computing & Virtualization Architecture Risk 13.3 Systems Resilience, High Availability & Infrastructure Redundancy 13.4 Technology Roadmaps, IT Portfolio & Project Risk Governance

14 Chapter 14: Information Security Principles, IAM & SSDLC

14.1 Information Security Principles & The CIA Triad 14.2 Identity & Access Management (IAM), PAM & RBAC 14.3 Cryptography, Encryption & Key Management (KMS) 14.4 Secure Software Development Lifecycle (SSDLC) & DevSecOps

15 Chapter 15: Incident Response, Resilience & Emerging Tech Risk

15.1 Incident Response Lifecycle & Containment Strategies 15.2 Business Continuity & Disaster Recovery Planning and Testing 15.3 Emerging Technologies Risk (AI/LLM, Quantum, IoT/OT) 15.4 Risk Program Maturity Assessment & Continuous Improvement

10.3 Testing Control Design & Operating Effectiveness

Key Takeaways Explain This Topic AI

Control assessment must evaluate two distinct dimensions: Design Adequacy (can the control work?) and Operating Effectiveness (did the control work consistently over time?).

The four primary testing techniques in increasing order of evidentiary rigor are Inquiry, Observation, Inspection (Examination), and Reperformance.

Control walkthroughs trace a single transaction end-to-end to validate process flows and confirm that designed control points exist.

Automated controls require testing the underlying logic and IT General Controls (ITGC) change management, allowing smaller sample sizes than manual controls.

Statistical and non-statistical attribute sampling for manual controls must be proportional to control frequency (e.g., 25-40 samples for daily manual controls).

Last updated: August 2026

10.3 Testing Control Design & Operating Effectiveness

An enterprise cannot assume that because a security control was documented in policy or configured during initial system deployment, it continues to provide the intended risk mitigation over time. Control assessment is the systematic, evidence-based evaluation of internal controls to determine whether they are properly designed ( Design Adequacy ) and functioning reliably over an extended operational period ( Operating Effectiveness ).

Risk practitioners, internal auditors, and control assessors must execute evaluations in alignment with ISACA guidelines, the ISACA IT Audit Framework (ITAF) , AICPA SOC 1 / SOC 2 (SSAE 18/21) , and PCAOB Auditing Standards (AS 2201) . +-----------------------------------------------------------------------------+

• | DESIGN ADEQUACY VS. OPERATING EFFECTIVENESS |
• | |
• | EVALUATION DIMENSION 1: DESIGN ADEQUACY (DESIGN EFFECTIVENESS) |
• | - Question: If operated as specified, is the control capable of meeting |
• | the control objective and mitigating the target risk? |
• | - Focus: Architecture, Policy, Configuration Specifications, Logic |
• | - Primary Technique: Walkthroughs, Documentation Inspection, Inquiry |
• | |
• | EVALUATION DIMENSION 2: OPERATING EFFECTIVENESS |
• | - Question: Did the control operate consistently, reliably, and as |
• | designed throughout the entire assessment period? |
• | - Focus: Execution evidence, Populated artifacts, Timestamps, Exceptions |
• | - Primary Technique: Sampling, Artifact Inspection, Reperformance |
• +-----------------------------------------------------------------------------+
• 1. The Four Primary Control Testing Techniques

Assessors utilize four primary testing techniques, which vary significantly in their evidentiary strength, cost, and reliability. +-----------------------------------------------------------------------------+

• | HIERARCHY OF CONTROL TESTING TECHNIQUES |
• | |
• | STRENGTH: HIGH ^ |
• | | [4. REPERFORMANCE] |
• | | - Assessor independently recalculates or executes flow |
• | | - Highest audit reliability; proves operational truth |
• | | ---------------------------------------------------- |
• | | [3. INSPECTION / EXAMINATION] |
• | | - Reviewing historical tickets, logs, configs, approvals|
• | | - Strong objective evidence of past operating state |
• | | ---------------------------------------------------- |
• | | [2. OBSERVATION] |
• | | - Watching personnel execute the control in real time |
• | | - Point-in-time limitation; subject to Hawthorne effect|
• | | ---------------------------------------------------- |
• | | [1. INQUIRY] |
• | | - Interviewing personnel and control operators |
• | STRENGTH: LOW + - Weakest evidence; MUST be corroborated by artifacts |
• +-----------------------------------------------------------------------------+
• 1. Inquiry

Inquiry involves seeking information from knowledgeable personnel inside or outside the organization through interviews, questionnaires, or written statements.

Evidentiary Weight: Lowest. Inquiry alone can never provide sufficient audit assurance of operating effectiveness because verbal statements are subjective and uncorroborated.

Appropriate Use: Preliminary scoping, understanding control design intent, and corroborating other audit findings.

2. Observation

Observation consists of viewing a process or procedure being performed by authorized staff in real time (e.g., watching an administrator perform a tape backup rotation or observing physical badge access at a data center portal).

Evidentiary Weight: Moderate.

Critical Limitations: Observation only validates control operation at the exact point in time when the assessor is present. Furthermore, it is subject to the Hawthorne Effect (personnel modify behavior and perform controls perfectly when they know they are being observed).

3. Inspection (Examination)

Inspection involves the detailed examination of records, historical documents, system configurations, automated log entries, ticket approvals, reconciliation sheets, or physical assets.

Evidentiary Weight: High.

Appropriate Use: Validating historical operating effectiveness over a multi-month or annual period by examining physical and electronic evidence generated during normal operations.

4. Reperformance

Reperformance involves the independent execution by the assessor of procedures, calculations, or automated workflows that were originally performed as part of the entity's internal control.

Evidentiary Weight: Highest.

Appropriate Use: Re-running automated reconciliation scripts, re-calculating risk scores, or attempting unauthorized access to test if automated blocking triggers as documented.

2. Control Walkthroughs

A Control Walkthrough is a specialized assessment technique where the assessor traces a single transaction, user request, or data flow from its initial origination, through all processing steps, authorization gates, and data transformations, to its final disposition in reports or general ledgers. +-----------------------------------------------------------------------------+

• | THE CONTROL WALKTHROUGH LIFECYCLE |
• | |
• | [1. ORIGINATION] --> User submits access request ticket in ITSM portal |
• | [2. AUTH GATE] --> Manager approval digital signature validated |
• | [3. SOD CHECK] --> Automated conflict check against ERP roles executed |
• | [4. PROVISIONING] --> IAM engine creates account in Active Directory |
• | [5. LOGGING] --> SIEM logs account creation event with timestamp |
• +-----------------------------------------------------------------------------+

Primary Purpose: Walkthroughs confirm the assessor's understanding of the end-to-end process, verify that documented control points exist in reality, and evaluate Design Adequacy .

Limitation: A walkthrough tests only a sample size of one transaction; it cannot prove year-long operating effectiveness on its own.

3. Sample Size Selection & Sampling Methodologies

When testing manual controls for operating effectiveness, evaluating 100% of transactions is typically impossible. Assessors utilize Attribute Sampling to test for the presence or absence of a control attribute across a representative population. +-----------------------------------------------------------------------------+

• | MANUAL CONTROL SAMPLING GUIDELINES (ATTRIBUTE) |
• | |
• | Control Frequency Total Annual Population Standard Sample Size |
• | ------------------------ ----------------------- -------------------- |
• | Annual 1 1 |
• | Quarterly 4 2 |
• | Monthly 12 2 to 5 |
• | Weekly 52 5 to 10 |
• | Daily 250 to 365 20 to 40 |
• | Multiple Times Daily 1,000+ 25 to 60 |
• +-----------------------------------------------------------------------------+
• Automated Controls vs. Manual Controls

Automated Application Controls: Because automated software logic executes deterministically, testing a sample size of one (1) instance is sufficient to evaluate operating effectiveness, provided that IT General Controls (ITGC) —specifically Change Management, Logical Access, and Program Operations—are tested and proven effective throughout the period ( Benchmarking Strategy ).

Manual Controls: Subject to human inconsistency, fatigue, and turnover; require larger sample sizes (20 to 40 samples for daily operations). +-----------------------------------------------------------------------------+

• | AUTOMATED VS. MANUAL CONTROL TESTING |
• | |
• | Dimension Automated Control Manual Control |
• | ------------------------ ----------------------- -------------------- |
• | Execution Consistency Deterministic (100% same) Variable (Human) |
• | Sample Size Required 1 (with effective ITGC) 20 to 60 samples |
• | Primary Failure Mode Configuration/Code bug Human error / bypass |
• | Testing Focus System logic & Change log Execution artifacts |
• +-----------------------------------------------------------------------------+
• Population Completeness & Exception Handling

Before selecting a sample, the assessor must verify the completeness and accuracy of the population . If testing user access terminations, the population must include all terminated employees from the HR master database, not merely a pre-filtered list provided by IT operations.

Important

The Zero-Tolerance Exception Rule in Attribute Sampling: In attribute testing of internal controls, a single unexplained exception within the selected sample generally invalidates the statistical reliance on the control. An assessor cannot simply replace an anomalous failed sample item with another item; the root cause of the deviation must be fully investigated.

• 4. CRISC Exam Traps & Real-World Scenarios
• Exam Trap 1: Relying Exclusively on Inquiry

The Trap: An audit team interviews the database manager, who assures them that backups are encrypted and tested weekly. The team marks the control as effective.

The Reality: Management inquiry without independent artifact inspection (e.g., backup logs, restoration tickets) provides virtually zero audit assurance.

Exam Trap 2: Testing Operating Effectiveness of a Flawed Design

The Trap: An assessor performs extensive sampling (60 transactions) on a manual review process, only to discover later that the reviewer's checklist lacked verification of payment amounts.

The Reality: Always test Design Adequacy first . If the control design is inadequate, testing operating effectiveness is a waste of resources because the control cannot achieve its objective regardless of how consistently it is performed.

Exam Trap 3: Applying Manual Sampling Sizing to Deterministic Automation

The Trap: An assessor selects 40 automated firewall rule evaluations across 40 different days to test if port 22 is blocked.

The Reality: If the firewall configuration script and ITGC change controls are verified, a single test of the automated rule logic provides complete assurance. Testing 40 automated instances provides no additional assurance.