Study guide

6.5 AI Attack Types, Evidence, and Compensating Controls

CompTIA SecAI+ CY0-001

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: AI and Machine Learning Foundations

1.1 Core AI/ML Paradigms: Supervised, Unsupervised, and Reinforcement Learning 1.2 Deep Learning Architectures and Neural Network Fundamentals 1.3 Foundation Models, Large Language Models (LLMs), and Tokenization 1.4 Prompting, Data Security, and the AI Lifecycle

2 Chapter 2: Modern AI System Architectures in Security

2.1 Embeddings, Vector Spaces, and Vector Databases 2.2 Retrieval-Augmented Generation (RAG) Architectures and Data Flow 2.3 Autonomous AI Agents, Tool Execution, and Orchestration

3 Chapter 3: Adversarial Machine Learning & Evasion Attacks

3.1 Adversarial Evasion Attacks and Perturbations 3.2 Defenses Against Evasion and Adversarial Robustness 3.3 Data Poisoning and Backdoor Attacks

4 Chapter 4: Model Privacy, Confidentiality & Inversion Attacks

4.1 Model Inversion and Training Data Extraction 4.2 Membership Inference Attacks and Differential Privacy 4.3 Model Theft, Extraction, and IP Protection

5 Chapter 5: LLM Vulnerabilities & CY0-001 OWASP 2023-24 Taxonomy (Part 1)

5.1 CY0-001 / OWASP 2023-24 LLM01: Prompt Injection 5.2 CY0-001 / OWASP 2023-24 LLM02 & LLM07: Output and Plug-ins 5.3 CY0-001 / OWASP 2023-24 LLM03 & LLM05: Poisoning and Supply Chain 5.4 CY0-001 / OWASP 2023-24 LLM04 & LLM06: DoS and Disclosure

6 Chapter 6: CY0-001 OWASP 2023-24 Taxonomy (Part 2) & Secure MLOps

6.1 CY0-001 / OWASP 2023-24 LLM08-LLM10: Agency, Overreliance, and Theft 6.2 Secure MLOps Pipeline Architecture and Hardening 6.3 Runtime AI Monitoring, Guardrails, and Drift Detection 6.4 AI Access, Data, Input, Output, and Audit Controls 6.5 AI Attack Types, Evidence, and Compensating Controls

7 Chapter 7: AI in Threat Detection and SOC Automation

7.1 AI for SOC Alert Triage and Incident Prioritization 7.2 Anomaly Detection and Behavioral Analytics (UEBA) 7.3 AI-Driven Phishing Detection and Email Security

8 Chapter 8: AI in Threat Intelligence, Hunting & Vulnerability Management

8.1 Cyber Threat Intelligence (CTI) Enrichment and Summarization 8.2 AI-Assisted Threat Hunting and Query Generation 8.3 AI in Secure Code Review and Vulnerability Detection 8.4 AI Security Tools, Attack Discovery, and Workflow Automation

9 Chapter 9: Operational Risks and Limitations of AI-Assisted Security

9.1 Hallucinations, Confabulation, and False Positives in SOC AI 9.2 Model Degradation: Data Drift, Concept Drift, and Monitoring Metrics 9.3 Dual-Use Risks: Adversary Exploitation of AI

10 Chapter 10: AI Risk Frameworks, Regulations & Standards

10.1 NIST AI Risk Management Framework (AI RMF 1.0) 10.2 ISO/IEC 42001 and International AI Standards 10.3 The EU AI Act and Global Legal Requirements 10.4 MITRE ATLAS: Adversarial Threat Landscape for AI Systems

11 Chapter 11: AI Red Teaming, Ethics & Enterprise Governance

11.1 AI Red Teaming and Adversarial Probing Methodologies 11.2 Transparency, Model Documentation, and Explainability 11.3 Enterprise AI Governance, Acceptable Use Policies & Data Privacy 11.4 Governance Roles, Risk Taxonomies, and Compliance Operations

6.5 AI Attack Types, Evidence, and Compensating Controls

Key Takeaways Explain This Topic AI

Model skewing and output-integrity attacks manipulate behavior or downstream decisions; transfer-learning attacks abuse inherited artifacts, data, or assumptions.

Integrations expand the blast radius because model output can cross into browsers, APIs, databases, code runners, and enterprise workflows.

Investigation must preserve prompts, retrieved context, artifacts, versions, tool traces, identities, and outputs without executing hostile evidence.

Compensating controls reduce residual risk when the preferred control is unavailable, but their scope, owner, evidence, and expiration must be documented.

Security teams should map a verified failure to the affected layer and select preventive, detective, responsive, and recovery controls.

Last updated: September 2026

6.5 AI Attack Types, Evidence, and Compensating Controls

AI incidents can originate in data, inherited models, prompts, runtime tools, or downstream consumers. Correct response begins by naming what changed and preserving enough evidence to reproduce it.

Distinguishing attack patterns

A training-data poisoning attack alters examples or labels so training produces attacker-favorable behavior. A transfer-learning attack exploits the reuse of a pretrained model, adapter, embedding model, or dataset. A compromised base artifact may contain a backdoor that survives fine-tuning, and a malicious adapter may redirect a previously safe model. Verify provenance, signatures, hashes, licenses, dependency versions, and behavioral test results before adoption.

Model skewing is deliberate manipulation that shifts a model's behavior or decision boundary. It may result from poisoned updates, fraudulent feedback, unrepresentative samples, or abuse of online learning. Ordinary drift is not automatically an attack; investigators need evidence of intent and mechanism.

An output-integrity attack changes, forges, suppresses, or reorders a model's result or the evidence attached to it. Examples include tampering with an inference response, modifying a risk score in transit, hiding a citation, or causing unsafe generated code to reach a consumer. Cryptographic transport protection, signed artifacts, schema validation, destination encoding, independent verification, and reconciliation with source evidence help detect it.

Evasion crafts inference input to avoid detection. Prompt injection attempts to override instructions. Jailbreaking attempts to bypass behavioral safeguards. Model extraction approximates or steals a model through artifacts or queries. Membership inference estimates whether a record was in training; inversion attempts to reconstruct features or data. Resource-exhaustion attacks target compute, tokens, memory, or agent loops. Similar symptoms can have different causes, so do not choose a control from the label alone.

Integration risk

An isolated answer has limited reach; an integrated answer may become an email, browser action, source-code change, database query, payment, or security rule. Inventory IDE, browser, command-line, ticketing, SOAR, repository, API, and personal-assistant integrations. For each, document data direction, identity, permissions, trust boundary, allowed operations, validation, approval, logging, and rollback.

A browser assistant that summarizes a malicious page faces indirect prompt injection. An IDE plug-in may expose proprietary code or insert a vulnerable dependency. A CLI agent can transform generated text into shell arguments. A ticket assistant may leak one customer's data into another ticket. Server-side permission checks and destination-specific validation are required even when the model claims an action is safe.

Using complementary risk sources

CY0-001 names several resources with different purposes. The OWASP Machine Learning Security Top 10 organizes recurring engineering risks in machine-learning systems and supports application threat modeling; always record its edition. The MIT AI Risk Repository is a research-based catalog and taxonomy for comparing AI risks across causal factors and domains; it is not an adversary technique matrix. The CVE AI Working Group works within the CVE ecosystem on consistent treatment and coordination of AI-related vulnerabilities; a CVE identifies a disclosed product vulnerability, not every harmful model behavior. Use these with the OWASP LLM list and MITRE ATLAS according to the question being asked: weakness category, broad risk, product vulnerability, or adversary behavior.

Evidence workflow

First contain the incident without destroying volatile evidence: revoke or narrow agent tokens, pause automated actions, preserve relevant services, and record the time. Capture request and session identifiers, user and workload identity, complete input chain, system and template versions, retrieved document IDs, model and adapter hashes, dataset and index versions, tool-call arguments and results, output, policy decision, network destinations, cost, and human approvals. Preserve logs with integrity controls and handle malicious files in an isolated environment.

Next build a timeline and compare the failing request with a known-good control. Reproduce only in a sandbox. Change one variable at a time: model version, prompt template, retrieval corpus, adapter, tool permission, or consumer. This separates a poisoned artifact from prompt injection, an integration flaw, benign drift, or response tampering. Record uncertainty; a suspicious output alone does not prove which attack occurred.

Selecting controls and alternatives

Choose controls at the affected layer:

For compromised transfer artifacts: trusted sources, signatures, hashes, software bills of materials, scanning, isolation, and independent behavior tests.

For model skewing: authenticated data and feedback, separation of duties, robust statistics, version comparison, drift and performance monitoring, and rollback.

For output integrity: TLS, message authentication or signing where appropriate, schema and range checks, fact verification, destination encoding, and transaction reconciliation.

For integrations: least-privilege identities, allowlisted tools, sandboxes, previews, approvals, egress rules, rate limits, and kill switches.

For investigations: synchronized time, immutable logs, protected snapshots, chain of custody, and retention aligned with law and policy.

A compensating control is an alternative that provides comparable risk reduction when the preferred control cannot be implemented. Suppose a legacy model endpoint cannot provide signed responses. A temporary combination of mutually authenticated transport, a tightly restricted network path, independent source reconciliation, read-only use, and human approval may compensate. Document the gap, threat addressed, control owner, tests, monitoring, residual risk, approving authority, review date, and expiration. A compensating control is not a permanent waiver and must not be described as equivalent without evidence.

Worked scenario

After an adapter update, a fraud model approves transactions containing a rare token. The responder freezes the rollout, preserves the adapter hash, training job, feedback records, inference traces, and approval history, then rolls back to the signed prior version. Sandbox comparison shows that the trigger works only with the new adapter. The team classifies this as suspected transfer-artifact compromise and model skewing rather than routine drift. Until a signed adapter pipeline is available, production accepts only internally built adapters, requires two-person approval and behavioral trigger tests, blocks online learning, and expires the exception in 30 days.

Loading diagram... AI Incident Evidence and Control Selection