AI threat map — the attack classes, and where the program meets them¶
The field's attack surface on one page, grounded in the four taxonomies worth knowing: OWASP Top 10 for LLM Apps (the
LLMxxIDs below), OWASP Agentic Security Initiative (agent-specific), MITRE ATLAS (the adversarial-technique matrix, ATT&CK for AI), and NIST AI 100-2 (adversarial ML). This page is a snapshot — taxonomies version and new classes land monthly; re-check before you lean on it.It does triple duty: the map of what you're defending against, the corpus manifest for the Week-2 attack-knowledge RAG, and the coverage checklist for the Week-7 scenario set and the Week-8 capstone. When you write the capstone launch post, name your detections by their OWASP/ATLAS IDs — that's the language buyers and interviewers already speak.
How the program meets each class¶
- Built / defended — you ship a control against it.
- Red-teamed — you attack it (against your own ships or a practice target) before you defend.
- Awareness — named so it's not a blind spot; not built this cycle.
- Day 91 — belongs to the continuation (endpoint/detection, deeper red-team).
| Attack class | What it is | Canonical | In the program |
|---|---|---|---|
| Prompt injection (direct) | User input overrides the system's intent | LLM01 · ATLAS | Built — W1 detector, every "Secure it" |
| Indirect / cross-domain injection | Instructions ride in on content the model reads (a page, doc, log, tool output) | LLM01 · ATLAS | Built + red-teamed — W3 poisoned trace, W4 browser page, W6 image |
| Jailbreak / refusal bypass | Persona/encoding/multi-turn tricks that defeat safety guardrails | ATLAS · NIST (evasion) | Red-teamed — W3 warmup (Gandalf) |
| System-prompt leakage | Extracting the hidden instructions/config | LLM07 | Red-teamed — W3 warmup; awareness as a detector class |
| Sensitive-information disclosure | PII, secrets, internal names leaking through outputs | LLM02 | Built — W1 detector, W6 screen-DLP |
| The lethal trifecta / exfiltration | Untrusted input + private data + outbound channel = turnable agent | LLM02+LLM06 · ATLAS (exfiltration) | Built + red-teamed — W3, capstone core |
| Excessive agency / tool misuse | Over-broad tool access, destructive capability, no gate | LLM06 · OWASP Agentic (tool misuse) | Built — W3 gating, capstone policy |
| Confused deputy / privilege abuse | An agent's legitimate authority tricked into misuse | OWASP Agentic (identity/privilege) · ATLAS | Built — W0 MCP threat model, W11–12 broker grants |
| Improper output handling (SSRF/XSS/RCE downstream) | Model output hits a sink unsanitized | LLM05 | Built — W1 sink rule; awareness for web sinks |
| Data / model poisoning | Corrupting training/fine-tune/RAG data to plant behavior | LLM04 · ATLAS · NIST | Built — W2 attacker-authored corpus; awareness at train time |
| RAG / retrieval + embedding poisoning | Malicious docs steer what gets retrieved | LLM08 · ATLAS (RAG poisoning) | Built — W2 |
| Model supply chain (pickle/format RCE, malicious HF models) | Loading a model file executes attacker code | LLM03 · ATLAS (supply-chain compromise) | Built — W5 "Secure it": scan before you load |
| Tool / MCP supply chain (tool poisoning, rug-pull, shadowing) | A malicious or mutated MCP server/tool hijacks the agent | OWASP Agentic (supply chain) · ATLAS | Built — W0 MCP threat model (mcp-scan) |
| Coding-agent / IDE-agent hijack | A malicious repo's hidden instructions drive your coding agent → RCE | ATLAS · (CVE-2025-53773 class) | Awareness — W0 (the agent you live in is a target) |
| Memory / context poisoning | Contaminating persistent memory to steer future decisions | OWASP Agentic (memory poisoning) · ATLAS | Awareness — W3 memory stretch |
| Insecure inter-agent comms | Attacks on the messages between agents | OWASP Agentic (inter-agent comms) | Built — W11–12 channel policy |
| Cascading failures / rogue agents | One compromised agent/component takes the system with it | OWASP Agentic (cascading, rogue) | Built — W11–12 containment/taint |
| Unbounded consumption (denial-of-wallet) | Forced token/compute burn to exhaust a budget | LLM10 | Awareness — W12 cost ceiling; W3 runaway postmortem |
| Privacy attacks (extraction, membership/attribute inference, inversion) | Recovering training data or facts about it from the model | NIST AI 100-2 (privacy) | Awareness / Day 91 |
| Evasion / adversarial examples | Crafted inputs that degrade or bypass a model | NIST · ATLAS (evasion) | Awareness — the detector's own evasion cases (W1/W5) |
| Misinformation / hallucination | Fabricated facts/sources exploited downstream | LLM09 | Built — W2 grounding, abstain paths |
| Human–agent trust exploitation | Deceiving people about what an agent did / can do | OWASP Agentic (trust) | Awareness |
| Package hallucination / slopsquatting | Models invent library names; attackers register them | ATLAS-adjacent (emerging) | Awareness — W0 review discipline |
Where to go deeper¶
Reading and hands-on targets for each class live in the JIT map's AI-security
section — MITRE ATLAS and NIST AI 100-2 for the
taxonomy, Johann Rehberger's Embrace The Red for the coding-agent/MCP exploit stream,
PortSwigger's LLM labs and Lakera Gandalf for hands-on offense, mcp-scan for tool poisoning,
and the agent-security landscape for who's building
defenses. The incident database and
huntr are where fresh real-world cases show up.