Traditional appsec assumes code you control processing data you validate. An LLM breaks both assumptions: its “logic” is statistical and opaque, and its instructions and its data arrive in the same channel — plain language. That collapses a boundary security has always relied on. The result is a genuinely new attack surface, catalogued by OWASP’s Top 10 for LLM Applications, sitting on top of — not replacing — every classic risk in the systems around the model.
1 · The New Attack Surface (OWASP LLM Top 10)
OWASP maintains a Top 10 specifically for LLM applications. Read it as the map of where AI systems actually break — the model, its data, its tools, and the trust we place in its output.
| Risk | What it is |
|---|---|
| LLM01 Prompt Injection | Crafted input overrides the system’s instructions — directly, or indirectly via content the model reads (a web page, a document, an email) |
| LLM02 Sensitive Info Disclosure | The model reveals secrets, PII or proprietary data from its context, training, or connected systems |
| LLM03 Supply Chain | Compromised third-party models, datasets, plugins or dependencies |
| LLM04 Data & Model Poisoning | Tampered training or fine-tuning data plants backdoors or bias |
| LLM05 Improper Output Handling | Downstream code trusts model output — enabling XSS, SSRF, SQL injection or command execution |
| LLM06 Excessive Agency | An agent has more permissions, tools or autonomy than it needs, so a bad decision does real damage |
| LLM07 System Prompt Leakage | The hidden system prompt (and any secrets in it) is extracted |
| LLM08 Vector & Embedding Weaknesses | Attacks on the RAG layer — poisoned or leaky vector stores and retrieval |
| LLM09 Misinformation | Hallucinated or wrong output relied on as fact (overreliance) |
| LLM10 Unbounded Consumption | Resource/cost exhaustion — denial-of-wallet, model extraction through mass querying |
⚠ Prompt injection has no complete fix. Because instructions and data share one channel, you can reduce it but not eliminate it — which is why the real control is limiting what the model is allowed to do (LLM06), not just what it’s told.
2 · The Vocabulary You Actually Need
A handful of terms carry most of the security conversation around AI. Knowing these is the difference between governing AI and nodding along.
| Term | Meaning |
|---|---|
| Prompt injection | Malicious input designed to override a model’s instructions |
| Jailbreaking | Bypassing a model’s safety guardrails to elicit restricted behaviour |
| Hallucination | Plausible but fabricated or wrong output — a reliability and security risk |
| Alignment | Making a model behave according to human values and intent |
| RLHF | Reinforcement Learning from Human Feedback — the tuning that shapes behaviour |
| Guardrails | Input/output filters and policy checks wrapped around the model |
| Model card | Documentation of a model’s capabilities, limits and intended use |
3 · Defenses That Hold
You can’t patch a probability distribution, so AI security is mostly about the system around the model — classic security discipline applied to a new component.
- Treat every model output as untrusted input — validate and encode it before it touches code, a browser, or a database (kills LLM05).
- Least privilege for agents — scope tools, permissions and autonomy tightly; require human-in-the-loop for high-impact actions (contains LLM06).
- Guardrails, but assume bypass — input/output filtering helps; design so a successful injection still can’t do much.
- Keep secrets out of prompts — don’t place data in the context or system prompt you couldn’t bear to leak (LLM02, LLM07).
- Secure the pipeline — verify model & dataset provenance, pin dependencies, secure the RAG/vector store (LLM03, LLM04, LLM08).
- Rate-limit & budget — quotas and cost caps blunt denial-of-wallet and model extraction (LLM10).
- Red-team & evaluate continuously — adversarial testing and model cards make behaviour measurable.
4 · The Governance Now Attached to AI
AI security is not only technical — three frameworks now define the compliance side, and they map onto the GRC muscle organisations already have.
| Framework | What it is | Use it for |
|---|---|---|
| NIST AI RMF | Voluntary risk-management framework (Govern, Map, Measure, Manage), with a Generative-AI profile | A structure to identify & manage AI risk — the “CSF for AI” |
| ISO/IEC 42001 | Certifiable AI management system (AIMS) standard | The “ISO 27001 for AI” — prove a managed AI programme |
| EU AI Act | Risk-tiered regulation (prohibited → high-risk → limited → minimal), phasing in through 2025–2027 | Legal obligations if you build or deploy AI touching the EU |
▸ The GRC tie-in: if you already run ISO 27001, ISO 42001 bolts on the same way ISO 27701 does for privacy — and the control overlap means much of the work is shared.
Bottom line
- The channel collapsed — instructions and data now arrive together, which is why prompt injection is structural, not a bug.
- Limit agency, not just input — the durable control is what the model may do, since what it’s told can’t be fully secured.
- Output is untrusted — treat model responses like user input everywhere downstream.
- AI sits on top of appsec — it adds risks; it doesn’t remove supply-chain, access or data ones.
- Governance arrived fast — NIST AI RMF, ISO 42001 and the EU AI Act now make AI a board and compliance topic.
Sources & method. Risk categories follow the OWASP Top 10 for LLM Applications (2025); the terminology reflects standard AI/ML usage. Governance references are the NIST AI Risk Management Framework (AI RMF 1.0 + Generative-AI profile), ISO/IEC 42001:2023, and the EU AI Act (phased application 2025–2027). General information, not legal advice. Original analysis; no third-party graphics or text reproduced.