LLM Penetration Testing: How to Find and Fix Security Vulnerabilities in AI Applications

LLM penetration testing is a security test in which experts attack an AI application built on a large language model. Chatbots and copilots now touch private files and business tools, so one malicious instruction can cause real harm. A good test ends with a ranked list of findings and fixes.

Traditional testing looks at code, servers and networks. LLM security testing asks a different question: what does the model do when it reads hostile text? A model follows plain language, so an attacker may need only the right sentence.

A good AI penetration test combines both. The web application and APIs around the model still need standard testing.

• Chatbots handle private customer and staff data.
• Connected tools can send email, edit records or run code.
• Customers and auditors increasingly ask how AI features have been tested.
• A public failure can damage trust for years.

If your application is in scope for SOC 2 or ISO 27001, AI features count too. See how testing fits into an audit in our guide to SOC 2 penetration testing.

The OWASP Top 10 for LLM Applications is the usual reference. Most reports include some of these:

• Prompt injection. A hidden command in a message, file or web page makes the model ignore its rules. Indirect attacks are the most dangerous because users never see the hidden text.
• Sensitive information disclosure. The model reveals its system prompt, customer details or internal notes.
• Improper output handling. The application runs or displays whatever the model writes, so attacker text can trigger harmful actions.
• Excessive agency. The assistant can do too much. If it can delete records, one crafted prompt may be enough.
• Broken access control in RAG. The retrieval layer fetches documents without checking who is asking, so users see files they shouldn't.
• Unbounded consumption. Floods of large requests run up cost or take the service down.
• Supply chain weaknesses. An unvetted plugin, model or dataset brings hidden risk.

A typical test follows these steps:

1. Get written authorisation and agree the scope.
2. Map the application: the model, its data sources and every tool it can call.
3. Mark the trust boundaries: who can see which data.
4. Attempt direct and indirect prompt injection.
5. Test for data leakage and missing access checks, especially in retrieval.
6. Probe tool calls, API connections and how the application handles model output.
7. Check whether logs and alerts would catch a real attack.
8. Report each finding with its severity, evidence and a fix.
9. Retest after fixes and record any remaining risk.

Never test without written permission. Log every prompt and response, because developers need the evidence to reproduce each issue.

The core question is whether the model obeys attacker text over trusted instructions.

• Direct tests: override rules, extract the system prompt, or change the model's role through role-play and long, confusing inputs.
• Indirect tests: hide instructions inside PDFs, web pages, emails or tickets that the model will read.
• Impact tests: once an injection works, check what follows. Can the model call a tool or send data out? That is what decides the real severity.

Combine manual testing with automated tools, and repeat it, because models change and new techniques appear constantly.

Strong AI application security never relies on the model alone.

• Enforce authorisation on the server for every sensitive action, never in the prompt.
• Give each tool and agent only the access it needs.
• Treat user input and retrieved documents as untrusted.
• Filter outputs and restrict data so private details stay private.
• Require human approval for high-impact actions.
• Add rate limits and alerts for unusual behaviour.
• Retest each fix with realistic attacks.

Give each finding an owner and a deadline, and retest after any change to the model, prompts or plugins. For the wider picture, see AI agent security and non-human identity security.