LLM Pentest: Prompt Injection and the AI-App Bugs Scanners Miss

Your AI feature is your least-tested attack surface

Every product has a chatbot now. A support assistant, a “summarize this,” a copilot wired into your data. They shipped fast, and almost none of them got tested like an attacker would test them.

Here’s the uncomfortable number. Across thousands of engagements, AI and LLM assets show the highest proportion of serious vulnerabilities of any asset type — and the lowest remediation rate (Cobalt State of Pentesting). Most teams found the bugs and shipped anyway, or never looked.

Why so bad? Because an AI feature isn’t a normal app. Its logic lives in a prompt, its input is natural language, and its output can quietly become an instruction. None of that shows up in a signature scan.

That’s the gap an LLM pentest fills. Not a model benchmark, not a bias audit — a real security test of the AI feature as an attacker meets it: through the text box.

And unlike most surfaces, this one talks back. It can be reasoned with, tricked, and flattered — which is exactly why testing it takes something that can hold a conversation, not a scanner running a checklist.

The truth is, the same jump that made these features useful made them dangerous. You handed a language model access to your data and your tools. An attacker only has to talk it into misusing them.

Why attackers go for the AI feature first

There’s a reason the chatbot is a target and not an afterthought.

Through 2025 and into 2026, security teams documented attackers wiring AI agents into real intrusions — and using AI apps as the way in. A model with access to your data is a shortcut. Instead of finding a bug in your code, an attacker just has to find the right sentence.

It’s also the least-defended door. Your web app has a decade of hardening behind it. Your three-month-old AI assistant has a system prompt and some hope. Attackers go where effort is lowest and payoff is highest, and right now that’s the LLM pentest gap.

Then add the economics. AI features ship under pressure, fast, often with no security review at all. The result is a surface that’s high-value, widely deployed, and barely tested at the same time. That combination never stays unexploited for long.

What an LLM pentest actually looks for

A good LLM pentest maps to the OWASP Top 10 for LLM Applications — the closest thing the field has to a shared checklist. It’s less about the model’s IQ and more about what the surrounding app lets the model do.

The questions an LLM pentest asks are simple to state and nasty to answer. Can I make it ignore its instructions? Can I read its system prompt? Can I get it to leak another user’s data? Can I turn its output into code that runs? Can I abuse the tools it’s allowed to call?

We also lean on the MITRE ATLAS knowledge base for adversarial ML tactics, and the NIST AI Risk Management Framework for the governance side. But the LLM pentest itself is hands-on-keyboard: real payloads against your real app.

Notice what’s missing from that list: the model’s benchmark scores. An LLM pentest doesn’t care whether your model tops a leaderboard. It cares whether the app around it can be talked into leaking, running, or exposing something. A brilliant model wired into a careless app is still a breach.

6 LLM vulnerabilities most tests miss

Each of these is common, each maps to an OWASP LLM category, and each is invisible to a scanner that doesn’t speak the model’s language. Sector examples below are patterns we see in the field, not a single named client.

1. Direct prompt injection (LLM01)

The simplest and the most common. A user types “ignore your previous instructions and…” and the model obeys. It sounds trivial until that model can issue refunds, read tickets, or change account settings.

The bug isn’t that the model is gullible. The bug is that the app wired a gullible thing to real power and never tested the seam.

2. Indirect prompt injection (LLM01)

This is the one that keeps us up at night. The attacker never talks to your model. They plant instructions in a document, a web page, or an email your AI will later read — and the model treats that content as commands.

Summarize-this-PDF features, RAG chatbots, email assistants: all of them ingest untrusted text and can be hijacked by it. A single poisoned page in your knowledge base can turn helpful into hostile.

The scary part is the trust boundary. Your app trusts its own knowledge base; the model trusts whatever it reads. An attacker who can write to either — a public doc, a shared ticket, a scraped page — reaches your users through your own assistant.

3. System-prompt and data extraction (LLM07, LLM02)

Your system prompt is your app’s secret sauce — and often its access rules. A determined LLM pentest can frequently coax the model into reciting it, revealing hidden instructions, internal tool names, and the guardrails an attacker now knows how to skip.

Worse is data leakage. Ask the right way and some assistants will surface another customer’s records, cached context, or PII the model saw earlier in a shared session.

The fix is rarely “tell the model to keep secrets.” Models don’t keep secrets reliably — that’s the whole point. The real fix is architectural: don’t put in the context what the user isn’t allowed to see. An LLM pentest is how you find every place that rule quietly got broken.

4. Insecure output handling (LLM05)

Teams treat model output as text. Attackers treat it as a payload. If your app renders LLM output in a browser, or feeds it into a query or a shell, a prompt can make the model emit working XSS, SQL, or a command.

5. Excessive agency and tool abuse (LLM06)

Agentic features are where this gets serious. Give a model tools — run code, query a database, call an API — and prompt injection stops being a chat problem and becomes remote code execution. The model is a confused deputy with real permissions.

6. Sensitive data and PII leakage (LLM02)

Retrieval-augmented apps pull from internal sources to answer questions. Scope that retrieval loosely and the assistant becomes a search engine for data the user should never reach — HR files, other tenants’ records, secrets pasted into a wiki.

What makes this one brutal is that it looks like the feature working. The assistant answered the question — it just answered it for the wrong person. Only an LLM pentest that asks as an attacker, from a low-privilege account, surfaces the gap between “helpful” and “leaking.”

How one poisoned page becomes a breach

Indirect prompt injection deserves its own walk-through, because it’s the class that quietly turns a helpful feature into an open door.

The chain is short and ugly:

1. An attacker plants instructions in content your AI will later read — a web page, a shared doc, a support ticket, a calendar invite.
2. A user innocently asks your assistant to summarize or act on that content.
3. The model reads the hidden instructions and follows them: exfiltrate data, call a tool, or rewrite its answer to phish the user.
4. Nobody typed anything malicious into your app. The payload rode in on trusted-looking content.

That’s why indirect injection worries practitioners more than the direct kind. The attacker never authenticates, never touches your UI, and the model does the damage on their behalf — a confused deputy holding your permissions.

An LLM pentest built for this doesn’t just try rude prompts in the chat box. It seeds hostile content in the places your AI actually ingests — the documents, the search index, the inbox — and watches what the model does next. That’s the only way to catch a bug that never comes through the front door.

Why a scanner can’t run an LLM pentest

A traditional scanner looks for known-bad signatures in code and traffic. An LLM vulnerability isn’t a signature. It’s a conversation that goes somewhere it shouldn’t.

To find these, you have to actually talk to the model — thousands of times, adapting each probe to how it answered the last one. That’s reasoning and iteration, not pattern-matching. It’s exactly the kind of testing that doesn’t fit a signature engine or a one-week manual window.

Now the honest part, because AI security is full of overclaims. An LLM pentest reduces and proves risk — it shows you concrete, reproducible ways your app fails. It does not hand you a certificate that says the model can never be jailbroken. Because models are probabilistic, no one can honestly promise that. Anyone who does is selling you something.

What a good LLM pentest gives you instead is real: a ranked list of exploitable failures, each with the exact prompts that trigger it, and the fix that closes it. That’s the difference between “we added a safety filter” and “here’s the message that walks straight through it.”

That honesty is the whole point. A vendor who promises an un-hackable AI is either naive or lying, and either way you shouldn’t hand them the test. What you want is someone who shows you exactly where your AI breaks, hands you the receipts, and helps you close the gap you can actually close.

How SelfHack AI runs an LLM pentest

This is the part that fits us a little too well: it takes an AI to properly pentest an AI. SelfHack AI runs a swarm of agents that probe your AI feature the way a patient, creative attacker would — at machine scale.

• Adaptive probing — agents generate and mutate prompt-injection and jailbreak attempts, learning from each response instead of replaying a static list.
• Full-chain testing — an LLM pentest here doesn’t stop at the chat box; it follows the model into the tools, APIs and data it can reach, and proves the downstream impact.
• Exploit-validated findings — every issue ships with the exact conversation that triggers it, so your team can reproduce and fix it, not argue about it.
• Mapped to standards — results line up with the OWASP LLM Top 10 and MITRE ATLAS, ready for an audit.

It’s the same platform behind our AI pentest benchmark and our work on externally-exposed systems — pointed at the newest, softest part of your stack.

And because it’s automated and reproducible, an LLM pentest fits where security has to live now — in the pipeline, re-run on every prompt change, not filed once and forgotten while the model quietly drifts.

How to scope an LLM pentest

Scoping an LLM pentest is less about paperwork and more about honesty. Here’s the path we walk teams through.

1. List every place a model meets untrusted input. Chatbots, summarizers, RAG search, email and doc assistants, agentic workflows. Each one is an LLM pentest target, and teams always forget two or three.
2. Map what the model can touch. Which tools, APIs, databases and user data are within its reach? That reach is your real blast radius if a prompt injection lands.
3. Demand reproducible proof. For every finding, insist on the exact prompt sequence. A vague “the model can be jailbroken” is useless; “this message extracts the system prompt” is actionable.
4. Re-test after every prompt change. A tweak to the system prompt can open a hole a previous fix closed. LLM behavior drifts, so an LLM pentest should repeat, not sit on a shelf.

Do it this way and the first run is usually a wake-up call — not because the model is dumb, but because nobody had tried to weaponize it before.

One layer down from the model sits the hardware your AI workload actually runs on. Our confidential computing pentest tested exactly that — GPU confidential compute under a hostile host — and found the CPU attestation report never measured the GPU’s security state at all.

FAQ

What is an LLM pentest?

An LLM pentest is a security test of an AI application — a chatbot, copilot, RAG search or agent — that probes it the way an attacker would, through natural language. It looks for prompt injection, data leakage, insecure output handling and tool abuse, and proves each finding with the exact prompts that trigger it.

How is it different from red-teaming a model?

Model red-teaming asks “can this model produce harmful content?” An LLM pentest asks “can an attacker make this application do something harmful?” The first tests the model in isolation; the second tests your whole app — the prompt, the tools, the data, the output handling — which is where real breaches happen.

Can you really automate LLM pentesting?

Yes, and it’s one of the few places automation clearly wins, because finding these bugs means running thousands of adaptive conversations. What automation can’t do is promise a model is permanently safe — LLMs are probabilistic, so an LLM pentest proves specific failures and reduces risk rather than certifying perfection.

Which OWASP LLM risks do you cover?

The full OWASP LLM Top 10 — prompt injection, sensitive information disclosure, improper output handling, excessive agency, system-prompt leakage and the rest. We map every finding to its category so it drops cleanly into an audit.

We already have guardrails. Do we still need an LLM pentest?

Guardrails are exactly what an LLM pentest tests. A safety filter you haven’t attacked is a hope, not a control. The point is to find the message that walks through your guardrail before an attacker does.

How often should we run an LLM pentest?

After every meaningful prompt or model change, and on a regular cadence otherwise. Because LLM behavior drifts with model updates and prompt tweaks, a one-time test goes stale fast.

Does an LLM pentest cover the model or just the app?

Both, where it matters. We test the application around the model — prompts, tools, output handling, data access — because that’s where exploitable bugs live. We also probe the model’s behavior under adversarial input, but the focus is the app an attacker actually reaches.

What do we get at the end of an LLM pentest?

A ranked list of exploitable findings, each mapped to an OWASP LLM category, each with the exact prompt sequence to reproduce it and a concrete fix. Not a vague risk score — a set of messages your team can paste in, watch fail, then watch pass once the fix lands.

The verdict

Your AI feature is the fastest-shipped, least-tested thing in your product. Attackers know it, the data proves it, and the text box is wide open.

An LLM pentest won’t promise you a model that can never be fooled — be suspicious of anyone who does. What it gives you is honest and useful: the exact ways your AI app can be turned against you, ranked and reproducible, with the fixes that close them.

The honest bottom line: shipping the AI feature was the easy part. Testing it is what decides whether it becomes your best feature or your next incident. Better you find the message that breaks it than the person who’s already out there looking for it.

Want to see what your chatbot does under real pressure? Start a scan — order an LLM pentest from SelfHack AI or contact our team. Point us at your AI feature; we’ll show you what an attacker can talk it into.

Sources

• OWASP Top 10 for LLM Applications — owasp.org
• MITRE ATLAS — adversarial tactics and techniques against AI systems — atlas.mitre.org
• NIST AI Risk Management Framework — nist.gov
• Cobalt State of Pentesting (LLM/GenAI asset findings; vendor-reported) — cobalt.io

Scope note: LLMs are probabilistic; an LLM pentest demonstrates and reduces exploitable risk but cannot certify a model as permanently un-jailbreakable. Statistics are attributed to their sources; “from the field” boxes are representative sector patterns, not specific named engagements.