A prompt injection is what happens when a customer-facing assistant treats untrusted text as an order. The text may come from the chat, a fetched page, a PDF or a ticket. If the assistant can look up an order, send an email or change a record, that sentence can trigger the action.
Contents
The model follows instructions. It cannot reliably separate your rule from a line inside a document. A chat filter misses the common case, because the hostile text sits in content you asked the assistant to read. OWASP documents this as LLM01 in the GenAI project risk list, with controls in the LLM prevention cheat sheet.
At CodeWiz we build the assistant and the APIs it calls. Defense means least privilege, confirmation when money or an account changes, and an allow-list of tools. This is for teams whose bot can see orders or send mail. The conversation is our AI chatbot and agent development. A wider check is a security audit.
Instructions are not the same channel as data
Instructions come from you: the system prompt, the tool list, the rules on refunds and tone. Data comes from the customer, a page, a PDF, a ticket or a help article. The work against prompt injection starts when you refuse to mix those channels. Data can be quoted. It cannot add a tool, change a role or widen what the assistant may do.
Put untrusted text in a labeled block and tell the model it is a document. That label is a hint, not a control. A long page, or one that imitates a system message, still confuses models. Delimiters reduce accidents. The control is code that never asks the model for permission.
- System channel. Policy, tools and tone, reviewed in Git, not editable by a customer or a fetched page.
- User channel. The current message, bound to a session, and still untrusted.
- Document channel. Pages, PDFs, tickets and retrieval chunks, marked as quotes.
- Tool channel. Structured API results the model may explain but must not extend.
If a document says the assistant is now an administrator, the session role stays what login set. That single rule removes a large class of prompt injection that only works when the prompt is also the access-control system.
Indirect prompt injection from a page, an email or a ticket
Direct prompt injection is the customer typing a hostile instruction into the chat. Indirect prompt injection is the same idea hidden in something the assistant was told to read: a product page, a PDF, an email thread or a ticket comment.
A support assistant that summarizes a ticket and sends status mail is the ordinary case. A line in the thread says to forward the full order to another mailbox and to call that a routine update. People miss one line. If the mail tool accepts any recipient and any body, the message goes out.
A shop assistant that opens a product page has the same exposure. Supplier feeds, reviews, a compromised editor and an uploaded PDF are writable by someone else. Fetched strings stay data. They do not choose the recipient, the order id or the tool.
Indirect prompt injection is why scanning only the chat box is not a design. The dangerous text arrives through the web reader, the mailbox or the ticket system. Those readers return text, not authority.
Allow-lists for tools that touch orders and email
Least privilege means a short allow-list, not a general API token. Each tool does one job and drops fields the model was not supposed to set. Order lookup is not a refund, and neither is a raw query the model describes.
The model may pick only a tool name you registered. Arguments must match a schema, and unknown names never leave your process. Customers and staff get separate assistants. The customer deployment does not register staff tools.
Risk follows the action. The model may suggest a row below. The server decides whether this session may perform it.
| Action | Risk if the model is trusted | Control outside the prompt |
|---|---|---|
| Read an order | A page or ticket names someone else’s order | Use the customer id from the session and ignore any id the model supplies |
| Send an email | Details go to an address that is not on the account | Templates only; the recipient is taken from the order, not the conversation |
| Issue a refund | Untrusted text says to refund the order in full | No refund tool on the customer assistant; staff confirm, then your payment function runs |
| Change a password | Text asks for a new address or a password the sender chose | Start the existing reset to the stored address; never set the password |
Refuse tools that send any body to any address, or that write arbitrary fields. A shipping-status template to the address on the order is enough. An open mailbox is a prompt injection amplifier, because the model can be talked into any recipient and any body.
The model never decides the refund
Separate the model from the permission check. It may draft what the customer will read. It must not decide that a refund is authorized. Authorization uses the session, the order state and your policy. The model is only a caller, and callers can be manipulated.
Teams often hide that rule in the system prompt. The prompt is advice. The next prompt injection can argue with a rule that lives only in the prompt. The date window and the ownership check belong beside the database, where a fetched page cannot edit them.
The assistant never calls the payment provider. Your function refuses unless the session is staff and a person has confirmed this order:
function issueRefund(session, orderId) { requireStaff(session); requireExplicitConfirmation(session, orderId); return payments.refund(orderId); }Do not register that function on a customer assistant. A sentence in a PDF cannot pass the staff check, because the check reads the login session, not the chat.
Human confirmation before money or account changes
Refunds, address changes, cancellations, mail to anyone but the customer, and password changes wait for a person. Confirmation is a button your application renders. It is not a sentence the model claims the user already said.
Show the order and the amount from the order record, not from the draft. Store the user id, the order id and the time of the click. The model cannot emit that click. A prompt injection that says the user already confirmed does not set that flag, because the button handler writes it.
A parcel-status read can run at once. A refund cannot. If you cannot explain the confirmation in one sentence, the tool is too vague to expose.
Assistant output that gets rendered as HTML
The reply is untrusted. Inserted as HTML, a script tag or an event-handler attribute is cross-site scripting. Escape the text, or use a markdown subset with no raw markup, and keep a Content-Security-Policy as you would for any other user content.
A prompt injection can also try to make the assistant show a tracking link that leaves your domain. Allow-list hosts, or build links from order data you trust. The model must not choose the address of a button that looks like your shop.
A widget that writes the reply with innerHTML has already trusted the model with the page.
Secrets do not belong in the system prompt
Anything in the system prompt can be repeated back. A prompt injection that asks the model to repeat its hidden instructions is enough to leak whatever you stored there. Keys, passwords, staff notes and other customers’ records do not belong in that text.
The model sees a tool name and a result, not the bearer token. Your backend holds third-party keys. Filter documents to what this user may see before they enter the prompt, and do not load staff-only ticket notes into a customer assistant.
Write only policy you would say in public, because a user can ask for a paraphrase. Enforcement stays in code, where quoting the rule changes nothing.
Tests you can rerun after every change
A few lines in a chat window are not a test. Cases against prompt injection belong in the repository and run when the prompt, the tools or the model change. Keep the set small and tie it to actions you cannot afford.
- A chat message and a fetched page that each ask for another customer’s order.
- A ticket or PDF that requests email to an outside address, or claims the writer is staff.
- A document that demands a refund and claims confirmation already happened.
- A request to print the system prompt, a key or another customer’s record.
- A reply that contains raw markup or a link to a host you do not own.
Send that text as fixture data. Assert the tool name, the arguments, whether confirmation was required, and whether the reply was escaped. Use fake orders and a mail sink that does not deliver. Fix failures in code, and pin the model version next to the cases.
What prompt injection defense does not solve
Stopping prompt injection does not make the model truthful. It can still invent a delivery date or a policy you do not have. Retrieval and a refusal when the source is missing are a separate problem. A wrong shipping date is not the same bug as mail sent to the wrong person.
Broken access control on the order API is still broken with no model involved. A stolen session skips the assistant. Secrets in a repository and a mail API that accepts any recipient are ordinary defects. The assistant makes them easier to reach.
Nobody can promise a model will never be confused by text. The reachable goal is narrower: a confused model cannot refund, read a stranger’s order, send mail to an address it chose, or change a password. You can test that on every change.
How we work
We treat prompt injection as a constraint on the assistant and on the APIs behind it.
- Discovery. List the actions available now and the ones you want next: orders, email, refunds, password resets and other writes.
- Threat notes. For each action, record how a chat, a page, a PDF or a ticket could request it, and which sources are fetched.
- Allow-list and checks. Narrow tools in code. The session chooses the customer. Money and account changes require confirmation or are not exposed.
- Tests. Store the cases in the repository and run them when the prompt, the tools or the model version change.
- Launch and care. Log tool name, arguments and result, without secrets. A new tool passes the same list before it is registered.
Why CodeWiz
- CodeWiz was founded in 2023 by Dragan Plavšić in Banja Luka, and works with clients locally and abroad, remotely.
- We build the assistant and the APIs it calls, so the permission check is part of the same delivery.
- AI, APIs and security are one practice. An assistant that writes a record is an API client with an untrusted caller.
- A short allow-list is safer than a long prompt that tries to talk the model out of a bad action.
- A parcel-status bot does not get a refund function just because a platform demo had one.
Frequently asked questions
Is this the same as jailbreaking the model?
Jailbreaking usually means pushing a model past a vendor safety policy. The prompt injection that matters here is untrusted text steering an assistant wired into your business. You can lose a jailbreak contest and stay safe if the refund function reads the session and ignores the model.
Will a stronger system prompt stop it?
A clearer prompt reduces accidents. It will not hold against text the model was asked to read. Use the prompt for tone and the tool list. Keep authorization in code. A control that works only while the model cooperates is not one you can rely on.
Should the assistant send email at all?
Yes, when the message is a template and the recipient comes from the order. No, when the model picks the address or the body. A free-form “email this address” tool is prompt injection waiting on the next ticket. Unusual mail should become a draft for a person.
Do we need this if the bot only answers questions?
You need output encoding if the answer is rendered as HTML, and a plan for the day someone adds order lookup or outbound mail. Leave tools unregistered until the checks exist.
How do you test without touching production?
Run the cases on staging with fake orders and a mail sink that does not deliver. Assert that a forbidden tool was not called, or that the customer id came from the session. No live refund and no real mailbox are required.
Review the assistant before it can act
If the assistant can look up an order, send an email or change a record, we can map the tools, move the checks into code and leave tests you can rerun. Tell us which actions should be allowed. Start at the contact page, or see our services.