Article
Every AI helpdesk has the same structural exposure: it reads text written by strangers, and then it acts. Here is what that looks like in practice, and which defences are real.
A language model reads one stream of text. Your instructions and the customer's email arrive in the same channel, and the model has no reliable way to tell which is which. So a customer who writes instructions gets instructions followed.
A support desk is an unusually good target, because unlike most AI features it is designed to accept unsolicited text from anyone in the world, and it usually has permission to do things: classify, prioritise, draft, escalate, sometimes act.
Priority manipulation. Text in the email instructing the triage step to mark it critical. Cheap, low-stakes, and effective — an attacker who can jump the queue at will has a denial-of-service against your SLA that looks like ordinary traffic.
Draft poisoning. Instructions aimed at the reply drafter, so the suggested response contains something the attacker chose — a link, a phone number, a claim about your refund policy. The defence people assume is "an agent reviews it", and the reason that is weaker than it sounds is that a plausible draft on a busy afternoon gets skimmed, not read.
Classification evasion. Text crafted to be scored as low severity so it sits unanswered past its SLA — useful if the message is a fraud report or a chargeback notice.
Data exfiltration through a reply. The serious one, in any desk where the model can see more context than the current ticket.
The attacks above fire once, against the person who sent them. A knowledge base changes that.
The obvious feature for an AI helpdesk is learning from resolved tickets — index the answers, reuse them. Now consider what an attacker does with it. They send a ticket containing instructions. The exchange is indexed. Weeks later a genuine customer asks a related question, the poisoned entry is retrieved as context, and the payload fires against a different customer than the one who planted it.
That inversion is what makes it worth designing around. The attacker is not in the conversation any more. The blast radius is everyone who asks a similar question, and the forensic trail runs backwards through a system that has no reason to log it.
The control we settled on is unglamorous: nothing enters the knowledge base automatically. An agent proposes an entry from a real exchange, reads the generalised version, and approves it. Human promotion is slower than automatic indexing, and it closes the persistence problem, the PII-leak problem and the "we indexed a wrong answer" problem in one move — which is why it survived the design review that automatic indexing did not.
Fencing untrusted content. Worth doing; not a solution. Wrap customer text in unambiguous delimiters, tell the model everything inside is data rather than instruction, and repeat the instruction after the content as well as before. Strip any fence markers the customer themselves wrote, or they will forge the closing delimiter. This meaningfully raises the bar. It does not make injection impossible, and anyone telling you otherwise is selling something.
Constrain the output shape. A classifier that may only return one of a fixed set of categories cannot be talked into returning prose. The model's blast radius is bounded by what its output is allowed to be, and that bound holds even when the injection succeeds.
Keep the model away from the decisions that matter. Ours proposes; a person disposes. Nothing the model produces reaches a customer without an agent sending it.
Scope every retrieval. If a model can see another tenant's data, injection
becomes exfiltration. Ours is scoped by path rather than by query filter — a forgotten
where clause returns everything, a wrong path returns nothing. The difference is
which way the mistake fails.
Write the check, not the comment. A note saying "remember to fence this" reaches whoever is already reading that file. We have a test that walks every call into the model and fails the build if one of them passes unfenced text, because the fifth call site added in a hurry is the one that will not have read the note.
That last one separates a vendor who has thought about this from one who fixed it once. The answer should be a test, not an intention.
Text written by a user that a language model reads as an instruction rather than as data. Because a model receives your instructions and the user's content in the same stream, a user who writes instructions can get them followed. On a support desk this is structural: the product exists to accept text from strangers and then act on it.
Because they make the attack persistent and shift who it hits. An attacker sends a ticket containing instructions; the exchange is indexed; weeks later that entry is retrieved as context for a different customer's question and the payload fires against them. Requiring a human to approve every entry closes it.
No, and vendors claiming otherwise should be treated with suspicion. It can be made much harder — fencing untrusted content, constraining output shape, keeping the model out of consequential decisions, scoping retrieval so injection cannot become exfiltration, and enforcing all of it with tests rather than conventions.