AI Security Is an Accreditation Problem, Not a Model Problem

Prompt injection has been the first entry on the OWASP list of LLM application risks (opens in a new tab) since the list existed, and it is still there because the underlying problem has no clean fix. The reason is not that vendors are careless. It is that the attack is not a defect in the model.

An instruction arriving inside a document, a web page, an email or a ticket is data. The model reads it as instruction. That is the design, not a bug in the design.

Most AI security coverage treats this as an embarrassment problem: a chatbot says something it should not, the screenshot circulates, the vendor patches. In a system that has to be accredited, the same mechanism is a control failure, and it invalidates an assumption the whole authorization process is built on.

The system's behavior is no longer determined by its code

An authorization to operate rests on a simple premise. The system does what its configuration says it does. You review the architecture, you baseline the configuration, you test the boundary, and you re-test when the configuration changes. Two identical deployments behave identically.

A model breaks that. Two identical deployments behave identically only while they read identical input. The moment the system retrieves a document, the behavior is a function of the document. You cannot baseline that, and you cannot diff it.

This is why "we hashed the model weights" is not a configuration control. The weights did not change. The input did.

Accreditation assumes behavior follows code. With a model, behavior follows input.

The second half of the problem is that the input is now a channel an outsider can write to. Anything the system ingests, from a vendor PDF to a support ticket to a calendar invite, is a place where an attacker can place text. If the agent can act on what it reads, that text is remote control.

Three attack classes worth planning for

The demonstrations change every year. The classes do not.

Instruction injection. Text in the data stream tries to redirect the agent. The simple version asks the model to ignore its instructions. The durable version does not need to. It supplies a plausible next step in a workflow the agent is already authorized to perform, and the agent takes it. Multi-turn erosion is the patient variant: each message is unremarkable, and the cumulative drift moves the agent outside its intended role.

The common thread is that the attack does not need to defeat a guardrail. It only needs the agent to have permissions worth misusing.

Training and retrieval poisoning. A model fine-tuned on data you did not curate inherits whatever was in that data. Retrieval systems have a related weakness: whoever can write to the index can shape the answer. The specific rates quoted in vendor research do not transfer to your corpus, so treat any percentage as a property of their experiment and not a forecast of yours.

Output handling. The model returns text that some other system executes. Concatenated into a query, a shell command or a config file, that text is code. This class is the easiest to fix and the most commonly skipped, because the demo always stops at the chat window.

Where the boundary actually is

In a federal environment, the interesting control is not the network boundary. It is the data flow boundary. An agent that can read a shared drive, query a database and send email sits across three boundaries at once, and it does so with one credential.

That is the specific failure mode to design against: an agent with broad read access and any egress can assemble a document nobody intended to assemble and send it somewhere nobody approved. Every individual permission was granted for a reason. The combination is the incident.

The controls that map to accreditation are unglamorous.

• Separate instruction from data. Tool calls with typed arguments, not prose that the model is expected to obey. A retrieval result should arrive in a field the model is told is untrusted.
• Bound the effects, not the words. An allowlist of tools and destinations does more for you than any content filter. If the agent cannot reach the destination, the injection has nowhere to go.
• Gate the consequential actions. Read freely. Require a blocking approval for outbound sends, permission changes and anything that spends money. This is the record argument, and I have written about it at length in the case for an agent chain of custody.
• Validate the output where it lands. Schema validation, parameterized queries, no raw shell, no raw HTML. Treat every model response as hostile input to the next system.
• Log the input, not just the action. The input hash is what lets you answer why a decision happened. Without it, the log tells you what the agent did and not what it was told.

A content filter is a mitigation. It is not a boundary, and it cannot be the thing your authorization depends on.

Stop calling it secure by design

Two claims deserve to be retired.

The first is that a model can be made safe by training. It can be made safer for a general population, which is a real achievement. It cannot be made safe against an adversary who controls the input to your specific deployment, because the adversary gets to choose the input.

The second is that red-teaming produces a security property. A red team finds inputs that worked on a Tuesday. It does not establish that no other input works. That is not a criticism of red teams. It is a statement about what their output can support in an authorization package, and the honest answer is "evidence of diligence," not "evidence of safety."

What you can put in the package is narrower and more useful: the effects the system is permitted to produce, the gate in front of each class of effect, and the record that proves the gate ran. Those are controls you can test, and a test is the only thing an assessor can accept.