Sovereign LLMs: Open Models for On-Prem Government AI

A model you can download is not a model you can defend.

That distinction is the whole problem with the sovereign AI conversation. The pitch is simple: take an open-weight model, run it on your own hardware, and keep the data inside the boundary. The data boundary is real, and it is the easy part. The hard part is everything that makes the model an accountable component of a system: where the weights came from, what they were trained on, how they change, and who can prove it six months from now.

Sovereignty is custody, not a license label

"Open source" is used loosely in this market. A model can have downloadable weights and still carry a license with restrictions on commercial use, acceptable use, or redistribution. That is open weights, not open source in the OSI sense. The difference matters to a lawyer and to a program office.

Sovereignty is a property of the deployment, not the license. A model is sovereign in your environment when you control the artifact, the runtime, the updates, and the data path. That means three things:

• You can inspect the weights and the inference code.
• You can run the model without a network connection to the vendor.
• You can show where the model came from and what changed between versions.

A model that satisfies the first two and not the third is a science experiment. A model that satisfies all three is a candidate for a controlled environment.

What open weights buy

The benefits are real. Local execution removes the data egress problem. Inspection lets a security team look for unexpected behavior. Fine-tuning lets a team adapt a general model to a narrow vocabulary and task. A pinned version lets a program reproduce a result.

The cost profile is also different. A hosted API charges per token. A local model charges for hardware, electricity, and the people who run it. The second number is harder to forecast and easier to hide. A program that compares the two by looking only at the token price is comparing a variable cost to a capital cost.

Open weights also create a maintenance obligation. A model is a dependency. It has versions, defects, and a support lifecycle. If the vendor stops publishing updates, the program owns the model forever. That is a legitimate choice. It is not a free one.

What open weights do not buy

They do not buy a certification. There is no stamp that says a model is safe for controlled unclassified information. The accreditation boundary is still built from controls, configuration, and evidence, and the model sits inside that boundary as one more component.

They do not buy provenance. A weights file tells you what the model is, not what it was trained on. A data card helps. A public paper helps. Neither is a substitute for a supply chain review when the model is part of a system that handles sensitive data.

They do not buy indemnity. If the model produces a wrong answer that causes harm, the license is not a shield. The program owns the decision to use it.

A local model moves the trust question from the vendor's network to your own. It does not remove the question.

A deployment pattern that works

Start with a bounded task. Summarizing internal documents, classifying incoming correspondence, drafting a first pass, searching a known corpus. A task with a clear input, a clear output, and a human reviewer is easier to evaluate than a general assistant.

Keep the model outside the trusted boundary. It should not have write access to production systems, credentials it does not need, or a tool that can send data out. The model proposes. A person or a deterministic service acts.

Build a small evaluation set from your own documents. Public benchmarks tell you how the model performs on public problems. They do not tell you whether it can read your forms, your acronyms, or your handwriting. A hundred representative cases with known answers is worth more than a leaderboard.

Pin the version and test upgrades in a sandbox. A model update is a change to a production component. Treat it like one.

Log the prompts and the outputs, with a retention policy and redaction. The log is how you investigate a bad output and how you show an auditor what the system did.

Have a fallback. A smaller model, a rules-based path, or a manual process. A system that stops working when the model stops working is not resilient.

The questions to ask

Before a program adopts a model, it should be able to answer five questions.

Where did the weights come from, and what license governs their use? What data was the model trained on, and what does the vendor disclose about it? What is the update cadence, and can the program freeze a version? What is the support path if the model fails or the vendor exits? Can the model run air-gapped, and what does it need to operate?

A vendor that cannot answer those questions is selling a demo. A program that cannot answer them is not ready to deploy.

Sovereign AI is not a model you host. It is a system you can explain. The weights are the beginning of that work, not the end.