Intelai
A six-question test for any AI tool you plan to use with a federal customer — and the answers that should disqualify it.
A deck lands in my inbox. The line above it is the whole question: can we put this in front of a federal customer?
I print it, so the objections sit next to the claims that caused them.
In the middle there is a benchmark chart with a bar that goes furthest right. Near the back, a slide titled Security with one line about encryption. The appendix carries a SOC 2 logo and no report.
I have written the same note more than once: where does the document go?
The deck is answering a question I already accept. Of course the model can summarize a regulation. I can too. What I have to decide, in writing, is whether I can put this in front of a federal customer and defend it to a contracting officer, a security reviewer, and the two people who will maintain it after I stop paying attention.
A capability comparison is a photograph of a moving thing. The model on the cover will be replaced before your renewal. The chart will be wrong, and nobody sends a memo.
Capability is the part that improves on the vendor's schedule. A new model does not move the boundary, make the record yours, add a gate, or build an exit. Those improve when someone builds them and keeps them working. That someone is usually you.
You are not buying capability. You are buying the part of the system that a better model does not improve.
Six questions. Not one of them is about what the model can do.
The order matters. Each question can end the conversation. Ask them in one call and write the answers in the vendor's words. Vagueness is an answer.
Ask where the prompt, the document, and the output live, who can read them, how long they are kept, and whether any of it can train a model. If the document is CUI, ask which clause defines where it may live, and who signs that it stayed there.
A good answer is specific: a region, a tenant, a retention window, a subprocessor list, and the clause behind the promise.
Disqualifying: encryption offered as a location. Encryption tells you how the bytes are stored. It does not tell you where they are. A SOC 2 report says someone examined a set of controls. It does not tell you where your document sat.
Ask for a record of every action, from a store you control. Not the log they keep for support. Yours.
A good answer names the fields: timestamp, actor, tool, input, output, and the approval. Append-only, and exportable.
Disqualifying: a history panel in the interface. A timeline is a view, not a record. If you cannot say which call produced a sentence, you cannot answer the contracting officer.
If you cannot reconstruct an action from your own records, you did not perform it. It happened to you.
Ask which actions require a human, and whether that is enforced in the execution path or only stated in a prompt.
A good answer is a list of gated actions with a runtime behind it. A denied action does not run. The denial is recorded, and the approver has a name.
Disqualifying: a standing instruction telling the model to ask before it deletes anything. An instruction is a preference. A gate is a wall.
A policy the model is asked to follow is not a control. It is a hope with a system prompt.
Ask what one real workload cost, including retries and tool calls.
A good answer is a measured number from a workload like yours, with the retry rate beside it and a clear answer on who pays when a run fails. Token price is a rate. Cost per task is a fact.
I see that only because I run my own gateway. Every call has a line. If a vendor cannot show you the line, you are budgeting from a rate card.
Ask what the system does when the model is wrong, and who catches it.
The best sentence a vendor can say is about what the system does when it does not know. An abstention, a threshold, a review step, a correction. A good answer describes a path, not an adjective.
Disqualifying: a claim that it is very accurate. Accuracy is a property of a test set, and your documents are not that test set.
The dangerous failure is not a wrong answer. It is a confident one nobody can trace.
Ask whether you can take the data, the prompts, and the logs with you.
A good answer is an export, a documented schema, prompts stored as text, and a transition term. A prompt you cannot export is configuration you rent.
Disqualifying: an assurance that your data is yours. It is true, and it is not a mechanism.
The exit test is not about trust. It is about whether the thing you built has a shape you can carry.
If the second, third, and sixth come back vague, the claim is marketing.
Write the answers down before you decide. One page is enough.
| Test | What to ask | A passing answer |
|---|---|---|
| 1. Data boundary | Where do the prompt, the document and the output live, and who can read them? If it is CUI, which clause defines the location? | A region, a tenant, a retention window, a subprocessor list, and the clause that binds them. |
| 2. Action log | Can I export a record of every action from a store I control? | Timestamp, actor, tool, arguments, result and approval, append-only and exportable. |
| 3. Permission gate | Which actions require a human, and is that enforced in the runtime? | A blocking gate that denies, records the denial and names the approver. |
| 4. Cost per task | What did one real workload cost, including retries and tool calls? | A measured number from a workload like mine, with retry rate and who pays on failure. |
| 5. Failure mode | What happens when it is wrong, and who catches it? | An abstention, a threshold, a review step, or a correction path. |
| 6. Exit path | Can I take the data, the prompts and the logs with me? | An export, a documented schema, prompts as text and a transition term. |
Twenty minutes will not make you an assessor. It will tell you whether this is a product or a demo with a logo on it.
The first failure is treating a demo as an assessment. In a demo the vendor's engineer is the human in the loop, and the checks are the ones they chose. You are buying the version that runs when their engineer is off the call, and the only way to test it is the record after the run.
A demo proves the system can work. It does not prove you can prove it worked.
The second is believing a model upgrade will fix a boundary problem. A better model produces a better answer from the same place. It does not move the data, add a gate, or build an exit.
If you build the audit side rather than buy it, I wrote about AI bills of material and what a reviewer can actually check.
I built these because I could not buy the answers to these questions about my own work.
HARBOR Port is a hosted gateway to federal data: eighteen federal sources across nineteen servers, 147 tools, with keys, quota, and an audit trail. port.harborgovcon.com
Now the part I cannot claim.
I cannot tell you which model is best for your documents. I have not measured your documents. A benchmark on someone else's corpus is evidence about that corpus.
I am not an assessor. When I say the record and the gate matter, I am telling you what I built, not what someone approved.
You will be asked to approve an AI tool this quarter, and the deck will make it a comparison. It is a decision about custody: where the data went, what record exists, who could have stopped it, what it cost, and whether you can leave.
Run the six questions in one call. If the record, the gate, and the exit come back vague, you have your answer, and it is no.
Capability will improve without you. The rest will not.
The stack you can defend is the one where the boring parts were already built, by someone who expected to be asked. If your team is stuck on exactly this call, working through it is what I do with a few firms each year: /services.