Pick the ML Type You Can Acceptance-Test

The machine learning type gets chosen in a design meeting and then defended in a contract. Those are different conversations, and the second one is harder.

In a design meeting you compare methods on accuracy, novelty and available data. In a contract you have to answer a narrower question: when the system is delivered, what will you measure to decide it works? That question eliminates some methods before the first experiment.

I have watched programs choose a method they could not test, then spend the acceptance period arguing about whether the output was good. The dispute was not technical. The method never came with a definition of correct.

The type is an evidence decision

Each family of methods implies a different kind of proof. Not a different kind of math, a different kind of argument.

Supervised learning gives you a target. You can hold out examples, compare predictions to labels, and report a number. The number is only as good as the labels, and the labels are only as good as the person who produced them, but the shape of the proof is clean: here is a set the model never saw, here is how often it agreed.

Unsupervised learning has no target. The model finds structure, and someone decides whether the structure is useful. There is no held-out answer key, because there is no answer. The proof has to be about the decision the structure feeds, not about the structure itself.

Reinforcement learning has a simulator. The policy is only as good as the simulator's fidelity, and the proof is a claim about the simulator as much as about the agent.

The method you cannot test is the method you will argue about in month nine.

What each one costs at acceptance

Supervised is the easiest to contract and the easiest to fake. The test set is the whole game, and if the vendor holds it, you have bought a claim rather than a measurement. Two things fix that: the government holds a set the vendor never trained on, and the acceptance criteria name the metric and the threshold before the award.

The subtle risk is leakage. If the test set was drawn from the same collection as the training data, or if near-duplicate records span both, the reported accuracy is inflated and nobody finds out until the system meets the world. Ask how the split was made. A vendor who cannot describe the split does not have one.

Unsupervised is the hardest to contract, because "useful clusters" is not a measure. The workaround is to stop measuring the model and measure the workflow it informs. If the output routes cases for review, acceptance is the review rate and the miss rate on a sample the analysts label. That turns an unsupervised method into a supervised evaluation, which is the honest way to test one.

Reinforcement learning is the most expensive to contract, because you are accepting a simulator and an agent at once. The simulator has to be validated against something real, the reward function has to be inspected for what it actually rewards, and the policy needs guardrails that hold outside the training distribution. Programs that skip the simulator validation are accepting a demo of a game.

The clauses that follow

Four clauses carry most of the weight, and none of them mention an algorithm.

1. Name the acceptance metric and threshold before award. Not "acceptable accuracy." A metric, a threshold, and the population it applies to.
2. The government holds the test data. Delivered with the system, sealed, drawn from a period the model was not trained on.
3. Define drift review. Who re-measures, how often, and what happens when the metric falls below the threshold in production. A model that degrades silently is a finding waiting to happen.
4. Require the failure analysis. For every false negative in the acceptance sample, a written cause. This is where you learn whether the vendor understands the system or only the score.

A fifth clause is worth adding for anything that informs a decision about a person, a payment or a case: the system must expose the inputs it used. Not the weights. The inputs. If the answer cannot be traced to data, the acceptance test is the last time anyone will be able to explain it.

Why this ordering works

Choosing by testability sounds like it puts the method last. In practice it puts it in the right place.

If you can define the acceptance test, you know what data you need, what labeling effort the test requires, and what the vendor has to expose. The method follows from those constraints, and it usually follows quickly. If you cannot define the acceptance test for any method, the program has a requirements problem, and no amount of model selection will fix it.

The alternative, choosing a method first and inventing a test afterward, produces the most common failure I see in AI procurement: a system that passes its acceptance review and fails its first year of use, with no mechanism to notice the difference.

Pick the method you can prove. Then write the proof into the contract, and keep the answer key.