Synthetic Data Does Not Solve the Classified Problem

The pitch is clean. A generator learns the shape of a classified dataset, produces a synthetic copy, and hands you something you can train on in a commercial cloud. The first half of that sentence works. The second half is where programs stall for a year.

I have watched this pattern in federal data work more than once. The team that can generate data is not the team that can authorize its use, and the gap between those two teams is not technical.

Impact levels are the wall, and they are not going to move

The Department of Defense separates cloud environments by impact level. Level 5 covers controlled unclassified information, including export controlled data. Level 6 covers Secret. The commercial AI toolchain that everyone wants to use lives below that line, and the gap is not a configuration setting.

So the practical situation is this. The data that would make a defense model useful is the data the model is not allowed to see. Training on it requires an environment that most modern tooling does not run in, and a cleared team to operate that environment.

Synthetic data is an attempt to route around the wall rather than move it. That is a legitimate engineering strategy. It is also a strategy that creates a new set of problems, and most programs budget for the first set only.

Generation is not where it breaks

The techniques are mature enough. Generative models can produce imagery, time series, and structured records that look like the real thing to a statistical test. Simulation can produce scenarios that never occurred and might.

That is the part vendors sell, and it works well enough to get a pilot funded.

The failure is downstream. A synthetic dataset is not automatically unclassified. If the generator learned from classified inputs, the output can encode what it learned. A synthetic image can carry resolution, sensor characteristics, or collection geometry that the real system was built to protect. A synthetic logistics record can carry supplier relationships and lead times that are themselves sensitive.

Classification attaches to the information, not to the pixels. A convincing fake of a secret is a secret about the original.

That is why the release path runs through a classification review, and why that review is the schedule. It is manual, it is done by people who are already busy, and it cannot be automated away by a tool that guesses, because the cost of a permissive guess is a spill.

The validation problem is worse than the review problem

Suppose the review clears the dataset. You still have to answer a harder question: did the model learn the mission or the simulation.

The standard tests do not get you there. A distance metric can tell you the synthetic distribution is close to the real one. It cannot tell you whether the rare case that decides the engagement survived the compression. In defense data the rare case is the point. The common case is what the peacetime system already handles.

Holdout testing has its own trap. To test on real data you need access to real data, which puts you back inside the boundary you were trying to leave. Programs that plan for this end up with two teams, one cleared and one not, and a handoff that doubles the overhead nobody estimated.

The test I trust is adversarial. Give a red team the real distribution and the synthetic one and ask them to tell them apart. If they can, the model can too, and it will have learned the difference.

A synthetic dataset is a hypothesis about what matters. Validation is the experiment.

What a workable pipeline looks like

Four stages, in this order, with different owners.

Generation, scoped to one question. Do not build a synthetic version of your enterprise. Build a synthetic version of the one dataset that is blocking the model, and make the fidelity target explicit. What must be preserved, what may be approximated, what must be invented because the real data does not contain it.

Validation, before the review. Statistical checks, coverage checks against the operational cases you care about, and a red team that tries to separate the fake from the real. Write the results down in a form the reviewer can read, because the reviewer is the gate.

Review, by someone who did not build it. This is a separation of duties question, not a staffing question. The team that generated the data has an interest in it being released.

Monitoring, after deployment. A model trained on synthetic data fails differently from one trained on real data. It fails on the inputs the generator never imagined, and it fails confidently. Instrument the deployed model for the input classes the synthetic set underrepresented, because those are where the surprises will come from.

The honest accounting

Synthetic data is not a substitute for access. It is a way to make progress while access is being arranged. Programs that treat it as a substitute spend two years building a model on a world that does not exist and discover it in the first operational test.

Programs that treat it as a bridge get something useful. A model that handles the common cases well, a validation method that survives scrutiny, and a clear list of the cases where a human has to be in the loop because the training data was a guess.

That list is the deliverable. Not the dataset. Not the model. The list of what you know you do not know, written down before the first deployment, and kept current.

The classification problem is not going to be solved by better generators. It is going to be managed by programs that treat the review and the validation as the primary engineering work and the generation as the cheap part.

The generator is the easy purchase. The evidence is the program.