Intelai
DeepSeek Halved Its Hallucination Rate. Yours Is Different.
DeepSeek says its update cut hallucinations by up to half. That number came from its benchmarks, not from your task, and the gap is the whole job.
On May 28, 2025, DeepSeek shipped R1-0528, an update to R1. The release notes claimed the hallucination rate fell by as much as half on the benchmarks the company published, alongside the usual gains in reasoning.
The number is real in the sense that a lab measured it and reported it. It is not a number about your work.
That distinction is the one I care about, because hallucination rate is not a property a model carries around like its parameter count. It is a property of a model answering a particular question, with particular context, under a particular scoring rule. Change any of those and the rate moves. Usually it moves a lot.
A benchmark is somebody else's task
A published hallucination benchmark is a set of questions with known answers. The lab picks the questions, decides what counts as a hallucination, and reports a rate against that choice. DeepSeek's reduction is meaningful evidence that the model behaves differently than R1 did on those questions.
Now consider your corpus. A contract from 2009 with an amendment that contradicts clause 12. A maintenance procedure with a table that only makes sense next to a diagram. A policy memo whose appendix was superseded. None of that resembles a short-answer benchmark, and the failure mode that matters is not "wrong answer." It is a fluent answer that cites the wrong clause with total confidence, which is worse than no answer because it costs somebody an afternoon to catch.
The useful question is not whether the model hallucinates less. It is whether the model abstains when it should, on your material, and whether you can measure that without reading every output.
What a single rate hides
A hallucination percentage collapses at least three different behaviors into one figure, and the three have different costs.
Wrong and uncertain. The model gives a bad answer and hedges. Annoying, cheap, and correctable by a reader who is paying attention.
Wrong and confident. The expensive one. It produces a citation that does not exist, a number with a decimal point, a clause reference that sounds right. This is the failure that makes a reviewer trust the system less after they catch it, which is a much bigger loss than the error itself.
Abstained when it could have answered. The hidden cost on the other side. A model tuned to avoid hallucination will start refusing questions it could handle, and a system that says "I don't know" to a third of the queue has not solved a problem, it has moved it to a person.
A lab optimizing a single rate can trade one for the other without telling you which. When you evaluate, split the three. They route to different fixes.
Measure your own, in an afternoon
You do not need a research program. You need three hundred examples and a rule for grading them.
Pull three hundred items from the work the system will actually see. Real questions, real documents, including the ugly ones. Write the answer you would accept for each, and label what a good response looks like: the specific citation, the number, the abstention if the source does not contain it.
Then run the model and grade each output into four buckets: correct, correct but unusable, wrong, and refused. Compute two numbers, not one. Accuracy on the items it answered, and coverage, meaning the share it was willing to answer at all.
The pair is the decision. A model at 95% accuracy on 60% coverage is doing less work than a model at 88% on 95% coverage, and which one you want depends on what the human downstream costs.
Run the same set against the previous model version when a vendor ships an update. That comparison is worth more than any published benchmark because it is the only one measured on the thing you are buying.
What to do with a vendor number
Treat a published rate as a filter, not a decision. It tells you the model is in the conversation. It cannot tell you whether it works in your workflow.
Two questions are worth more than the benchmark table in any procurement conversation.
What does the model do when the answer is not in the provided context, and can you see that decision in the output? A system that surfaces its abstention is auditable. One that guesses is not.
What happens to your data when the model is wrong? If a wrong answer flows into a record that a regulator or an auditor sees, the hallucination rate is a compliance question, not a quality metric, and the threshold should be set by the consequence rather than by the average.
A model that is wrong less often is not the same as a system that fails safely. The second one is a design choice you make, and it survives the next version bump.
R1-0528 is a good model and the reduction is a genuine improvement. It just is not evidence about your corpus. Go get that evidence, because nobody else is going to produce it for you, and a vendor's benchmark will keep improving whether or not your system does.