A Million Tokens Did Not Kill Retrieval

When Gemini 2.5 Pro arrived in March 2025 with a context window of about a million tokens, the first reaction was that retrieval was obsolete. Why build a pipeline when you can paste the corpus into the prompt?

The second reaction, usually a month later, is that the bill went up and the answers did not get better. Both reactions are about the same misunderstanding. A bigger window changes where you put the evidence. It does not change what the model does with it.

What a large window actually buys

A million tokens is a lot of reading. It is enough for a contract set, a year of reports, or a technical library. For analysis that touches a bounded corpus once, that is a real capability and it removes a class of engineering work.

The work it removes is the plumbing: chunking, embedding, indexing, and the ranking layer that decides what the model sees. For a one-off review, skipping that plumbing is cheaper and simpler.

The work it does not remove is the decision about what the model should see. That decision is still yours. A window is a capacity, not a judgment.

The middle of the window is still the middle

The most useful research on long context predates the million-token window. A 2023 paper, Lost in the Middle (opens in a new tab), showed that language models retrieve information more reliably from the beginning and end of a long context than from the middle. Performance degraded as the relevant passage moved toward the center, and the degradation was not small.

That finding has been refined since, and it has not been repealed. A model that accepts a million tokens does not attend to all of them equally. It is a statistical system with position effects, and the effects show up in the cases that matter: the clause buried in section 14, the exception in the appendix, the sentence in the middle of a 400-page production.

The practical consequence is unglamorous. If you are going to use a long window, you have to test position sensitivity on your own documents. Put the same critical fact at the beginning, the middle, and the end of the same prompt and see whether the answer changes. If it does, your architecture has a silent failure mode.

Retrieval is an evidence function

The stronger argument for retrieval is not cost or accuracy. It is provenance.

A retrieval system records what it retrieved. That record is the difference between an answer and a defensible answer. When a reviewer asks why the model concluded that a contract term was acceptable, the retrieved set is the evidence. A full-context prompt gives you a transcript of everything you pasted and no record of what the model used.

Retrieval also carries the access control. A document store has permissions. A retrieval layer can enforce them per user, per query, and per document. Pasting a corpus into a prompt collapses that model into a single boundary, which is fine for a public library and unacceptable for anything with compartments.

There is a third function that gets ignored until it breaks: freshness. A retrieval index can be updated in minutes. A prompt with a corpus baked into it is as current as the last time someone assembled it.

The architecture that survives

The useful pattern is not long context versus retrieval. It is retrieval to a set, then long context to read the set.

1. Retrieve broadly. Use search to narrow a large corpus to a candidate set, with permissions applied and the query logged.
2. Read deeply. Pass the candidate set to a long-context model and ask for analysis across the whole set, not a summary of each document.
3. Log the set. Store the retrieved documents, their versions, and the prompt. That log is the audit trail, and it exists nowhere else.
4. Test position. On your own corpus, verify that the answer does not change when the key document moves.
5. Cache what repeats. If the same corpus is queried repeatedly, cache the processed context rather than reprocessing it. That is a cost control, not an architecture.

This pattern uses the window for what it is good at, which is holding a lot of text in one reasoning pass. It uses retrieval for what it is good at, which is deciding what belongs in the pass and proving it later.

The window is a tool, not a strategy

The million-token window is a genuine step. It makes a class of analysis practical that used to require a pipeline, and it will keep getting cheaper.

It does not repeal attention. It does not create a record. It does not enforce permissions. Those are properties of the system around the model, and they are the properties that decide whether the output can be used for anything that matters.

The teams that get value from long context will be the ones that treat it as a bigger desk, not a replacement for the filing system.