Long Context vs RAG
Compare long context, caching and RAG on cost, speed, accuracy and scale, and combine them sensibly.
If the model can read everything, why search?
Context windows have grown enormously. Early large language models could read a few thousand tokens at once; several current models accept a million tokens or more, enough for the institute's entire set of regulations, handbooks and circulars many times over. That raises a fair question: why build a retrieval system at all? Why not paste every document into the prompt and let the model find the answer itself?
Sometimes that really is the better design. Often it is not. The difference comes down to four things: cost, speed, accuracy on very long inputs, and how much material there is.
Cost and speed
A model charges for, and spends time processing, every input token on every request. Long context sends the whole collection with every question; RAG sends only a few relevant passages. Use the calculator to compare them for your own situation. The price is an illustrative figure, not a quote.
Long prompts are also slower: the model must process every token before it starts writing, so a 300,000-token prompt can take many seconds where a 2,500-token prompt takes a fraction of one.
Prompt caching narrows the gap. When the same long prefix (such as the whole handbook) is sent again and again, several providers store it and charge much less for reusing it, and respond faster. An approach sometimes called cache-augmented generation (CAG) builds on this: load a small, stable collection into the model's context once, cache it, and answer every question against that cached context, with no retrieval step at all.
Accuracy over very long inputs
Can a model reliably find and use one sentence in a million tokens? Simple "needle in a haystack" tests, which hide one unusual sentence in a long text and ask for it, are often passed almost perfectly by modern models. Real questions are harder: the relevant facts are not unusual, several similar passages compete, and answers need combining facts from different places. Research such as the 2023 "lost in the middle" study found that models used information in the middle of long inputs less reliably than information at the start or end, and quality on realistic tasks often drops as inputs get very long.
Retrieval has the opposite risk: if search misses the right passage, the model never sees it. So the comparison is between two kinds of error: missing evidence (RAG) versus evidence that is present but not used well (long context).
Size and change
Long context has a hard limit: the collection must fit. Many organisations' knowledge bases are far larger than any context window, often millions of pages. Long context also has to re-read everything on every question, so it gains nothing from documents not changing. RAG scales to any size and only re-processes what changes.
Small collections with few questions suit long context. Large collections, or heavy question volumes, favour retrieval.
Combining them
The two approaches combine naturally. A common pattern is to retrieve generously, then let a long-context model read more: retrieve the top 20 or 30 passages, or the whole of the 3 most relevant documents, rather than 3 small chunks. This keeps cost reasonable while reducing the chance that a needed passage is cut. Another pattern uses long context for a small, frequently used core (the main regulations, cached) and retrieval for the long tail of other documents.
The program below compares the daily token use of three designs for the same workload.
Expected output:
Long context (everything, every time) 150,000 tokens/prompt 300.0 M tokens/day
RAG (5 passages of 400 tokens) 2,500 tokens/prompt 5.0 M tokens/day
RAG + long read (top 3 whole docs) 24,500 tokens/prompt 49.0 M tokens/day
Each design is reduced to one number, the tokens in a single prompt. Multiplying by the 2,000 questions asked each day gives the daily total. Change questions or the passage sizes to match a system you have in mind.