Retrieve a relevant slice, or ship the whole library with every request.
| RAG | Long-context window | |
|---|---|---|
| How it works | Search first, prompt with the top matches | Put whole documents in every prompt |
| Practical limit | Corpus size, effectively unbounded | The window: 1M tokens on current Claude models, 272K on GPT-5.5 |
| Cost per query | Input tokens for a few retrieved chunks | Input tokens for everything you send, every time |
| Long-prompt surcharge | Not applicable | Gemini bills input at 2x past 200K tokens; Claude 1M is standard rate |
| Latency | Retrieval plus a short prefill | Long prefill; cached repeats are faster |
| Accuracy risk | The retriever misses the right passage | Context rot: reliability drops as input grows |
| Freshness | Reindex the changed documents | Resend the changed documents, cache re-warms |
| Engineering effort | Chunking, embeddings, index, retrieval evals | Almost none to start |
| Cost control | Cheap by construction | Depends on prompt caching discipline |
| Best for | Big or growing corpora, high query volume | One document set, one deep pass, fast prototypes |
Prices below come from the vendors' own pricing pages as of August 2026. Anthropic bills its 1M-token context window at standard per-token rates on Claude 4.6 and later models, so a 900K-token request costs the same per token as a 9K one: $2 per million input tokens on Sonnet 5, $5 on Opus 5. Google tiers Gemini by prompt size: Gemini 2.5 Pro input is $1.25 per million tokens up to 200K and $2.50 beyond it, with output at $10 and $15 respectively, and the newer Gemini 3.1 Pro preview doubles those input rates. OpenAI lists a 272K window for GPT-5.5, and its GPT-5.6 family is priced in separate short-context and long-context tiers.
Now the math that matters. Suppose your knowledge base is 500K tokens and you answer questions against it on Sonnet 5. Stuffing the whole thing into each request costs about $1.00 per query in input tokens alone. A RAG pipeline that retrieves 5K tokens of relevant context costs about a penny. That is a 100x gap, and it compounds with every query.
Prompt caching narrows it. Anthropic bills cache reads at 0.1x the input price, so that cached 500K-token corpus drops to roughly $0.10 per query, and OpenAI lists cached input at about 10 percent of the standard rate. But caches come with terms: Claude cache writes cost 1.25x the base rate for a 5-minute window or 2x for an hour, and Gemini charges $4.50 per million tokens per hour just to keep a cache warm. Idle gaps re-bill the full prompt, and any change to the documents invalidates the cache. Caching makes long context viable for bursts of questions against one document set. It does not make it free.
Pick RAG when the corpus is bigger than any window, when it changes often, or when query volume is high enough that per-request token cost dominates. Retrieval also gives you citations and per-user access control, since you decide what enters the prompt on each request. Those two properties alone settle the question for most internal knowledge tools.
RAG is also the sober choice when accuracy at depth matters. A retriever that surfaces the right 5K tokens puts the evidence next to the question. The model does not have to find a needle in 500K tokens of hay, which the research below suggests models are worse at than their benchmark scores imply.
Pick long context for one-shot deep work on a bounded document set: reviewing a contract, auditing a codebase, comparing a handful of long reports. Retrieval would fragment exactly the cross-references you care about, and the whole point is that the model sees everything at once.
It is also the right way to start. If your corpus fits in the window, skip the vector database, paste the documents, and find out whether the product idea works at all. With caching, a prototype serving a small team against one document set can run surprisingly cheap. Add retrieval when volume, corpus growth, or the bill tells you to, and by then you will know exactly what the retriever needs to be good at.
Context rot is real and measured. Chroma's July 2025 technical report evaluated 18 models, including GPT-4.1, Claude 4, and Gemini 2.5, and found that reliability degrades as input length grows, even on tasks as simple as retrieval and text copying. The decline is continuous rather than a cliff, and distractor text that resembles the answer makes it worse. The earlier Lost in the Middle work by Liu et al. found the same shape: models use the start and end of a long context better than the middle. A million-token window is not a million tokens of uniform attention.
Second, the pricing pages quietly agree. Google charges double for input past 200K tokens, and OpenAI prices its newest models higher on long context than short. When vendors tier prices by prompt length, treating the full window as your default rather than a special case gets expensive fast. Anthropic is currently the exception, billing 1M-token requests at standard rates.
Third, this is a spectrum, not a fork. The pattern that wins in production is a hybrid: retrieval selects the candidate documents, and the big window lets you be generous about how much of each one you include. Coarse retrieval plus long context beats both extremes, because it spends tokens on recall where the retriever is weakest.
For a production system answering many queries against a large or changing corpus, I build RAG, and I use generous context per query because windows are big now. For bounded, deep, one-off analysis, I paste the documents and let the model read everything. The 100x cost gap and the context rot research both point the same way at scale. Long context changed how much retrieval you need. It did not replace it.