Two different levers: what the model can look up versus how the model behaves.
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model can look up at answer time | The model weights, so its default behavior |
| Good at | Grounding answers in private, current, or changing facts | Consistent format, tone, and narrow task behavior |
| Weak at | Fixing tone or output structure | Reliably adding or updating facts |
| Updating knowledge | Reindex the documents, minutes | Run a new training job, re-evaluate, redeploy |
| Main cost drivers | Embeddings, a vector store, extra input tokens per query | Training runs, evaluation sets, hosting or price premiums |
| Data you need | Your documents, roughly as they are | Hundreds to thousands of curated example pairs |
| Attribution | Can cite the retrieved source | None, knowledge is baked into the weights |
| Typical failure mode | Retrieval pulls the wrong context | Overfitting, regressions on general tasks |
| Hosted support in 2026 | Universal | Shrinking; OpenAI is winding down self-serve fine-tuning |
| Do they combine? | Yes, retrieval supplies the facts | Yes, tuning shapes how the model uses them |
RAG has mostly boring infrastructure costs. Embedding a corpus is a one-time job that usually costs less than lunch, and a small vector database is cheap or free to run. The real recurring cost is tokens: every query carries retrieved context as input tokens. Prompt caching softens this a lot. Anthropic bills cache reads at 0.1x the base input price, and OpenAI lists cached input at about 10 percent of the standard rate, so a stable system prompt plus stable context gets close to 90 percent off on repeat requests.
Fine-tuning costs show up in different places. Before its wind-down announcement, OpenAI listed supervised fine-tuning for GPT-4.1 at $25 per million training tokens and reinforcement fine-tuning for o4-mini at $100 per training hour, with fine-tuned inference priced above the base model. On open-weight models the training itself, usually LoRA on a rented GPU, is often cheap. The ongoing costs are serving the custom model and, above all, evaluation. Every base model upgrade invites you to retrain and retest, and that engineering time is the line item nobody budgets for.
The asymmetry that matters is the cost of change. When facts change, RAG reindexes in minutes. A fine-tuned model has to be retrained, re-evaluated, and redeployed, so stale knowledge is its default state.
Pick RAG when the job is answering from facts: internal docs, product data, policies, tickets, anything private or changing. Retrieval gives you three things fine-tuning cannot: fresh knowledge without retraining, citations you can show a user, and per-query access control, since you can retrieve only what the person asking is allowed to see.
It is also the cheaper experiment. You can stand up a useful RAG prototype in a day with an embedding model and a vector index, and the failure modes are inspectable. When an answer is wrong, you can look at what was retrieved and fix the chunking, the index, or the query. When a fine-tuned model is wrong, your main lever is more data and another training run.
Pick fine-tuning when the problem is behavior, not knowledge: a strict output format the model keeps drifting from, a house writing style, a narrow classification task, or tool-use patterns you want to make reflexive. Fine-tuning can also make a small, cheap model perform like a bigger one on a single narrow task, which matters at high volume where per-token cost dominates.
Be honest about the prerequisites. You need hundreds to thousands of good examples, an evaluation set you trust, and the willingness to repeat the loop whenever the base model changes. In 2026 this mostly points at open-weight models served through providers that host LoRA adapters, since OpenAI is retiring self-serve fine-tuning and its current flagship models never offered it.
First, this is not actually a versus. The two techniques move different levers, and the strongest production systems I have seen use retrieval for facts and prompting, or a light fine-tune, for behavior. The moment someone frames it as either-or, they are usually about to fine-tune facts into a model, which fails slowly and expensively.
Second, the vendor signal from 2026 is loud. OpenAI told developers on May 7, 2026 that self-serve fine-tuning is winding down: new organizations were cut off immediately, and no customer can create new fine-tuning jobs after January 6, 2027. Its stated reason was that newer base models follow instructions and formats well enough to make most fine-tuning unnecessary, and its GPT-5 flagship models never supported fine-tuning at all. Whatever you think of that reasoning, betting a product on hosted fine-tuning now carries platform risk that RAG does not.
Third, a fine-tuned model still needs retrieval. Tuning does not un-freeze the model in time. If your fine-tuned support bot needs to know this week's pricing, you are back to putting documents in the prompt anyway.
Start with a strong prompt and RAG. That combination covers most business use cases I get hired for, updates in minutes, and produces answers you can trace to a source. Reach for fine-tuning only when the behavior you need is stable, prompting has demonstrably failed, and you have the examples and evaluation discipline to maintain it, which today usually means an open-weight model rather than a hosted API. And treat them as layers, not rivals: retrieval for what the model knows, tuning for how it acts.