RAG vs fine-tuning: when each one wins

Retrieval-augmented generation and fine-tuning solve different problems. RAG gives a model facts at query time by searching your data and putting the relevant passages in the prompt. Fine-tuning changes the model's weights so it behaves differently: a consistent format, a house tone, a narrow skill it does reliably.

The common mistake is fine-tuning to teach a model facts. It works poorly, it goes stale the day your data changes, and you can't cite where an answer came from.

When RAG wins

The answer depends on information that changes. Product catalogues, policies, tickets, contracts and internal docs all change weekly. A retrieval index can sync in minutes. A fine-tune has to be rebuilt.

Answers need citations. Legal, finance, healthcare and support use cases usually need to show the source. RAG returns the passage it used. A fine-tuned model can't tell you where a fact came from.

Access varies by user. Permissions-aware retrieval filters documents by who is asking. Knowledge baked into weights is visible to everyone who can call the model.

You have a lot of content and little labelled data. RAG needs documents and a set of test questions. Fine-tuning needs hundreds to thousands of high-quality input and output examples.

When fine-tuning wins

You need a consistent output format or style at high volume, and prompting gets you most of the way but not reliably enough.

You want a small, cheap model to do one narrow task as well as a large one. Distilling a frontier model's behaviour into a smaller model can cut inference cost and latency sharply for classification, extraction or routing.

The task is a skill rather than knowledge. Writing in a specific clinical note format, following a domain-specific reasoning pattern, or using a proprietary query language are behaviour problems.

When to use both

Many production systems combine them. Retrieval supplies the facts. A lightly fine-tuned model formats the answer, follows house rules and refuses when the retrieved context doesn't support a claim.

The order matters. Build retrieval first and measure it. Most teams find that good retrieval plus a well-written prompt on a strong model closes the gap, and the fine-tune never becomes necessary.

Cost and maintenance compared

RAG costs more per request because the prompt carries retrieved context, and it needs an ingestion pipeline and an index to maintain. It costs less to change, because updating knowledge means re-indexing, not retraining.

Fine-tuning costs more up front in data preparation and training runs, and every base model upgrade means retraining and re-evaluating. It can cost less per request when it lets you run a smaller model with shorter prompts.

A quick decision rule

If the problem is that the model doesn't know something, use retrieval. If the problem is that the model knows enough but doesn't behave the way you need, consider fine-tuning. If you can't tell which problem you have, build an eval set first. The scores will tell you.