ENGINEERING · 11 SEP 2026 · 9 MIN

RAG vs fine-tuning: which one your product needs

For most product teams, the answer to RAG vs fine-tuning is RAG: if the feature needs to know your content, look it up at the moment of the question. Fine-tuning changes how a model behaves, not what it knows, and it earns its place later, for a fixed format, a house tone, a narrow classification task or a cheaper model at volume. Plenty of features need neither. This post covers what each approach changes, what it costs to keep running, how each one fails, and where the newer terms (MCP, agentic RAG, long context and CAG) fit.

What each one changes

Retrieval-augmented generation, or RAG, leaves the model alone. When a question arrives, your system searches your own content, puts the most relevant passages into the prompt and asks the model to answer from them. The idea comes from Lewis et al. (2020), who paired a trained model with a searchable index. Their argument was that the index lets you show where an answer came from, and update what the system knows without retraining it.

Fine-tuning changes the model itself. You train an existing model further on examples of an input and the output you want, and you get a new variant of it that you call instead of the original. It learns patterns: the shape of an answer, the labels you use, the way your support team writes.

The line that matters is knowledge against behaviour. Fine-tuning is a poor way to teach a model facts. Ovadia et al. compared the two for adding knowledge to a model and found that RAG did consistently better, both for facts the model had seen in training and for entirely new ones. Models struggled to pick up new facts through unsupervised fine-tuning (further training on plain text), and it helped only when the training data repeated each fact in many different wordings. Few teams want to prepare data that way.

A quick test: if the right answer would change when a document changes, it is knowledge, and it belongs in retrieval. If the right answer is the same content presented differently, it is behaviour, and fine-tuning might be the tool. Try a better prompt first in either case.

Cost and keeping it current

RAG costs an index and the work of keeping it in step with your content. When a document changes, you re-embed that document, which is cheap: OpenAI lists its text-embedding-3-small model at $0.02 per million tokens (pricing). Its hosted file search charges $0.10 per GB of storage per day after the first free GB, plus $2.50 per thousand tool calls. The real cost is engineering. You need parsers for your document formats, a sensible way to split them, permission checks so nobody retrieves a document they could not open, and a job that re-indexes when the source changes. Each answer also costs more, because the retrieved passages travel in the prompt.

Fine-tuning costs a labelled dataset, training runs, evaluation and a fresh round of all three whenever the behaviour you want changes. On OpenAI’s current price list, training gpt-4.1-mini costs $5 per million training tokens, and the tuned model is billed at $0.80 per million input tokens and $3.20 per million output tokens. That is double the $0.40 and $1.60 of the base model (pricing). The examples cost more than the compute. OpenAI accepts as few as 10 but recommends starting with 50 well-crafted ones and evaluating before adding more (guide). Google suggests thinking in terms of 100 or more (Gemini tuning docs). Someone has to write and check every one.

There is platform risk too. In May 2026 OpenAI told developers it was winding down self-serve fine-tuning. Since July, organisations that have not run a fine-tuned model in the past 60 days can no longer create training jobs, and the remaining customers lose that ability on 6 January 2027. Tuned models keep serving until their base models are retired (deprecations). Google still offers supervised tuning on current Gemini models, including Gemini 3.5 Flash (supported models). A tuned model ties you to one provider’s platform. An index works with whichever model you point at it.

Freshness follows from the same split. With RAG, a new price is live once its page is re-indexed. With fine-tuning, it reaches the model at the next training run, if at all.

So does debugging. A RAG answer can carry the IDs of the passages it used, so a user can check the source and an engineer can see exactly what the model was given. When a fine-tuned model says something wrong, there is no passage to inspect.

Where RAG fails

Most bad RAG is a retrieval problem. When answers come back wrong, teams tend to blame the model, try a bigger one or reach for fine-tuning. Usually the right passage never reached the prompt.

Barnett et al. catalogued seven ways RAG systems failed across deployments in research, education and biomedicine. The first two happen before the model sees anything: the answer is not in the documents at all, or it is but did not rank high enough to be returned. Others belong to the model: the answer was in the prompt and the model missed it, or ignored the format it was asked for. Their conclusion is that a RAG system can only really be validated in operation, and that it improves over time rather than being right at launch.

Retrieval improvements are measurable. Anthropic’s contextual retrieval adds a line to each chunk before indexing, saying which document it came from and what surrounds it. Combined with keyword search, that cut the rate at which the relevant chunk was missing from the top 20 results by 49%. Adding a reranking step took the cut to 67%, from a 5.7% failure rate to 1.9%.

So build retrieval so you can see it. A minimal flow looks like this:

question = user_input
chunks = search(index, question, top_k=20)     # keyword and vector
chunks = rerank(question, chunks)[:5]
prompt = f"""Answer only from the sources below. Cite source IDs.
If they do not answer the question, say so.

{with_ids(chunks)}

Question: {question}"""
answer = model.generate(prompt)
log(question, [c.id for c in chunks], answer)  # what you debug from

Collect a few dozen real questions with known answers. For each one, check whether the right passage is in the retrieved set before you read the generated answer. If retrieval is right and the answer is wrong, then look at the prompt or the model.

Retrieval gives a model something to answer from. Fine-tuning changes how it answers. When a RAG feature is wrong, check what it retrieved before you touch the model.

When fine-tuning wins

Fine-tuning is the right tool when the facts are fine and the behaviour is not, and a well-written prompt has already been tried. The providers’ own lists agree on where it helps. OpenAI names classification, nuanced translation, generating content in a specific format and fixing instruction-following failures (guide). Google names classification, summarisation, extractive question answering and chat (supervised tuning).

In product terms, that means a few recognisable cases.

A narrow classifier. Routing support tickets into your fifteen categories, or tagging listings with your own taxonomy.
A format that has to hold. Output a downstream system parses, where an occasional deviation breaks something.
A house voice. When examples show the register better than instructions can describe it.
Cost at volume. A smaller tuned model doing one repetitive job a large model was doing, without the long instructions and examples in every prompt.

That last case is the one OpenAI and Google both lead with: shorter prompts, lower latency and a smaller, cheaper model for a task where a large one is not cost-effective (OpenAI, Google). Run the arithmetic before you commit, since a tuned model costs more per token than the same model untuned.

Two conditions come first. You need an evaluation set, a fixed list of inputs with the outputs you would accept, and OpenAI says plainly to build it before investing in fine-tuning. You also need a task that will stay still for a while. Google recommends starting with prompting and moving to tuning only if you need to.

Using both together

The two are not rivals. A 2024 study on agricultural question answering by Balaguer et al. found fine-tuning raised accuracy by over 6 percentage points, and RAG added a further 5 on top.

The usual split is that retrieval supplies the facts and a tuned model supplies the form. A support assistant might retrieve from the help centre and answer in a fixed structure the ticketing system understands. If you do both, train on examples that include retrieved passages, so the model learns to answer from what it is given rather than from what it memorised.

Keep the order: prompt first, then retrieval, then fine-tune only what the evaluation set shows is still wrong.

Beyond plain RAG

Four other terms come up in the same conversations.

RAG vs agentic RAG. Plain RAG runs one search and one answer. Agentic RAG lets the model plan its own lookups: search, read, search again, check the result, pull from several sources. Singh et al. describe it as putting agents inside the retrieval pipeline so it can handle multi-step questions a fixed pipeline cannot. The price is more model calls, more latency and behaviour that is harder to test. Use it when questions genuinely need several lookups, and only once plain retrieval works.

RAG vs MCP. These are not alternatives. The Model Context Protocol is an open standard for connecting AI applications to external systems (MCP). A server exposes tools, resources and prompts, and the spec covers only how that context is exchanged, not how the application uses it (architecture). Your search can be one of those tools. MCP is the plug and RAG is one thing you plug in. It pays off when you want your data reachable from AI clients you do not build. Inside your own product, a direct call to your search does the same job.

RAG vs long context. Models now accept very long inputs, and current Claude models take up to a million tokens (context windows). Anthropic’s advice is that a knowledge base under 200,000 tokens, about 500 pages, can simply go in the prompt (contextual retrieval). With prompt caching, repeat reads of the same content are billed at a tenth of the normal input price, though the cache lasts five minutes by default (prompt caching).

RAG vs CAG. Cache-augmented generation takes that further. Chan et al. preload every document, cache the model’s working state and skip retrieval altogether. They are clear it suits a knowledge base of limited, manageable size.

Long inputs have limits. Anthropic’s own documentation says accuracy and recall fall as the token count grows. Liu et al. found models use information at the start and end of a long input best, and the middle worst. Putting everything in the prompt also skips per-user permissions unless you build a separate context for each user. For small, stable content that everyone may see, long context is the simplest option. For large, changing or permissioned content, retrieve.

Choosing RAG vs fine-tuning

Work down this list and stop at the first line that fits.

Does it need a model at all? Search, a filter or a rules table often does the job. When we integrate AI into existing products, we say so when a feature does not need a model.
Small, stable content that every user may see. Put it in the prompt and cache it.
Large, changing or permissioned content. Use RAG, and spend the effort on measuring retrieval.
Right facts, wrong shape, tone or labels. Fine-tune, once prompting has been tried and an evaluation set exists.
A narrow, repetitive task at high volume. Fine-tune a smaller model, and check the per-token arithmetic first.
Your data needed in AI tools you do not own. Publish an MCP server over your search.
Questions that need several lookups. Agentic RAG, after plain RAG works.