RAG vs Fine-Tuning: How to Choose for Your AI Product

Every team building on large language models hits the same question within a few weeks: the model does not know our products, policies or data, so should we give it our documents at answer time, or train it on them? The first approach is retrieval-augmented generation (RAG). The second is fine-tuning. They are often described as alternatives, but they change different things, and choosing the wrong one is one of the most common reasons AI projects stall.

The one-sentence difference

RAG changes what the model sees; fine-tuning changes how the model behaves. With RAG, a search step finds the passages relevant to each question and adds them to the prompt, so the model answers from your content. With fine-tuning, you train the model on many examples of inputs and desired outputs, so it learns a pattern: a format, a tone, a classification scheme or a narrow skill.

The term RAG comes from a 2020 research paper by Lewis and colleagues at Facebook AI Research, which combined a language model with a searchable index of Wikipedia and showed better results on knowledge-heavy questions than the model alone (Lewis et al., 2020). The idea has since become the standard architecture for business assistants that need to answer from private knowledge.

Start with prompting, then RAG

OpenAI's own model optimisation guidance recommends evaluating a base model with good prompts and retrieval first, to establish a baseline, before reaching for fine-tuning. That order matters for a practical reason: you cannot judge whether fine-tuning helped unless you know what the simpler approaches achieve on the same test set.

A sensible sequence for most products is: write a clear system prompt with a few examples; add retrieval if the model needs facts it does not have; build an evaluation set from real questions; and only then fine-tune for the specific failures that remain. Our guide to AI agents vs workflows vs chatbots covers the architecture choices that come before this decision.

When RAG is the right choice

• The information changes. Prices, policies, stock, product documentation and procedures change weekly. Updating an index is quick; retraining is not.
• Answers must be traceable. Support, legal, finance and healthcare answers should show their sources. RAG makes citations straightforward.
• Access depends on the user. A sales rep should not see HR files. RAG can filter documents by permission before the model sees anything.
• The knowledge is large. Thousands of documents will not fit in a prompt and cannot be reliably memorised by training.
• You want to switch models. The index works with any model, so you can move to a cheaper or private model later without redoing the work.

When fine-tuning earns its cost

• Strict output formats. Extracting fields into a fixed JSON schema, writing reports in a house structure, or always following a template.
• Narrow classification at volume. Routing tickets, tagging documents or scoring leads, where a small fine-tuned model can be fast and cheap.
• Consistent tone and terminology. A brand voice or a specialised vocabulary that prompts alone do not hold reliably.
• Shorter prompts at scale. If every request carries a long block of instructions and examples, training them in can cut token costs and latency.
• Smaller or private models. Adapting an open-weight model so it performs well enough to run on your own infrastructure. See private LLM deployment.

What modern fine-tuning looks like

Fine-tuning used to mean retraining every weight of a model, which was expensive. Today most teams use parameter-efficient methods. LoRA freezes the original weights and trains small low-rank matrices; its authors reported roughly 10,000 times fewer trainable parameters and a threefold reduction in GPU memory compared with full fine-tuning of GPT-3 175B (Hu et al., 2021). QLoRA goes further by quantising the frozen model to 4 bits, and its authors fine-tuned a 65-billion-parameter model on a single 48 GB GPU while matching 16-bit fine-tuning performance on their benchmarks (Dettmers et al., 2023).

Hosted providers also offer fine-tuning through their APIs. The practical constraint is rarely compute any more; it is the quality of your training examples. A few hundred carefully reviewed examples usually beat thousands of noisy ones.

Why most RAG problems are search problems

When a RAG assistant gives a poor answer, the model is often not to blame: the right passage was never retrieved. Anthropic published a useful illustration. Adding a short, generated context to each chunk before embedding cut the rate of failed retrievals by 35%, combining that with keyword (BM25) search cut it by 49%, and adding a re-ranking step brought the reduction to 67% in their tests (Anthropic, Contextual Retrieval).

Long prompts are not a substitute for good retrieval either. Researchers found that models use information at the start and end of a long context better than information in the middle, and that performance drops as contexts grow (Liu et al., 2023). Retrieving a few highly relevant passages tends to beat pasting in everything.

1. 01Clean the source documents: remove duplicates, outdated versions and boilerplate.
2. 02Chunk by meaning (sections, headings, Q&A pairs) rather than fixed character counts.
3. 03Combine semantic search with keyword search so product codes and names are matched exactly.
4. 04Re-rank the top candidates before sending the best few to the model.
5. 05Attach metadata (date, product, region, permission) and filter on it.
6. 06Instruct the model to answer only from the provided passages and to say when they do not cover the question.

Using both together

The strongest systems often combine the two. RAG supplies the facts for each request, and a fine-tuned model has learned to use retrieved context well: to quote precisely, follow your answer format, decline when the sources are silent and keep a consistent tone. This combination can let a smaller, cheaper or private model match a larger general model on your specific task.

A decision checklist

Measure before and after

The only reliable way to choose is to test. Collect 50 to 200 real questions with known good answers, score the current system, then score each change. Track answer correctness, whether the right sources were retrieved, refusals when the answer is unknown, latency and cost per request. The same evaluation discipline applies to agents; see our guide to why agentic AI projects fail and the wider AI automation guide for business.

If you are planning an assistant that answers from your own documents, our team builds AI chatbots and RAG systems with permission-aware retrieval, citations and accuracy testing, and can deploy them on private models when data must stay in-house.

Sources and further reading

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• OpenAI, Model optimization guide
• Anthropic, Introducing Contextual Retrieval
• Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models
• Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs
• Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts