What RAG is and when you don't need it

Every few months we get asked the same question: "so how does a model know things?" Usually the answer someone expects involves magic: a fine-tune, a bigger model, a dataset they never had. In most cases the real answer is simpler: retrieval.

What RAG is, in one paragraph

RAG (retrieval-augmented generation) is a pattern: before the model answers, you search your own documents for the parts that matter, and hand those pieces to the model as context. The model then answers from that context, with a source, in many cases. The model itself doesn't "learn" your company; it reads the relevant page, every time, right before it speaks.

The pattern was formalized in the original RAG paper, but it has become the default starting point for "AI over your documents" projects, and for good reason.

Why it's the default starting point

• Your knowledge lives in documents, not weights. Prices, policies, procedures, tickets: none of it is going into model training. Retrieval reads what you already wrote.
• It updates on its own. When the policy changes, the new document goes in and the old one goes out. No retraining, no waiting.
• It's auditable. "Here's the answer, here's the page it came from" is something your team can check. An ungrounded model answer is a hope; a retrieved one is a citation.
• It stays cheap. You're reading a few paragraphs, not shipping your whole corpus to the model.

When you don't need it

RAG is the right tool when the answer depends on facts that live in your files and change over time. It's the wrong tool when:

• The "knowledge" is really behavior: tone, judgment, a style of reasoning. That's what fine-tuning or plain prompting is for.
• The document set is small enough to fit in the context window. If you can put it all in front of the model, you don't need a retrieval step.
• You haven't cleaned the documents first. Retrieval finds the most similar text, not the most correct text. Garbage in the index, garbage in the answer.

Our rule of thumb

Start with the simplest thing that can answer the question: the document in the prompt. Add retrieval when the corpus stops fitting, or when the answer needs a source. Add fine-tuning only when the model's behavior, not its knowledge, is the problem.

Most "AI assistant" projects we talk about need the first two. The third comes rarer than the hype suggests.