RAG, fine-tuning or long context? How to give an AI model your own data
Three ways to make a model know about your documents, products or codebase, what each is good at, and a simple way to choose.
Out of the box, an AI model knows what was in its training data, and nothing about your company’s policies, your product catalogue or last week’s meeting notes. There are three main ways to change that. They’re often confused, and they solve different problems.
Option 1: Long context (paste it in)
Modern models can read a lot of text in a single request, often hundreds of thousands of words. The simplest approach is to put the relevant documents directly into the prompt, then ask your question.
Good for:
- One-off questions about a specific document or small set of documents
- Prototyping, before you know what you need
- Tasks that need the model to see everything at once, like reviewing a whole contract
Watch out for:
- Cost and speed. You pay for every token you send, every time. Prompt caching, which many providers now offer, helps a lot when you reuse the same documents.
- Limits. Even huge context windows can’t hold a large company’s entire knowledge base.
- Attention. Models can miss details buried in the middle of long inputs. Test with your real data.
Option 2: RAG, look it up first
Retrieval-augmented generation (RAG) adds a search step. Your documents are split into chunks and indexed. When a question comes in, the system retrieves the few most relevant chunks and gives only those to the model, along with the question.
Good for:
- Large and frequently changing knowledge bases: help centres, internal wikis, product catalogues
- Answers that should cite their sources
- Keeping costs predictable, since each request only includes a few chunks
Watch out for:
- Retrieval quality is everything. If search doesn’t find the right chunk, the model can’t use it. Most RAG problems are search problems.
- Chunking choices matter. Split documents badly and you separate a question from its answer.
- Access control. Make sure users can only retrieve documents they’re allowed to see.
Option 3: Fine-tuning, change the model itself
Fine-tuning means training an existing model further on your own examples, adjusting its internal weights.
Good for:
- Teaching a consistent style, format or tone
- Specialised tasks with lots of examples, like classifying support tickets into your own categories
- Getting a smaller, cheaper model to perform a narrow task as well as a larger one
Watch out for:
- It’s poor at teaching facts. Fine-tuning shapes behaviour more reliably than it stores knowledge. For facts that change, use retrieval.
- It needs good data. Hundreds or thousands of high-quality examples.
- It goes stale. Every update to your information means another training run.
How to choose
Ask two questions:
- Is the problem what the model knows, or how it behaves? Knowledge → long context or RAG. Behaviour → fine-tuning (or better prompting first).
- How much data, and how often does it change? A few documents → long context. A large or changing corpus → RAG.
Most real systems combine them: a well-written prompt, retrieval for up-to-date facts, and fine-tuning only when there’s a clear behavioural gap that prompting can’t close.
Start simple. Put the documents in the prompt, see what goes wrong, and add complexity only to fix a real problem.
