RAG, Prompting, Fine-tuning or Long Context
Choose between prompting, RAG, fine-tuning and long context for a scenario, using accuracy, freshness, cost, latency and maintenance.
Four ways to teach a model about your college
The Kaveri Institute wants an assistant that answers students' questions about rules, fees and hostel life. An ordinary language model cannot do this well, because it has never seen the institute's documents. There are four common ways to close that gap:
- Prompt engineering: write better instructions for the model.
- RAG (retrieval-augmented generation): search the documents for each question and give the model the relevant passages.
- Fine-tuning: train the model further on your own examples.
- Long context: paste all the documents into every request.
Teams often pick one out of habit or because it is fashionable, and the costly mistakes come from using the wrong one. Fine-tuning a model to "learn" a rulebook that changes every semester is one such mistake; building a whole search system for a single ten-page document is another. The easiest way to choose is to ask what each approach actually changes.
Prompt engineering changes the instructions
Prompt engineering means improving the text you send with each request: clearer instructions, a few example answers, the format you want. It needs no extra systems and takes effect immediately. It is the right first step for shaping how the model behaves: its tone, its format, what it should refuse, and what it should do when unsure.
Its limit is that it cannot give the model facts it does not have. You can tell the model to "answer using Kaveri Institute's medical leave policy", but if the policy is not in the prompt, the model can only guess what it says.
RAG changes the evidence
RAG adds a search step. For each question, it finds a few relevant passages in your documents and places them in the prompt. The model itself stays the same, but the evidence in front of it changes with every question. To update what the assistant knows, you update the documents, and the very next question uses the new version. Because each answer comes from specific passages, the assistant can show its sources.
The cost is extra moving parts. Documents have to be read, split, indexed and kept up to date, and any of those steps can go wrong. If the search misses the right passage, the answer will be wrong no matter how good the model is. RAG also struggles with questions that need the whole collection at once, such as "list every rule that changed this year", because it only reads a few passages per question.
Fine-tuning changes the model itself
Fine-tuning continues training the model on your own examples, usually hundreds to thousands of question-and-answer pairs. It is very good at teaching behaviour: always answering in a particular format, writing in a house style, or doing one narrow task really well, such as sorting support tickets into 120 categories.
It is a poor way to add facts. Studies comparing the two have generally found retrieval more reliable for adding new knowledge, and some found that fine-tuning on unfamiliar facts can even make a model hallucinate more. A fine-tuned model also cannot say where a fact came from, and every time the facts change you need new training data and another training run. A simple way to remember it: fine-tuning changes how a model behaves much more reliably than what it knows.
Long context changes how much the model reads
Models can now read very long inputs. Several current models accept a million tokens or more in one request, roughly several thick textbooks. (A token is a small chunk of text, on average about three-quarters of an English word.) For a small set of documents, the simplest design is to skip searching and paste everything into the prompt. There is no index to maintain and nothing for a search to miss.
The catch is cost and speed, because you pay for every token on every question. The Kaveri Institute's 300 pages of rules come to roughly 150,000 tokens. If 10,000 students ask a question each day, sending everything every time adds up to about 1.5 billion tokens a day. A RAG system that sends only five relevant passages plus the question, about 2,000 tokens, uses about 20 million tokens: around 75 times less. Very long prompts are also slower, and models tend to use information at the start and end of a long input more reliably than information buried in the middle, a pattern researchers call "lost in the middle".
The arithmetic is worth running yourself. The price below is only an example; put in any figure you like.
Expected output:
Long context 1,500 million tokens a day, about Rs 375,000
RAG 20 million tokens a day, about Rs 5,000
Each pass of the loop multiplies the tokens sent per question by the number of questions, then prices the total. Whatever price you enter, the two rows stay 75 times apart, because the gap depends only on how much text is sent with each question.
Long context sends the whole collection with every question. RAG sends only what each question needs.
Comparing the four side by side
| Prompt engineering | RAG | Fine-tuning | Long context | |
|---|---|---|---|---|
| What it changes | The instructions | The evidence for each question | The model | How much the model reads |
| Can add private or new facts? | No | Yes | Not reliably | Yes, if it all fits |
| How you update knowledge | Not applicable | Edit documents | Retrain | Edit documents |
| Shows its sources? | No | Yes | No | Possible |
| Effort to set up | Minutes | Days to weeks | Weeks, plus example data | Minutes |
| Biggest risk | Model guesses | Search misses the passage | Old or mixed-up facts | Cost and speed |
These approaches are not rivals; they stack. Nearly every real system uses prompt engineering. Many use RAG for the facts and prompting for the behaviour. Some add a fine-tuned model on top, for example to make a small, cheap model follow a strict answer format while RAG supplies the content. Long context and RAG combine well too: search generously, then let a long-context model read twenty passages instead of three.
A simple way to decide
Ask these questions in order:
- Does the model need knowledge it does not have? If not, start with prompt engineering.
- If it does, will all of that knowledge fit easily in a prompt, and are there few enough questions to afford sending it every time? If so, long context is the simplest choice.
- If the material is large, changes often or answers must show sources, use RAG.
- If the model still does not behave the way you need after good instructions and examples, consider fine-tuning for that behaviour.
A first pass at the decision. Add fine-tuning only if the behaviour still falls short after good prompting.
Now try the decision yourself. Each scenario has one approach that makes the most sense to build first; the feedback explains why.
Whichever approach you start with, treat the choice as a first guess. Build the simplest version, test it on real questions, and let the mistakes you see tell you whether to add search, a longer context or fine-tuning.