Should I use RAG or fine-tuning for my documents?
For an assistant that must answer from a changing document collection, start by testing retrieval-augmented generation, or RAG. It finds relevant material and supplies it with the question. Fine-tuning trains a model on examples to change its behavior. Supplying a file in a chat does not, by itself, mean that you have fine-tuned the model.
Our practical starting rule is: use a small set of supplied documents for a small task, test retrieval when selecting the right evidence becomes difficult, and consider fine-tuning for a repeatable behavior problem that remains after clearer instructions and examples. These techniques can be combined; they are not competing subscriptions that every project must buy.
This guide helps you choose and test a document-answering workflow. It does not require uploading private files, training a model or paying for a database. The practice documents below are fictional.
What changes in each approach?
Prompt context means including the necessary instructions and material in the model's current input. If three short public documents fit comfortably and you can check the answer, this is a useful baseline. Adding every available file is not automatically better: the assistant still needs the relevant version and passage.
RAG adds a retrieval step. A system may split documents into passages, index them, search for relevant passages, and pass selected results to a model. Keyword search can help with exact identifiers; semantic search uses representations called embeddings to find related meaning. Anthropic describes combining these approaches and adding document context so an isolated passage remains understandable.
Supervised fine-tuning uses input-and-output examples to adjust model weights for a defined task. Google describes uses such as classification and summarization. It requires suitable data and evaluation; a folder of PDFs is not automatically a ready-made training dataset. Fine-tuning a response format also does not create a live connection to tomorrow's policy changes.
Choose by the failure you need to fix
For a fictional repair shop, consider three jobs. Summarizing one supplied manual needs a clear prompt and that manual. Answering warranty questions across changing manuals needs a way to retrieve the correct product and revision. Consistently classifying thousands of repair notes into agreed categories might justify testing fine-tuning after a simpler baseline.
If an answer uses the wrong warranty revision, investigate document selection before training the model. If the correct evidence is present but the answer adds an unsupported promise, investigate instructions and answer verification. If a narrow output format repeatedly fails across a representative test set, collect reviewed examples and compare a tuned model against the baseline.
This is a troubleshooting method, not a universal claim that one architecture wins. Write the actual failure beside the proposed change. A retrieval problem, a permissions problem and a formatting problem should not all become a request for a more expensive model.
How do I prepare documents for retrieval?
Keep a source identifier, title, revision or effective date, and location for each passage. Preserve the heading or nearby context needed to interpret it. A fragment saying “covers 90 days” is ambiguous without knowing which product, which service and which policy revision it belongs to.
Use a small, inspectable collection first. Check whether headings, lists and tables survive extraction, then ask questions whose supporting passages you already know. For a table, verify that row labels still match their values. Define how a corrected document replaces an old version and how a deleted document stops appearing in results.
Keep the source passage available with the answer so a reader can inspect the evidence. Grounding connects an answer to sources, but you still have to check whether those sources support the conclusion. A citation to the right file can point to the wrong paragraph.
Five-minute practice: build a three-document test
Create three fictional source cards. Card A, Public warranty v1, applies before October 1: repairs have a 30-day workmanship warranty. Card B, Public warranty v2, applies from October 1: repairs have a 90-day workmanship warranty; accidental damage is excluded. Card C, Staff note, is restricted to staff and says internal training starts at 09:00. These are invented terms, not advice about a real warranty.
Test 1: a public visitor asks about workmanship coverage for an October 10 repair. The expected evidence is Card B and the answer is 90 days. Test 2: the visitor asks whether accidental damage is covered. The answer is no under Card B. Test 3: they ask how much a screen repair costs. None of the cards establishes a price, so the answer must identify the missing information.
Test 4: the visitor asks when internal training starts. Card C must not be retrieved for that visitor. The app should decline to disclose staff-only information without quoting or summarizing the note. A helpful-sounding answer of 09:00 would be a permissions failure even though it matches a document.
First do the exercise manually by selecting the eligible card before writing an answer. Then, if you have a test tool, use these same invented cards and compare its behavior. This shows what you need the system to do; it does not demonstrate that a production implementation already does it.
Measure retrieval and answers separately
Microsoft distinguishes checking retrieved material from checking the generated response. Apply that distinction to your own test log: record the question, allowed source set, passages returned, answer, citation and verdict. A relevant passage missing from the results points to a different repair than a correct passage being misrepresented.
For an original practice score, imagine eight answerable questions. The correct passage appears in the selected results for six, so passage availability is 6 ÷ 8 = 75%. Only five final answers are both correct and supported, so the answer pass rate is 5 ÷ 8 = 62.5%. These are invented results and a simple worksheet metric, not a vendor benchmark or a full retrieval-quality score.
Score unanswerable questions and permission checks separately. An assistant that invents a price should fail the missing-information test. An assistant that leaks one restricted passage fails that access test; a high average on easy questions should not hide it. Keep a few unseen questions for evaluation instead of repeatedly improving only the examples used to write the prompt.
What about private files, cost and speed?
For a multi-user app, enforce access when retrieving documents. Do not send a restricted passage to a model and rely only on a request to keep it secret. Microsoft recommends retrieval-time access controls and treating retrieved content as untrusted input. A document that tells the assistant to ignore its rules is still source content, not permission to change the workflow.
RAG can add document processing, storage, search and model-input costs, plus retrieval latency. Fine-tuning adds dataset preparation, training and evaluation work; deployment costs depend on the provider and model. Compare the total cost per accepted answer for your actual task instead of assuming retrieval is always cheaper or training is always faster.
For a practical budget, record one-time preparation work separately from recurring query costs. Measure the same questions with the same source set. Include the time spent checking answers and updating documents. Check the provider's current retention, regional and account controls before introducing sensitive records.
When is fine-tuning worth a separate trial?
Consider a trial when the task is well defined, repeated examples have reliable desired outputs, and a baseline exposes a consistent behavior gap. Google recommends representative inputs and correct labels, with separate validation data. Poor or contradictory examples do not become a good training set merely because there are many of them.
For the repair shop, classification examples should resemble real repair notes and use a stable category definition. Keep evaluation notes separate from training examples. Compare accuracy, rejected or ambiguous inputs, format compliance, review time and operating cost. Record the model and dataset versions so a later change can be investigated.
Use the exercise to write an acceptance brief before choosing tools: answer current public warranty questions with traceable evidence; say when price information is missing; exclude staff-only sources for public users. Only then decide whether prompt context, RAG, fine-tuning or a combination is needed.
Practice prompt: Using only the supplied public source cards, answer each question and name the supporting card and passage. Respect the effective dates. If the evidence is missing or conflicting, say so. Do not infer prices or use restricted cards. Return the answer, evidence and unresolved gap. This prompt is a test aid; access restrictions must also be enforced by the application.
Sources & further reading
- Anthropic: contextual retrieval and the limits of isolated chunks
- Google Cloud: supervised fine-tuning changes model behavior
- Google Cloud: prepare representative training and validation data
- Microsoft: RAG, access control, cost and latency
- Microsoft: evaluate retrieval separately from the answer
- Google Cloud: grounding responses with retrieved evidence
Educational content. Examples are illustrative. Protocols, products and rules can change; consult the linked documentation for current details.