RAG vs Fine-Tuning: When to Use Each for Your LLM Application

RAG and fine-tuning solve different problems. RAG changes what the model knows at inference time by injecting retrieved context. Fine-tuning changes how the model behaves by updating its weights. Teams waste months choosing between them because they frame it as a binary. It is not: they are complementary tools, and most production systems use both.

What Each Approach Actually Does

Retrieval-Augmented Generation (RAG) retrieves relevant documents from a knowledge base and stuffs them into the prompt:

query ──► retriever ──► top-k documents ──► prompt ──► LLM ──► answer

• Knowledge lives outside the model, in a vector database or search index.
• No training required. Update the index and the model's answers change.
• Cost scales with tokens: every query carries the retrieved context.

Fine-tuning updates model weights on a dataset of examples:

dataset ──► training loop ──► updated weights ──► deployed model

• Knowledge is baked into the weights.
• Requires a training pipeline, GPU budget, and evaluation discipline.
• Inference cost is unchanged (same token count), but you pay training cost.

The Decision Framework

Question	RAG	Fine-Tuning
Does the answer change frequently?	Yes	No
Is the knowledge private or new?	Yes	Maybe
Do you need citations/sources?	Yes	No
Do you need a specific output format?	No	Yes
Is latency critical?	No	Yes
Do you have a labeled dataset?	No	Yes

Rule of thumb: RAG for knowledge, fine-tuning for behavior.

• Your docs, your codebase, your product data, your customer records → RAG.
• A specific tone, a fixed JSON schema, a domain's jargon and style, a task the base model does poorly → fine-tuning.

A Working RAG Pipeline

from openai import OpenAI
import chromadb

client = OpenAI()
chroma = chromadb.PersistentClient(path="./kb")
collection = chroma.get_or_create_collection("docs")

def embed(text: str) -> list[float]:
    resp = client.embeddings.create(model="text-embedding-3-small", input=text)
    return resp.data[0].embedding

def index_document(doc_id: str, text: str, metadata: dict = None):
    collection.upsert(
        ids=[doc_id],
        embeddings=[embed(text)],
        documents=[text],
        metadatas=[metadata or {}],
    )

def answer(question: str) -> str:
    results = collection.query(query_embeddings=[embed(question)], n_results=5)
    context = "\n\n".join(results["documents"][0])
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": (
                "Answer using only the provided context. "
                "If the context does not contain the answer, say so. "
                "Cite the source for each claim."
            )},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
    )
    return response.choices[0].message.content

The system prompt is doing critical work: it forces grounded answers and citations, which is what makes RAG trustworthy. Without it, the model happily hallucinates from the retrieved context.

A Working Fine-Tuning Pipeline (LoRA)

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

dataset = load_dataset("json", data_files="training_examples.jsonl")

training_args = TrainingArguments(
    output_dir="./lora-out",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    num_train_epochs=3,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
    max_seq_length=2048,
)
trainer.train()

LoRA trains a small adapter on top of frozen weights, so a single 24 GB GPU can fine-tune an 8B model. The adapter is a few hundred MB and can be swapped at serving time without redeploying the base model.

The Hybrid: RAG + Fine-Tuning

The strongest systems combine both. Fine-tune for behavior, use RAG for knowledge:

1. Fine-tune the model to follow your output format, tone, and domain conventions (e.g. "always respond as a JSON object with summary and confidence fields").
2. RAG supplies the facts at inference time, so answers stay current without retraining.
3. Evaluate the combination against your golden set, not each piece in isolation.

def hybrid_answer(question: str) -> str:
    context = retrieve(question)  # RAG layer
    return generate_with_format(context, question)  # fine-tuned behavior

Cost and Latency Comparison

Dimension	RAG	Fine-Tuning
Setup cost	Index + retriever (days)	GPU + dataset + training (weeks)
Per-query cost	Higher tokens (context in prompt)	Same as base model
Latency	+retrieval time (10-100ms)	No added latency
Update cost	Re-index documents (minutes)	Retrain (hours to days)
Freshness	Instant	Stale until retrained

RAG's hidden cost is token spend: a 2,000-token context on every query adds up at scale. Fine-tuning's hidden cost is staleness: every knowledge change requires a retraining cycle.

Common Mistakes

• Fine-tuning for facts you could retrieve. You will retrain every time the facts change.
• RAG for style. Retrieval cannot teach the model to write like your brand; that is a weights problem.
• Skipping evaluation. Both approaches need a golden set. "It looks better" is not a metric.
• Ignoring retrieval quality. Garbage retrieval produces garbage RAG. Fix the retriever before blaming the model.
• Fine-tuning on unlabeled data. Fine-tuning without ground-truth labels amplifies the model's existing errors.

Decision Checklist

• Do answers need to cite sources? → RAG
• Does the knowledge change weekly? → RAG
• Is the knowledge private and small? → RAG (or fine-tune if it must be offline)
• Do you need a strict output format? → Fine-tune
• Do you need a specific tone/domain style? → Fine-tune
• Is per-query latency critical? → Fine-tune
• Do you have a labeled dataset? → Fine-tune
• Both? → Fine-tune behavior, RAG the knowledge, evaluate together

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub