LLM Fine-Tuning: LoRA vs QLoRA vs Full Fine-Tuning

Fine-tuning adapts a pretrained model to your domain. The three dominant methods differ in how many weights they update and how much memory they need: full fine-tuning updates everything, LoRA updates small adapters, and QLoRA does LoRA on a quantized base. This guide covers the trade-offs and gives you working recipes for each.

The Memory Problem

Training memory is dominated by four things:

1. Model weights (e.g. 16 GB for a 7B model in FP16)
2. Optimizer states (AdamW keeps 2 states per parameter: ~8 bytes/param in FP32)
3. Gradients (2 bytes/param in FP16)
4. Activations (scales with batch size and sequence length)

Full fine-tuning a 7B model in FP16 needs roughly 16 GB (weights) + 56 GB (optimizer + gradients) + activations. That is why full fine-tuning of 7B+ models requires multi-GPU setups, while LoRA fits on a single consumer GPU.

Method Comparison

Method	Trainable Params	Memory (7B)	Quality	Speed	Best For
Full FT	100%	80-120 GB	Highest	Slow	Large budgets, domain shifts
LoRA	0.1-1%	20-30 GB	Near-full	Fast	Most production use cases
QLoRA	0.1-1%	8-12 GB	Near-full	Fast	Single consumer GPU, 4-bit base

Full Fine-Tuning

Every weight is updated. Maximum capacity to absorb a new domain, but expensive and prone to catastrophic forgetting: the model can lose general capabilities it had before.

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from datasets import load_dataset

model_id = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

dataset = load_dataset("json", data_files="data.jsonl")

def tokenize(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=2048,
        padding="max_length",
    )

tokenized = dataset.map(tokenize, batched=True, remove_columns=["text"])

training_args = TrainingArguments(
    output_dir="./full-ft",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=1e-5,          # full FT uses a lower LR than LoRA
    num_train_epochs=3,
    bf16=True,
    save_strategy="epoch",
    logging_steps=10,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
)
trainer.train()

Use full fine-tuning when you have a large, high-quality dataset (100k+ examples), a real GPU budget, and the domain shift is large enough that adapters cannot capture it.

LoRA: Low-Rank Adapters

LoRA freezes the base weights and inserts trainable low-rank matrices alongside them. For a weight matrix W of shape d x d, it learns A (d x r) and B (r x d) such that the update is W + BA, with r typically 8-64. The number of trainable parameters collapses from d² to 2dr.

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    r=16,                    # rank: higher = more capacity, more memory
    lora_alpha=32,           # scaling factor, commonly 2x r
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
print(f"Trainable params: {model.num_parameters(only_trainable=True):,}")

# ~4M trainable params out of 8B — a 0.05% adapter

Training is identical to full fine-tuning, but with a higher learning rate (1e-4 to 3e-4) and much smaller memory footprint. The output is a small adapter file:

model.save_pretrained("./lora-adapter")
# adapter_model.safetensors — typically 20-100 MB for an 8B model

At serving time, load the base model and merge the adapter:

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "./lora-adapter")
model = model.merge_and_unload()  # fuse adapter into weights, zero inference overhead

QLoRA: LoRA on a 4-Bit Base

QLoRA quantizes the base model to 4-bit (NF4) and trains LoRA adapters on top. The base weights stay frozen and quantized; only the adapters are trained in full precision. This is what makes 7B fine-tuning possible on a single 12 GB consumer GPU.

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_use_double_quant=True,  # saves ~0.4 bits/param
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

model = get_peft_model(model, lora_config)  # same LoRA config as above

Key QLoRA details:

• NF4 quantization: a 4-bit data type designed for weights, better than naive int4.
• Double quantization: quantizes the quantization constants themselves, saving more memory.
• Paged optimizers: offload optimizer states to CPU when GPU memory runs out.
• Gradient checkpointing: trade compute for memory by recomputing activations.

model.gradient_checkpointing_enable()
model.enable_input_require_grads()

Choosing the Rank

The rank r controls adapter capacity:

• r=8: minimal capacity, good for style transfer and small datasets.
• r=16-32: the sweet spot for most domain adaptation.
• r=64+: large capacity; only helps with big datasets, and risks overfitting small ones.

lora_alpha scales the adapter output. A common heuristic is alpha = 2 * r. If the model ignores the adapter, raise alpha; if it overfits, lower it.

Dataset Quality Beats Method

No method fixes a bad dataset. The rules that matter more than the training recipe:

1. 10x more data beats 10x more compute for most tasks.
2. Deduplicate near-identical examples; duplicates inflate loss curves and cause overfitting.
3. Balance labels: a dataset that is 95% one class trains a model that always predicts it.
4. Hold out a golden set before training starts, and never tune hyperparameters on it.
5. Include failure modes: if the base model fails on edge cases, those exact cases belong in the training data.

Evaluation Protocol

from datasets import load_dataset
from evaluate import load

golden = load_dataset("json", data_files="golden.jsonl")["train"]
bleu = load("bleu")
rouge = load("rouge")

def evaluate(model, tokenizer, examples):
    predictions = []
    for ex in examples:
        inputs = tokenizer(ex["prompt"], return_tensors="pt").to(model.device)
        out = model.generate(**inputs, max_new_tokens=256)
        predictions.append(tokenizer.decode(out[0], skip_special_tokens=True))
    return {
        "bleu": bleu.compute(predictions=predictions, references=examples["reference"]),
        "rouge": rouge.compute(predictions=predictions, references=examples["reference"]),
    }

Compare three models on the golden set: the base model, your fine-tune, and a baseline method (e.g. LoRA vs QLoRA). If the fine-tune does not beat the base model on your task metric, the dataset or recipe is wrong — do not ship it.

Implementation Checklist

• Start with LoRA; escalate to QLoRA only if GPU memory is tight
• Use full fine-tuning only for large datasets and large domain shifts
• Set alpha = 2 * r and tune from there
• Deduplicate and balance the dataset before training
• Hold out a golden set before training starts
• Evaluate against the base model, not in isolation
• Merge the adapter at serving time for zero inference overhead
• Track training loss vs eval loss to catch overfitting

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub