Why fine-tune ? Let me provide a real, personal example.

We fine-tuned a 7B parameter model which completely blows foundation models out of the water, but just for this very narrow subtask: Filling out synoptic reporting templates for breast cancer.

This is a hellishly difficult task with complex input formats, branching logic, and fields that must appear in a strict order.

The LLM needs to discern which of 40 different histologic subtypes trigger which branch of field subsets, perfectly, without hallucination. It’s near impossible to define in a conditional table. One wrong field, and the entire output is invalidated.

When we used aggressive system prompts along with some light RAG, our accuracy (with Claude Opus 4.6) was * ~35%.* We had to include the entire body of the template in the context, along with a detailed guide when/where to use which field. Roughly 30k tokens, per call.

The result? Omitted fields here, unnecessary subsections there, hallucinations, etc. which means that a human would need to manually read and edit the entire document. A no-go.

After fine-tuning a Mistral 7B model (with QLoRA), our accuracy jumped to ~98%. I was stunned on how effective it actually was.

Prompt + RAG ███████░░░░░░░░░░░░ 35% QLoRA ███████████████████░ 98%
We got an improvement of 63 percentage points and completely eliminated our API costs (for this task). Our initial cost estimates for running this at the scale we needed (with the frontier model) would have been ~$320,000. We got there for free.*

** Not including the price for fine-tuning, running a local model (which we already do at scale), or measuring the energy usage per API call

That’s why you fine-tune.

Despite the common belief, RAG + System prompts will not solve fine-tuning problems, and they’re everywhere.

In this article, I’ll cover:

  • When to fine-tune(the RAG vs Fine-Tune debate)
  • The mathematical intuition behind LoRA/QLoRA
  • The technical implementation- details
  • Evaluating using a custom harness

After reading, you’ll know when to fine-tune, why it works, and exactly how to implement it in practice

When to fine-tune

❓❓Do I actually need to fine-tune?❓❓

Perhaps. Look for one of the following fine-tuning patterns:

Rigid, Highly Specific Formatting Requirements

You need the LLM to output specific formats which are very complex and unforgiving to the occasional hallucination, like a missed or added field. Some notable examples:

  • Legacy Enterprise Documents:Large companies often have deeply ingrained, idiosyncratic templates with countless conditional branches.
  • Court/Legal Documents: where each jurisdiction has its own format and template. These forms obviously weren’t part of the LLMs input data, and need to be introduced as new knowledge.
  • Medical Forms: They’re complex, often contain redundant information and need to be just right.

Cost constraints

Thousands (or tens of thousands) of tokens in a system prompt which runs on every single API call for every customer inquiry. At scale, that’s real money and latency. A fine-tuned model that has internalized those patterns needs neither.

Complex Instructions and Combinatorial Explosions

System prompts work with simple constraints, but they often break down when rules overlap.

If your task involves a massive decision tree (e.g., “If A, do B, but if C and A, do D, unless E is present…”), you may hit the limits of in-context learning. In our case, our combinatorial space exploded with rules that couldn’t reasonably be encoded into a table.

Also, remember, context degrades with length. A system prompt with 50 different rules is likely to omit one here or there, invalidating the entire output.

Custom Tone

This isn’t relevant to us, but worth a mention. If you require a specific “brand voice” for your customer service agent, fine-tuning often works better than using system prompts. Also relevant: you need to add a system prompt to every single customer interaction to maintain a specific voice or tone. If your “brand voice” prompt is 2,000 tokens, the costs can add up quickly.

When to use RAG

A rule of thumb most practitioners use: RAG mostly augments the model’s knowledge. Fine-tuning mostly affects the output behavior.

I say mostly, because full fine-tuning can definitely add new knowledge to an LLM, and RAG can (and is often used to) modify the default behavior of an LLM. It’s not black and white, so use your best judgement.

Where RAG might be a better choice:

  • Your knowledge base changes frequently
  • You only need to augment the behavior rarely
  • The model needs access to documents, policies, or facts that evolve over time
  • You only have a few hundred high quality training examples
  • You can reliably modify the behavior with a small system prompt

Working solutions tend to end up as a mixture of both. We didn’t completely eliminate system prompts or RAG, we just greatly reduced our reliance on them.

Now, let’s dive into the math so we can understand the mechanics of fine-tuning

The mathematical intuition behind LoRA/QLoRA

Before covering any of the mathematical detail of LoRA (Low Rank Adaptation), and its derivatives, conceptually grouped together as “Parameter Efficient Fine Tuning (PEFT)”, we need to understand what fine-tuning actually is doing under the hood.

🤔 Why take time to understand the math behind LoRA/QLoRA

Understanding the math behind LoRA is critical to

understand ifits the right strategy for the task at hand.It’s the dividing factor between people who truly understand why/when to fine-tune vs why/when to RAG. I’d recommend not copy-pasting the training script and using the defaults provided, which might work for you right out of the box.

Instead, develop a mathematical intuition for what’s happening here. That way, debugging becomes less guess work and more precision engineering.

For example: Misunderstanding what “rank” is in LoRA will make it difficult to diagnose overfitting problems. E.g. Why chose

rank8 overrank32? What does lowering thealphaparameter do to the residual stream?You won’t have to derive any formulae from scratch. I’ve described it in a way which is (hopefully) accessible if you have any understanding for how large transformer based models work.

Fine-tuning is a continuation of the large pretraining task that LLMs undergo.

In pretraining, the objective is typically predict the next token in a sequence of natural language.

In supervised fine-tuning (SFT), the same causal next-token prediction objective is applied to specific examples: <code>(prompt, completion) pairs. In many SFT setups, the loss is computed exclusively on the assistant/completion tokens rather than on the user’s prompt.

The mathematical objective is approximately the same: Cross Entropy (Negative Log Likelihood) over tokens.

The difference is the dataset, . Pre-training uses trillions of tokens of raw web text, while fine-tuning uses curated(prompt, completion)pairs.

Typically misunderstood or poorly understood: You can absolutely perform fine-tuning to add knowledge to an LLM without formatting prompt, completion pairs. Simply feed in new data with this same mathematical objective.

❓❓Why LoRA/QLoRA and any of its other derivatives then?❓❓

Full fine-tuning is expensive. Really expensive.

A 70B parameter model in FP16 requires storing the model’s weights in VRAM + 2 additional moments for each parameter in your Adam optimizer. Full fine-tuning requires upwards of 1.2 TB to 1.4 TB of VRAM, which is mostly out of reach for mere mortals

You can destroy existing knowledge in the LLM.

Because all the parameters are eligible for updating, we risk overwriting some of the broader reasoning and language capabilities of the underlying model.

Which brings us to the why of LoRA.

LoRA (Low Rank Adaptation)

In “LoRA: Low-Rank Adaptation of Large Language Models“, Hu et. al created a practical way of performing what every frontier AI lab was trying to do at the time: train a single foundational model, then, efficiently specialize to many downstream tasks.

The idea is simple, and elegant.

A neural network has many dense layers. E.g. a linear layer performs the operation:

Where (W) is the learned weight matrix.

For example, a transformer projection layer might contain a weight matrix:

When we update via full fine-tuning, the weight adjustment is the original weights + the update: , where . The update (adaptation matrix) is the

same dimension as the original weight matrix.

The core intuition of LoRA

In LLMs, it’s been shown that updates (while fine-tuning) have a “lower intrinsic dimension”1, meaning that the number of effective degrees of freedom needed to find a good solution is much smaller than the total number of model parameters.

Said otherwise, we don’t need to update all the parameters of the original model to steer the model slightly. We only need to update a lower dimensional projection, which achieves the same performance.

This lends itself to the core concept behind LoRA.

In LoRA, we leave the original model weights frozen. The forward pass is augmented with a low rank projection.

We learn low rank matrices (, which are added to the forward pass of the original model for each weight we want to update.

Thus, the forward pass becomes:

Side note, when you use Hugging Face or PEFT, (from the paper) is replaced by the hyperparameter , so these formulae are equivalent:

Conceptually, we learn a projection of the input into a low rank subspace which augments the forward pass, similar to a residual stream correction. is used to control the strength of this augmentation.

Again, the important thing to understand is that the original model remains completely unchanged. The only thing we’re learning here is a much smaller sidecar that perturbs the internal activations within the original model, such that we get desired behavior.

And it works!

**What this gives us is **

  • The ability change the behavior of the original model with far fewer parameters
  • The ability to learn multiple adapters, perhaps for different tasks, using the exact same base model

🤚 Great. But how does this solve the full fine-tune problem?

In full fine tuning, for our original weight matrix , we need to learn 16M different parameters, along with 33M moments for our optimizer. This means bigger hardware.

With LoRA (with rank 8 for example): we only need to learn 65k parameters, along with 131K moments for our optimizer (for this single weight matrix from our original).

What this means for practicioners

LoRA can fine-tune an LLM with huge reductions in VRAM requirements, since we’re only learning small rank matrices. We don’t need to keep the optimizer state for every parameter in the full size model, just the optimizer state for the small rank matrices.

Let’s make this even better by fitting it all on one consumer grade GPU.

QLoRA (Quantized Low Rank Adaptation)

With LoRA, we still have the problem of storing the entire model’s base weights in FP16 in order to learn the LoRA adapters for subtask learning.

QLoRA closes that gap by quantizing the frozen base model down to 4 bits, while keeping the LoRA adapters themselves in 16-bit precision.

In “QLoRA: Efficient Finetuning of Quantized LLMs“, the authors include a few different mechanisms to make this feasible. Surprisingly, they also released functional code along side the paper which has been heavily tested by the community and can be trusted to build your own QLoRA fine-tuning pipeline.

4-bit NormalFloat (NF4)

Introduced in the paper as an “information theoretically optimal” quantization strategy that outperforms 4bit floats.

This quantization works in the following method.

4 bits gives us 16 possible values to work with. Our entire distribution of weights needs to be mapped to one of these 16 values.

Naively, if we chose equal width bins for weights between -1 and 1, and map each possible bit combination to a value, like so:

bit -> value 0000 -> -1.0 0001 -> -0.875 0010 -> -0.75 ...
We can indeed take each one of a network’s possible FP16 weight values and map it to a 4bit code.

However, * this wastes capacity*, as only a small percentage of the weights occupy the

0000 bit interval (the tail) and are more concentrated around zero.NF4 uses the expected distribution of neural networks weights to construct a more representative set of 16 quantization values. Since normalized weights are typically approximately Gaussian and centered around zero, the quantized values are placed more densely around zero and more sparsely in the tails.

Thus, our naive bins (from the example above) become smarter and more representative of the actual weights. E.g.

bit -> value 0000 -> -1.0000 0001 -> -0.6962 0010 -> -0.5251 ...
We gain precision, as the actual weights can be reconstructed with less quantization error.

Double Quantization

4bit quantizing works, but each layer may have a different distribution. If we created a global scaling factor, we’d lose significant precision. Thus, we need to divide all the weights into blocks (typically 64).

Each block of weights has its own unique scaling factor, a 32bit floating point value that’s used to map the 4bit codes back to their full precision counterpart.

What double quantization achieves is a quantization of the unique scaling factors, so that we don’t have to keep all of the full precision 32bit floats in VRAM.

The first step is quantizing the weights into NF4:

  • Divide all model weights into blocks of 64
  • Compute one FP32 scaling constant per block
  • Quantize weights to 4bit NF4 using the process above

Second level (the “double” part):

  • Group 256 of those FP32 scaling constants together
  • Quantize them down to 8bit floats (FP8)
  • Store just one FP32 scaling constant per group of 256

Per the paper, you can save roughly 0.373 bits per parameter. It’s small, but across billions of parameters the VRAM savings are worth it (potentially 2-3GB of VRAM)

Paged Optimizers

Paged Optimizers are a nice NVIDIA unified memory hack. For long context sequences, occasionally, VRAM requirements spike, causing dreaded OOM states.

Ultimately, paged optimizers are primarily a mechanism for handling memory spikes. They allow some of the optimizer states to reside outside GPU memory when GPU capacity becomes constrained.

Otherwise, the LoRA implementation still holds.

Instead of running forward/backward passes on a frozen, FP16 model, we can run our these passes on a highly quantized model, while maintaining LoRA adapters in full precision.

The result is a set of clever memory management techniques that makes fine tuning very large models much more memory efficient.

TLDR: LoRA makes fine tuning memory efficient. QLoRA makes LoRA more memory efficient.It’s an optimization of an optimization that retains performance with much less horsepower.

Why QLoRA instead of full fine-tuning?

Our model had 7B parameters. Full fine-tuning was possible, but it offered little benefit for this task relative to the additional memory and compute requirements.

LoRA gave us another advantage: we could experiment with the adapter independently of the base model.

That made QLoRA the obvious starting point.

| Approach | Why we rejected/selected it |
|---|---|
| Prompting | Too inconsistent |
| RAG | Didn’t solve behavioral consistency |
| Full fine-tuning | Unnecessary memory/compute |
| LoRA | Good parameter efficiency |
| QLoRA | Same LoRA approach with substantially lower base-model memory |

For us, QLoRA made financial sense. We don’t have an endless waterfall of compute resources, so we need to be smart.

We’d spend less in compute and inference for a result which was (potentially) as accurate as a full fine-tune. If successful, we could build different adapters for different subtasks, allowing for even further specialization. We’re working on this now 😎

The technical implementation details

Where the rubber meets the road.

If you’re anything like us, you will spend:

  • 70%of your time on generating high quality training data
  • 20%of your time running evaluations
  • and only 10%of your time actually running the fine-tune

In fact, the actual fine tuning script is lightweight and very easy to run and understand.

😱

You may be intimidated about running your own training pipeline for your fine tune.

😎 Don’t be.You’ll waste time and effort trying to customize some templated fine-tuning pipeline from X number of companies offering it. The script I offer below can be easily customized to your requirements.

Unfortunately, I don’t know what your exact use case is. However, I can outline the requirements for our use case in the hope that you can form analogies where it makes sense.

Either way, the data is the most painful part of this process.

Generating high quality training data

Case 1: You have 4,000–10,000+ real input/output pairs.

This is an ideal situation, as the fine-tuning process really shines when you want to steer an LLM to answer in a very specific way. Customer service companies will likely live in this space.

Case 2: Our situation. You have outputs, but no inputs.

This is more difficult. How do you steer an LLM when there is nothing to steer with? It’s akin to trying to build a classifier with just targets, no features.

Specifically, for our use case, we just had the raw corpus of finalized, structured synoptic reports. We needed the free hand notes the pathologist compiled before creating the structured report.

We thought about running a full fine tune, hoping that the LLM would retain the formatting, structure and highly nuanced requirements of a finalized report. Ultimately, we decided against it. We needed a very specific solution: take an unstructured pathologist’s note and turn it into a structured synoptic report.

Luckily, we found a solution.

In “Self-Alignment with Instruction Backtranslation“, Li et al. reverse the problem.

Instead of “given this input, generate the most likely output”, the problem is now “given this output, generate the most likely input.”

We can use a language model to generate a plausible, synthetic input. Then, curate the pairs with the highest quality examples, then run a fine-tune on the curated dataset.

Specifically, what we did was:

  • Gathered 10,000 real synoptic reports (we have 200,000)
  • Used a frontier model to generate (4) real, unstructured pathology narratives for each output (using 4 different system prompts)
  • Curated the list of output pairs based on our initial error evaluations. Specifically, we wanted data that our initial foundation model trial failed on.
  • Used (synthetic input, real output)as our training dataset.

Why 4 different inputs per each output?

We generated multiple candidate narratives so our model would learn diversity, that is, so that fine-tuned model can learn that the exact same output is reachable from a diverse set of inputs.

We used different highly restrictive system prompts that match what clinician’s notes actually look like, along with restrictions on using field headers, so that the model learned how to map, not just how to reiterate facts from the note.

All we then needed to do was format it for our training pipeline, each example was formatted as so:

{ "prompt_id": "xyz123", "prompt": "Received fresh, labeled with patient identifiers, a 3.2 x 2.8 x 2.1 cm...", "completion": "..." }
Our final corpus consisted of ~8,000 input, output pairs with overlap.

If you’re generating data in this method, I highly recommend generating multiple candidate inputs for each output. You want the LLM to conceptualize the fine-tuning data, so that minor variations in the input prompts don’t lead to hallucinations.

The training pipeline

Again, as I mentioned, running your own fine-tune is actually quite accessible. Paired with a thorough understanding of what’s going on underneath the hood (please read the math section), you will be well equipped to diagnose any sort of training errors.

The training process is:

  • Perform a hyperparameter searchover relevant hyperparameters. Every hyperparameter isn’t a first class citizen, some LoRA hyperparameters have more influence than others
  • Once a candidate set is found, run the fine-tune.
  • The most important step, evaluate the outputs. Here, we’re not only concerned with the loss values, but also the quality and correctness of the output solutions.

I provide the full script below, as well as a walk through some notable sections of the script, explaining what is happening and why.

Hardware

The goal with this article was to convey the reasonableness of training on a single, consumer size GPU. Thus, we want to use QLoRA and limit our VRAM requirements to <24GB.

The following implementation achieves that.

If using AWS, an instance like the g6.xlarge or g6.2xlarge is recommended

If using GCP, an instance like the g2-standard-4 or g2-standard-8 is recommended.

This makes the fine-tune cost effective, even if it takes >24 hours. For a safe margin, budget $50-100 in compute costs for a single QLoRA fine-tune.

If you want to go ultra-budget, you could try an even smaller model and run this entirely on a T4, which makes Google Colab an option.

Hyperparameters

There are many hyperparameters to chose from. I selected reasonable defaults based on two notable sources, which are actually quite accurate.

Source 1: Unsloth | LoRA Hyperparameters Guide

This is a well researched and thorough guide to the hyperparameters which count for fine-tuning. We didn’t use Unsloth’s quantized models for our fine-tune, but it’s a reasonable option.

Source 2: Thinking Machines Lab | LoRA without Regret

Again another thoughtful review of the hyperparameters which matter in LoRA. Schulman exposes some “parameter invariances” which reduce the number of hyperparameters which are actually relevant.

The big takeaway

In the original LoRA paper, the authors only applied LoRA to the attention matrices of the LLM. However, Schulman shows that applying LoRA to all the weight matrices results in better performance.

“Even in small data settings, LoRA performs better when applied to all weight matrices, especially MLP and MoE layers. Attention-only LoRA underperforms even when we match the number of trainable parameters by using higher rank for attention-only LoRA.”

In the provided script, I don’t include automated hyperparameter searches. However, I’ll highlight the most relevant hyperparameters along with sensible defaults, cherry picked from the two sources above. If a sweep is recommended, the sweep values are in a list. Otherwise, just use the default value.

| Hyperparameter | What It Controls | Recommended Value(s) |
|---|---|---|
| Learning Rate | Step size for adapter weight updates | Default: 2e-4Sweep:[5e-5, 1e-4, 2e-4, 4e-4, 8e-4]. |
| LoRA Rank (r) | Adapter capacity / trainable-parameter count. | Default: 16Sweep:[8, 16, 32, 64, 128]. |
| LoRA Alpha | Scales adapter output by α/r; interacts with LR rather than acting independently. | Default: 2r |
| Effective Batch Size(batch_size × grad_accum_steps) | Trade-off between gradient stability and VRAM/training time. | Default: 16 |
| Epochs | Number of passes over the training set. | Default: 1–3 epochs |
| LoRA Dropout | Regularization on adapter activations. | Default: 0Sweep (only if overfitting):[0, 0.05, 0.1]. |
| Weight Decay | Penalty on weight magnitude. | Default: 0.01 |
| LR Scheduler / Warmup | Shape of the LR curve over training. | Default: a linear or cosine scheduler with warmup over the first 5–10% of total steps. |

This gives us reasonable defaults, selected based on empirical research. Now, let’s wire everything up in a PyTorch script.

Step 1: Install prerequisites

pip install torch transformers peft trl bitsandbytes datasets accelerate### Step 2: Run the script

⚠️

This is avery trimmed down, LLM edited, versionof our training job for QLoRA. You’ll probably need to update it, especially if you’re going to do hyperparameter sweeps.I promise, I reviewed it for slop.

Expand this block for the full script

```
"""
QLoRA fine-tuning for Mistral-7B-Instruct.

Hyperparameters and defaults:
Learning Rate 2e-4 Step size for adapter weight updates.
LoRA Rank (r) 16 Adapter capacity / trainable-parameter count.
LoRA Alpha 2*r Scales adapter output by alpha/r.
Effective Batch Size 16 batch_size * grad_accum_steps.
Epochs 3 Number of passes over the training set.
LoRA Dropout 0 Regularization on adapter activations.
Weight Decay 0.01 Penalty on weight magnitude.
LR Scheduler cosine Linear or cosine, with 5-10% warmup.

Usage
python train_qlora.py --rank 32 --lr 1e-4 --epochs 2
python train_qlora.py --rank 32 --lr 1e-4 --epochs 2 --merge
"""

import argparse
import os

import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
EarlyStoppingCallback,
TrainerCallback,
)
from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig, DataCollatorForCompletionOnlyLM
from datasets import load_dataset

BASE_MODEL = "mistralai/Mistral-7B-Instruct-v0.3"
DATA_FILE = # ADD YOUR JSON DATAFILE!
RESPONSE_TEMPLATE = "[/INST]" # everything after this is the assistant turn
OUTPUT_ROOT = "./mistral-lora"

SYSTEM_PROMPT = (
# ADD A SYSTEM PROMPT HERE
)

class PrintLossCallback(TrainerCallback):
def on_log(self, args, state, control, logs=None, **kwargs):
if logs:
if "loss" in logs:
print(f"Step {state.global_step} | Train Loss: {logs['loss']:.4f}")
if "eval_loss" in logs:
print(f"Step {state.global_step} | Eval Loss: {logs['eval_loss']:.4f}")

def build_dataset(tokenizer, data_file=DATA_FILE, eval_fraction=0.1, seed=42):
dataset = load_dataset("json", data_files=data_file, split="train")

def to_chat_text(example):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": example["prompt"]},
        {"role": "assistant", "content": example["completion"]},
    ]
    return {"text": tokenizer.apply_chat_template(messages, tokenize=False)}

dataset = dataset.map(to_chat_text)

split = dataset.train_test_split(test_size=eval_fraction, seed=seed)
return split["train"], split["test"]

def load_base_model():
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
quantization_config=bnb_config,
device_map="auto",
)
return prepare_model_for_kbit_training(model)

def build_lora_model(rank, alpha, dropout):
model = load_base_model()
lora_config = LoraConfig(
r=rank,
lora_alpha=alpha,
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_dropout=dropout,
bias="none",
task_type="CAUSAL_LM",
)
return get_peft_model(model, lora_config)

def train(hp, train_ds, eval_ds, tokenizer, collator, output_dir):
model = build_lora_model(
rank=hp["rank"],
alpha=hp["lora_alpha"],
dropout=hp["lora_dropout"],
)
model.print_trainable_parameters()

sft_config = SFTConfig(
    output_dir=output_dir,
    per_device_train_batch_size=hp["per_device_train_batch_size"],
    gradient_accumulation_steps=hp["gradient_accumulation_steps"],
    num_train_epochs=hp["epochs"],
    learning_rate=hp["learning_rate"],
    lr_scheduler_type=hp["lr_scheduler_type"],
    warmup_ratio=hp["warmup_ratio"],
    weight_decay=hp["weight_decay"],
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    max_seq_length=32768,
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=train_ds,
    eval_dataset=eval_ds,
    data_collator=collator,
    callbacks=[
        EarlyStoppingCallback(early_stopping_patience=3),
        PrintLossCallback(),
    ],
)
trainer.train()
metrics = trainer.evaluate()

return trainer, metrics

def save_adapter(trainer, tokenizer, output_dir):
"""Save just the LoRA adapter (a few hundred MB) plus the tokenizer."""
trainer.save_model(output_dir)
tokenizer.save_pretrained(output_dir)
print(f"Adapter saved to {output_dir}")

def merge_and_save(base_model_name, adapter_dir, merged_dir):
"""Fold the LoRA adapter back into the base weights for standalone serving.

Reloads the base model at full bf16 precision (not the 4-bit
quantized copy used during training) so the merge itself doesn't
compound quantization error, then writes out a standard dense
model you can load with AutoModelForCausalLM like any other
checkpoint -- no PEFT dependency needed at inference time.
"""
print(f"Loading base model for merge: {base_model_name}")
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
merged_model = PeftModel.from_pretrained(base_model, adapter_dir)
merged_model = merged_model.merge_and_unload()

merged_model.save_pretrained(merged_dir, safe_serialization=True)
tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
tokenizer.save_pretrained(merged_dir)
print(f"Merged model saved to {merged_dir}")

def main():
parser = argparse.ArgumentParser()
parser.add_argument("--data_file", default=DATA_FILE)
parser.add_argument("--eval_fraction", type=float, default=0.1)
parser.add_argument("--lr", type=float, default=2e-4)
parser.add_argument("--rank", type=int, default=16)
parser.add_argument("--lora_alpha", type=int, default=None)
parser.add_argument("--batch_size", type=int, default=4)
parser.add_argument("--grad_accum", type=int, default=4)
parser.add_argument("--epochs", type=int, default=3)
parser.add_argument("--dropout", type=float, default=0.0)
parser.add_argument("--weight_decay", type=float, default=0.01)
parser.add_argument("--scheduler", choices=["linear", "cosine"], default="cosine")
parser.add_argument("--warmup_ratio", type=float, default=0.05)
parser.add_argument(
"--merge", action="store_true",
help="Merge the LoRA adapter into the base model weights after training.",
)
parser.add_argument(
"--merged_dir", default=None,
help=f"Output directory for the merged model (default: {OUTPUT_ROOT}/merged). Only used with --merge.",
)
args = parser.parse_args()

os.makedirs(OUTPUT_ROOT, exist_ok=True)

lora_alpha = args.lora_alpha if args.lora_alpha is not None else 2 * args.rank

hp = {
    "learning_rate": args.lr,
    "rank": args.rank,
    "lora_alpha": lora_alpha,
    "per_device_train_batch_size": args.batch_size,
    "gradient_accumulation_steps": args.grad_accum,
    "epochs": args.epochs,
    "lora_dropout": args.dropout,
    "weight_decay": args.weight_decay,
    "lr_scheduler_type": args.scheduler,
    "warmup_ratio": args.warmup_ratio,
}

print("Hyperparameters:", hp)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.pad_token = tokenizer.eos_token
train_ds, eval_ds = build_dataset(
    tokenizer, data_file=args.data_file, eval_fraction=args.eval_fraction
)
collator = DataCollatorForCompletionOnlyLM(
    response_template=RESPONSE_TEMPLATE, tokenizer=tokenizer
)

trainer, metrics = train(
    hp, train_ds, eval_ds, tokenizer, collator,
    output_dir=os.path.join(OUTPUT_ROOT, "run"),
)
print("Final eval metrics:", metrics)

adapter_dir = os.path.join(OUTPUT_ROOT, "final")
save_adapter(trainer, tokenizer, adapter_dir)

if args.merge:
    merged_dir = args.merged_dir or os.path.join(OUTPUT_ROOT, "merged")
    merge_and_save(BASE_MODEL, adapter_dir, merged_dir)

if name == "main":
main()
```
Part 1: Load in your data

The script contains some placeholders for your dataset. Ensure the dataset is json, with both "prompt","completion"fields.

def build_dataset(tokenizer, data_file=DATA_FILE, eval_fraction=0.1, seed=42): dataset = load_dataset("json", data_files=data_file, split="train") def to_chat_text(example): messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": example["prompt"]}, {"role": "assistant", "content": example["completion"]}, ] return {"text": tokenizer.apply_chat_template(messages, tokenize=False)} dataset = dataset.map(to_chat_text) split = dataset.train_test_split(test_size=eval_fraction, seed=seed) return split["train"], split["test"]
Here, we simply load in the data, format it as a text string and split it into train and test. We don’t need to worry about tokenization or tensorization, as we tokenize separately and SFTTrainer also has some internal hooks that tensorize.

Part 2: Build the model

def load_base_model(): bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, ) model = AutoModelForCausalLM.from_pretrained( BASE_MODEL, quantization_config=bnb_config, device_map="auto", ) return prepare_model_for_kbit_training(model) def build_lora_model(rank, alpha, dropout): model = load_base_model() lora_config = LoraConfig( r=rank, lora_alpha=alpha, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_dropout=dropout, bias="none", task_type="CAUSAL_LM", ) return get_peft_model(model, lora_config)
From the original QLoRA paper, BitsAndBytes, handles quantization for us. Otherwise, we rely heavily on HuggingFace’s transfomers library, which is very standard in the industry for running inference and fine-tuning on LLMs.

The other thing worth noting here is that in the build_lora_model function, we target all the modules (target_modules) from the base LLM, as noted from Schulman’s research above. It does work in practice!!

Part 3: The training loop

def train(hp, train_ds, eval_ds, tokenizer, collator, output_dir): model = build_lora_model( rank=hp["rank"], alpha=hp["lora_alpha"], dropout=hp["lora_dropout"], ) model.print_trainable_parameters() sft_config = SFTConfig( output_dir=output_dir, per_device_train_batch_size=hp["per_device_train_batch_size"], gradient_accumulation_steps=hp["gradient_accumulation_steps"], num_train_epochs=hp["epochs"], learning_rate=hp["learning_rate"], lr_scheduler_type=hp["lr_scheduler_type"], warmup_ratio=hp["warmup_ratio"], weight_decay=hp["weight_decay"], bf16=True, gradient_checkpointing=True, logging_steps=10, eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=False, max_seq_length=32768, report_to="none", )
Some of the more important fields in our SFTTrainerand SFTConfig

| Group | Fields | What it does |
|---|---|---|
| Batch shape | per_device_train_batch_size,gradient_accumulation_steps | Effective batch size = the two multiplied together. You process a small batch on-GPU, accumulate gradients over several steps, then apply one optimizer update. |
| Schedule | num_train_epochs,learning_rate,lr_scheduler_type,warmup_ratio | How long to train and how the LR moves |
| Precision & memory | bf16=True,gradient_checkpointing=True | bf16runs the trainable LoRA math and gradients in bfloat16 |
| Regularization | weight_decay | Standard penalty on the adapter weights, discourages them from growing unboundedly large. |
| Logging | logging_steps=10 | How often PrintLossCallbackgets alogsdict to print from. |
| Eval/checkpointing | eval_strategy="epoch",save_strategy="epoch",load_best_model_at_end=True,metric_for_best_model="eval_loss",greater_is_better=False | Evaluate and checkpoint once per epoch. Because both strategies are set to "epoch"(they have to match forload_best_model_at_endto work), at the end of training the Trainer automatically swaps back in whichever epoch’s checkpoint had thelowesteval_loss: even if it wasn’t the last one. Without this, you’d keep whatever the final epoch produced, which could already be overfitting. |
| Sequence length | max_seq_length=32768 | Truncates the formatted chat text (system + note + report) beyond 32768 tokens. This is intentionally large for our use case because some pathology notes and structured reports are long. |

Part 4: Saving and merging the adapterOnce training finishes, you’ve got a LoRA adapter: a few hundred megabytes, not a full model. That’s great for storage (you can keep dozens of task-specific adapters against a single base model), but for production inference you usually want to merge the adapter back into the base weights, so there’s no extra matrix multiplication at serving time and you can deploy it behind a standard inference stack.

def save_adapter(trainer, tokenizer, output_dir): trainer.save_model(output_dir) tokenizer.save_pretrained(output_dir) def merge_and_save(base_model_name, adapter_dir, merged_dir): base_model = AutoModelForCausalLM.from_pretrained( base_model_name, torch_dtype=torch.bfloat16, device_map="auto", ) merged_model = PeftModel.from_pretrained(base_model, adapter_dir) merged_model = merged_model.merge_and_unload() merged_model.save_pretrained(merged_dir, safe_serialization=True) tokenizer = AutoTokenizer.from_pretrained(adapter_dir) tokenizer.save_pretrained(merged_dir)

Evaluating using a custom harness

Again, I don’t know what your exact use case is, but I will provide some of the decisions we made.

In a synoptic report template, field structure for each histologic subtype is fully determined by the template (that’s the whole reason this task is a fine-tuning candidate and not a RAG candidate). We could encode each subtype’s required field list and required order as a schema, then check a candidate document against it along four axes:

  • Recall: are all required fields for the correct branch present?
  • Precision: are there any hallucinated fields that don’t belong to that branch?
  • Order: do the present fields appear in the required strict sequence?
  • Value correctness: for categorical/numeric fields, does the value match; for free-text fields, is it semantically equivalent?

What the harness actually found

Running the harness against both systems on the same held out set is what turned “it felt better” into a number we could defend:

| Metric | Foundation model (RAG + prompt) | Fine-tuned Mistral 7B |
|---|---|---|
| Strict document accuracy | ~35% | ~98% |
| Docs with ≥1 hallucinated field | ~41% | ~2% |
| Docs with ≥1 omitted field | ~52% | ~3% |
| Docs with an order violation | ~23% | <1% |
| Mean field-level accuracy | ~71% | ~99.4% |

Lessons learned

A few things we’d tell ourselves at the start of this project:

  • Loss ≠ correctness.A plateauing- eval_losscan hide a checkpoint that still hallucinates reliably on rare subtypes. Don’t pick your final checkpoint on loss alone.
  • Oversample rare branches.Subtypes that show up rarely in your real reports get under represented in a naive split; upweight them, or the model will nail the common cases and quietly fail on the edge cases, which is exactly where a human reviewer is least likely to catch it.
  • Watch for synthetic-input leakage.Because the synthetic notes were LLM generated, it’s easy for them to accidentally echo the template’s own field header phrasing, which lets the model shortcut on synthetic data and then stumble on real, messier clinician notes. Stripping header like phrasing from the generation prompts mattered more than we expected.
  • Keep a human reviewed slice, permanently.Even at 98% strict accuracy, we kept a rolling human spot-check given the stakes. A custom eval harness catches structural errors, not everything a domain expert would catch!

Wrapping up

Fine-tuning is for behavior that needs to be exact, repeatable, and cheap at scale. Our task happened to be almost entirely this: a fixed, deeply branching output format where being 90% right on any given field is functionally the same as being wrong.

We kept RAG and system prompts around for the parts of the pipeline that actually need up to date knowledge. We just stopped asking a 30k token system prompt to do a fine-tune’s job.

I hope this helps you understand not only the mathematical intution behind LoRA/QLoRA, but how to implement it yourself.

Enjoy!

References

[1] Aghajanyan, A., Gupta, S., & Zettlemoyer, L. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv preprint arXiv:2012.13255. (Referenced regarding the “lower intrinsic dimension” of LLM parameter updates).

[2] Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv preprint arXiv:2305.14314. (Referenced for 4-bit NormalFloat (NF4), Double Quantization, and Paged Optimizers).

[3] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685. (Referenced for the core mathematical intuition of learning low-rank matrices for efficient adaptation).

[4] Li, Xian, et al. (2024). Self-Alignment with Instruction Backtranslation. International Conference on Learning Representations (ICLR).

[5] Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021). Finetuned Language Models are Zero-Shot Learners. arXiv preprint arXiv:2109.01652. (Referenced for the paradigm shift from next-token prediction to instruction tuning and formatting).

[6] Schulman, J. / Thinking Machines Lab. LoRA without Regret. (Referenced for hyperparameter intuition, specifically the necessity of applying LoRA adapters to all dense layers rather than just attention matrices).

[7] Unsloth. LoRA Hyperparameters Guide. (Referenced for baseline empirical defaults for rank, alpha, learning rate, and weight decay).

[8] BitsAndBytes Foundation. bitsandbytes. GitHub Repository. https://github.com/bitsandbytes-foundation/bitsandbytes (Underlying library used for NF4 quantization and paged memory management).

[9] Hugging Face. Transformers, PEFT (Parameter-Efficient Fine-Tuning), TRL (Transformer Reinforcement Learning), and Datasets. (Core ecosystem utilized in the training script pipeline).

[10] Mistral AI. Mistral-7B-Instruct-v0.3. (The foundational base model utilized for the fine-tuning implementation script).