DPO trains models to prefer good responses over bad ones using human preference data — without the complexity of reinforcement learning. This guide covers collecting preference data, training with TRL's DPO trainer, and evaluating results.
Supervised Fine-Tuning (SFT) teaches the model to produce specific outputs for specific inputs. Direct Preference Optimization (DPO) teaches the model to prefer certain types of outputs over others — making it better at following instructions, being helpful, or matching your brand voice — without specifying exact outputs.
Use SFT when you have a clear correct answer (classification, extraction, specific formats). Use DPO when you want to improve the model's judgment and style — reducing verbosity, improving tone, better handling of refusals, or aligning with your organisation's communication style.
DPO requires paired preference data: for each prompt, a 'chosen' response (the better one) and a 'rejected' response (the worse one). The format is `{"prompt": "...", "chosen": "...", "rejected": "..."}`. Aim for 1000+ pairs for reliable results.
The most efficient way to collect pairs: run your base model on 500 prompts, generating 2 different responses per prompt (use `n=2` in the API call), then have human raters choose the better response. Alternatively, use a stronger model as an automatic judge to rank pairs from your weaker model.
Install the required packages: `pip install trl transformers peft accelerate bitsandbytes`. Load your base model with 4-bit quantisation: `from transformers import AutoModelForCausalLM; model = AutoModelForCausalLM.from_pretrained('mistralai/Mistral-7B-Instruct-v0.3', load_in_4bit=True)`.
Configure the DPO trainer: `from trl import DPOTrainer, DPOConfig; config = DPOConfig(beta=0.1, max_length=2048, output_dir='dpo-output'); trainer = DPOTrainer(model=model, args=config, train_dataset=dataset, tokenizer=tokenizer)`. The `beta` parameter controls how strongly the model is pushed away from the rejected responses — 0.1 is a good starting value.
Run training: `trainer.train()`. DPO training is generally faster than SFT because datasets are smaller. On a single A100 with a 7B model, 1000 pairs takes around 30–60 minutes at 4-bit quantisation.
Evaluate by running your test prompts through both the base model and the DPO-trained model, then using an LLM judge to compare outputs. Good DPO training should show: more concise responses if you trained for that, stronger instruction following, and reduced occurrence of the specific failure patterns captured in your 'rejected' examples.
Low-quality preference labels are the biggest failure mode. If raters were inconsistent about what 'better' means, the model learns noise. Provide clear annotation guidelines before collecting data: specify what makes a response better in measurable terms.
Catastrophic forgetting is worse with DPO than SFT. Always evaluate general capabilities after training. If the model regresses on tasks unrelated to the preference data, reduce `beta` or add a small SFT mix to the training data to anchor general capabilities.
Unsloth is an optimised fine-tuning library that makes LoRA training 2–5× faster with 70% less VRAM. This guide walks through fine-tuning a Llama or Mistral model on your own data using Unsloth on a free Google Colab GPU.
Read guideThe quality of your training data is the biggest factor in fine-tuning success. This guide covers data collection strategies, formatting standards, quality filtering, and the minimum viable dataset size for different tasks.
Read guide