DPO (direct preference optimization)

DPO, direct preference optimization, trains a model on pairs of answers — one preferred, one rejected — using a plain fine-tuning loss instead of reinforcement learning.

RLHF reaches its goal by a long detour: train a separate reward-model to imitate human taste, then run a reinforcement-learning loop against it. Like hiring a food critic to write a scoring rubric, then training your cook against the rubric. DPO's authors proved the detour is mathematically unnecessary: the model can learn from the taste-tests directly. Show the cook both plates — "this one was preferred" — and adjust immediately.

Concretely, DPO takes the same data RLHF would use (prompt, chosen answer, rejected answer) and applies a loss that raises the model's probability of the chosen answer and lowers the rejected one, relative to a frozen reference copy of the model. The reference copy is the leash: it stops the model drifting into nonsense while chasing preferences. The 2023 paper's title says it plainly: "Your Language Model is Secretly a Reward Model."

What this buys practically: no reward model to train, no PPO machinery to babysit, roughly the cost and stability of ordinary fine-tuning — which put preference training within reach of small teams, and made DPO the default in the open-source world (Zephyr, Llama-3-family recipes, many others). Frontier labs still often use RL-based methods, which can squeeze further; variants like IPO and KTO tweak DPO's loss. But the working rule stands: preference data in, one supervised-style loss, aligned model out.

Where to go next#

• Full lesson: RLHF
• Related terms: rlhf, reward-model, instruction-tuning, fine-tuning