4 Direct Preference Optimization

Motivated by the challenges of applying reinforcement learning algorithms on large-scale problems such as fine-tuning language models, our goal is to derive a simple approach for policy optimization using preferences directly. Unlike prior RLHF methods, which learn a reward and then optimize it via RL, our approach leverages a particular choice of reward model parameterization that enables extraction of its optimal policy in closed form, without an RL training loop. As we will describe next in detail, our key insight is to leverage an analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functions into a loss function over policies. This change-of-variables approach avoids fitting an explicit, standalone reward model, while still optimizing under existing models of human preferences, such as the Bradley-Terry model. In essence, the policy network represents both the language model and the (implicit) reward.

Deriving the DPO objective

We start with the same RL objective as prior work, Eq. 3, under a general reward function $r$. Following prior work [31], [30], [19], [15], it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective in Eq. 3 takes the form:

$$\pi_r(y\mid x)=\frac{1}{Z(x)}\pi_{\mathrm{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right), \tag{4}$$

where $Z(x)=\sum_y\pi_{\mathrm{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)$ is the partition function. See Appendix A.1 for a complete derivation. Even if we use the MLE estimate $r_\phi$ of the ground-truth reward function $r^*$, it is still expensive to estimate the partition function $Z(x)$ [19], [15], which makes this representation hard to utilize in practice. However, we can rearrange Eq. 4 to express the reward function in terms of its corresponding optimal policy $\pi_r$, the reference policy $\pi_{\mathrm{ref}}$, and the unknown partition function $Z(\cdot)$. Specifically, we first take the logarithm of both sides of Eq. 4 and then with some algebra we obtain:

$$r(x,y)=\beta\log\frac{\pi_r(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x). \tag{5}$$

We can apply this reparameterization to the ground-truth reward $r^*$ and corresponding optimal model $\pi^*$. Fortunately, the Bradley-Terry model depends only on the difference of rewards between two completions, i.e., $p^*(y_1\succ y_2\mid x)=\sigma(r^*(x,y_1)-r^*(x,y_2))$. Substituting the reparameterization in Eq. 5 for $r^*(x,y)$ into the preference model Eq. 1, the partition function cancels, and we can express the human preference probability in terms of only the optimal policy $\pi^*$ and reference policy $\pi_{\mathrm{ref}}$. Thus, the optimal RLHF policy $\pi^*$ under the Bradley-Terry model satisfies the preference model:

$$p^*(y_1\succ y_2\mid x)=\frac{1}{1+\exp\left(\beta\log\frac{\pi_{\mathrm{ref}}(y_2\mid x)}{\pi^*(y_2\mid x)}-\beta\log\frac{\pi^*(y_1\mid x)}{\pi_{\mathrm{ref}}(y_1\mid x)}\right)}. \tag{6}$$

The derivation is in Appendix A.2. While Eq. 6 uses the Bradley-Terry model, we can similarly derive expressions under the more general Plackett-Luce models [32], [23], shown in Appendix A.3. Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parametrized policy $\pi_\theta$. Analogous to the reward modeling approach (i.e. Eq. 2), our policy objective becomes:

$$L_{\mathrm{DPO}}(\pi_\theta;\pi_{\mathrm{ref}})=-\mathbb{E}_{(x,y_w,y_l)\sim D}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right]. \tag{7}$$

This way, we fit an implicit reward using an alternative parameterization, whose optimal policy is simply $\pi_\theta$. Moreover, since our procedure is equivalent to fitting a reparametrized Bradley-Terry model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution [4]. In Section 5, we further discuss theoretical properties of DPO in relation to other works.

What does the DPO update do?

For a mechanistic understanding of DPO, it is useful to analyze the gradient of the loss function $L_{\mathrm{DPO}}$. The gradient with respect to the parameters $\theta$ can be written as:

$$\begin{aligned} \nabla_\theta L_{\mathrm{DPO}}(\pi_\theta;\pi_{\mathrm{ref}}) ={}&-\beta\mathbb{E}_{(x,y_w,y_l)\sim D}\left[\sigma\left(\hat r_\theta(x,y_l)-\hat r_\theta(x,y_w)\right)\right.\\ &\left.\qquad\qquad\qquad\quad\cdot\left(\nabla_\theta\log\pi_\theta(y_w\mid x)-\nabla_\theta\log\pi_\theta(y_l\mid x)\right)\right], \end{aligned}$$

where $\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}$ is the reward implicitly defined by the language model $\pi_\theta$ and reference model $\pi_{\mathrm{ref}}$ (more in Section 5). Intuitively, the gradient of the loss function $L_{\mathrm{DPO}}$ increases the likelihood of the preferred completions $y_w$ and decreases the likelihood of dispreferred completions $y_l$. Importantly, the examples are weighed by how much higher the implicit reward model $\hat r_\theta$ rates the dispreferred completions, scaled by $\beta$, i.e. how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naïve version of this method without the weighting coefficient can cause the language model to degenerate (Appendix Table 3).

DPO outline

The general DPO pipeline is as follows: 1) Sample completions $y_1,y_2\sim\pi_{\mathrm{ref}}(\cdot\mid x)$ for every prompt $x$, label with human preferences to construct the offline dataset of preferences $D=\{(x^{(i)},y_w^{(i)},y_l^{(i)})\}_{i=1}^N$ and 2) optimize the language model $\pi_\theta$ to minimize $L_{\mathrm{DPO}}$ for the given $\pi_{\mathrm{ref}}$ and $D$ and desired $\beta$. In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using $\pi_{\mathrm{SFT}}$, we initialize $\pi_{\mathrm{ref}}=\pi_{\mathrm{SFT}}$ whenever available. However, when $\pi_{\mathrm{SFT}}$ is not available, we initialize $\pi_{\mathrm{ref}}$ by maximizing likelihood of preferred completions $(x,y_w)$, that is,

$$\pi_{\mathrm{ref}}=\arg\max_\pi\mathbb{E}_{x,y_w\sim D}[\log\pi(y_w\mid x)].$$

This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and $\pi_{\mathrm{ref}}$ used by DPO. Further details related to the implementation and hyperparameters can be found in Appendix B.