Over the past two years, large-model training has advanced along two more pragmatic technical paths: DPO (Direct Preference Optimization) eliminates the reward model, while GRPO (Group Relative Policy Optimization) reduces the computational cost of reinforcement learning. The shift is not merely about simpler algorithms; it reflects an industry move away from asking “How accurate is the reward model?” toward “Can limited compute reliably train models with useful capabilities?”
What This Is
DPO was proposed by Rafael Rafailov and colleagues at Stanford University. Its core idea is to use preference data—“good answers” versus “bad answers”—to directly adjust a model, without separately training a reward model. Traditional RLHF (Reinforcement Learning from Human Feedback) typically requires the simultaneous participation of a policy model, value model, reference model, and reward model. It consumes substantial GPU memory, is sensitive to parameter choices, and is prone to “reward hacking,” in which a model exploits flaws in the scoring system.
DPO turns preference alignment into a training task similar to supervised learning. It needs only the model being trained and a frozen reference model, substantially shortening the engineering pipeline. GRPO takes a different route: it retains reinforcement learning but has the model generate multiple answers to the same question, then allocates rewards according to relative performance within the group, reducing reliance on the traditional value model. Reasoning models such as DeepSeek-R1 have adopted this approach.
Industry View
We believe DPO’s value lies first in cost and stability. Preference data is generally easier to obtain than point-by-point scoring, and the training process removes one model-fitting and repeated-sampling stage, making the approach easier for enterprises to reproduce. Over the past year, DPO has become a foundational tool for many open-source models and fine-tuning solutions.
That does not mean traditional RLHF is obsolete. The objections are clear: preference data contains labeling bias, and a model may learn to please annotators rather than improve factual accuracy. DPO is also better at adjusting response style and may not consistently unlock complex reasoning. Although GRPO lowers some costs, it still requires generating and validating large numbers of candidate answers, so its computational overhead remains considerable. Algorithm paths are diverging, and no single approach can solve every problem.
Impact on Regular People
For enterprise IT: Training a model aligned with corporate rules no longer necessarily requires an expensive reward model, allowing even small teams to customize models using preference data. Data governance and evaluation, however, still determine the final results.
For individual careers: Model responses will align more closely with a team’s preferred style of expression. The value of AI assistants will increasingly appear in writing, summarization, and workflow execution rather than general-knowledge display.
For the consumer market: Reasoning models will enter more search, productivity, and developer products. What users perceive will be more stable answers and more complete steps—not the guarantee of a correct answer every time.