The InstructGPT paper reveals a fact: for LLMs transitioning from "understanding knowledge" to "understanding human intent," core stability relies almost entirely on the PPO algorithm. This is our key to understanding why LLMs can be deployed safely.
What this is
When LLMs learn human preferences through RLHF (Reinforcement Learning from Human Feedback, a method adjusting AI output based on human preferences), they easily go to extremes. If an answer receives a high score, traditional methods will make the model bet all its probability on it next time, causing the model to "go crazy" and generate gibberish. PPO (Proximal Policy Optimization, an algorithm for AI to robustly learn human preferences) solves this. We note that it acts like a cautious coach, limiting the magnitude of each update through "clipping"—capping the step size at a maximum of 20%; while adding a KL penalty (a constraint limiting the degree to which the AI deviates from its original knowledge), ensuring the model doesn't lose basic language capabilities just to chase high scores.
Industry view
Currently, PPO is the de facto standard for the RLHF phase at top-tier companies like OpenAI, and its stability has been proven over time. However, what concerns us is that industry complaints against it are rising: its computational cost is extremely high. During training, it requires four models to operate simultaneously—the policy model, reward model, reference model, and value model—consuming staggering amounts of VRAM. Additionally, new routes like DPO (Direct Preference Optimization, a resource-saving algorithm that bypasses the scoring model) are challenging it. Critics argue that for resource-constrained companies, the engineering complexity and tuning difficulty of PPO are often the primary reasons alignment projects fail.
Impact on regular people
For enterprise IT: The computing bill must be rewritten. The hardware cost of PPO training far exceeds the fine-tuning phase, requiring ample budget reserves.
For the workplace: As model self-correction capabilities improve, the dividend period of manually tweaking prompts is shortening; business understanding is now more important than prompt-crafting skills.
For the consumer market: The improvement in LLMs' "human-like" and "safe" experiences is driven by this very training mechanism, raising the baseline of product experience.