We've noticed an underrated fact: pretraining consumes 90% of compute, but the 5% mid-training stage is what actually determines whether a model is smart. A long technical post on Juejin lays this out clearly.
What this is
The piece breaks LLM training into a "pipeline." The typical flow has four stages: pretraining (letting the model see the world), mid-training (adding pressure in math, code, and reasoning), supervised fine-tuning or SFT (teaching the model to follow instructions), and alignment (using RLHF, or reinforcement learning from human feedback, or DPO, direct preference optimization, to make answers "helpful, harmless, honest"). A side path is distillation — condensing a large model's capability into a smaller one. Cross-referencing public disclosures from DeepSeek-V3, Qwen2.5, LLaMA-3.1, and Kimi K1.5, the conclusion is: top Chinese and US players' training flows have converged.
Industry view
The article targets training engineers, but reveals a cost structure worth noting for non-technical readers: pretraining consumes 90%+ of GPU-hours, mid-training roughly 5%, SFT under 1%, and alignment has the smallest compute share but the highest engineering complexity.
"Where the money goes" and "where capability grows" aren't the same thing. China's LLMs catching up under compute constraints isn't about stacking GPUs in pretraining — that's a capital and supply chain contest — but about differentiating in mid-training, SFT, and alignment, where compute is small but the technique matters. DeepSeek R1's GRPO breakthrough (a critic-free reinforcement learning method) on reasoning is exactly this kind of play.
Counterpoints exist. Framing the pipeline as this industrialized risks obscuring the importance of fundamental research. When everyone is copying the same SFT+DPO+distillation template, the next breakthrough may come from a small team or academic lab that doesn't play by the rules.
Impact on regular people
For enterprise IT: with the training flow standardized, the engineering bar for "building your own LLM" is lower than people think, but the complexity concentrates in alignment and stability. For most scenarios, buying a third-party API is still the better deal.
For professionals: understanding "models aren't trained in one shot" helps you read vendor pitches — the capability of a "self-developed foundation model" comes from pretraining or mid-training? The former needs capital, the latter needs data and methodology.
For consumer markets: open-source models are rapidly closing the gap with closed ones. The output of the Qwen, Kimi, and DeepSeek line means AI service marginal costs will keep falling — the application layer deserves more attention than the model layer.