A paper on arXiv delivers an uncomfortable conclusion: when large language models are compressed with Nvidia's NVFP4 4-bit floating-point format, judging only on outputs masks structural degradation inside the model. On the surface, the model answers almost identically to the original; underneath, its middle-layer "way of thinking" has already been distorted.
What this is
NVFP4 is an ultra-low-precision format Nvidia pushed hard last year (each parameter uses roughly 4 bits, far below the common 16 bits), designed to make large-model inference faster and less memory-hungry. Compressing a model from high precision down to NVFP4 usually costs accuracy, and the engineering community typically compensates with "distillation" — making the compressed "student" model mimic the uncompressed "teacher" model's outputs.
The researchers found that conventional distillation only checks whether outputs match, the equivalent of looking at student exam scores without checking whether anything was actually learned. Using CKA (Centered Kernel Alignment, a metric for measuring representation similarity across neural network layers) to probe the model's "internal geometry," they discovered that NVFP4-distilled models deviate sharply from the original in middle layers, and the effect is especially pronounced on models that have gone through RL post-training (post-training, further fine-tuning with human feedback or reward signals). This internal drift shows up directly as capability drops on reasoning and coding tasks.
The paper proposes an improved method, CKA-QAD, which adds a regularization term (a constraint) to distillation that forces middle-layer structure to align as well. The approach delivers significant gains on Nemotron 3 Nano and Qwen3-4B-Thinking with minimal extra training overhead.
Industry view
Supporters argue the paper hits a blind spot in today's low-precision deployment: everyone is racing for 4-bit and 6-bit inference speed, but almost no one is rigorously checking what the model actually loses. If internal representation drift is widespread, NVFP4's vaunted cost advantage may need to be discounted, and small methodological improvements could buy back real capability — worth taking seriously on the deployment side.
Skeptical voices remain. NVFP4 has not been in production environments for long, the paper experiments on only two models, and the absolute drop in reasoning-task metrics is limited. Practitioners point out that real production workloads are usually backstopped by RAG (Retrieval-Augmented Generation, where the model searches external material before answering) and prompt engineering, so internal representation drift may not propagate to the end-user experience. Others note that the paper's method has not been validated for scalability inside industrial training pipelines — for now it reads more like an academic increment than an engineering paradigm shift.
Impact on regular people
For enterprise IT: when teams plan to switch models to 4-bit precision to save GPU (graphics processors, the core AI compute hardware) costs, inference accuracy metrics need to be broken down more granularly rather than just checking that a demo runs — capability loss can hide in edge-case scenarios.
For individual professionals: workflows that depend on tight reasoning, like writing code or doing data analysis, may feel the jitter from low precision more sensitively than chat-style Q&A, so leave room for an extra set of baseline comparisons when picking tools.
For the consumer market: ordinary users will probably not notice a difference, because vendors compensate with engineering tricks; but if a service provider over-compresses and strips out internal checks, the end experience will eventually get slower or dumber.