This week on r/LocalLLaMA, developer Daxfortuna published an audit: out of 443 "low-VRAM versions" of the NVIDIA Nemotron LLM circulating in the wild, none were actually low-VRAM. The file labels claimed 2–3 bit (a quantization precision unit—the lower the number, the more aggressive the compression), but real measurements showed around 4.70 bit.
He sifted through 25 repositories before uncovering it: it's not a model problem—it's a bug in llama.cpp (the mainstream open-source engine for running LLMs locally). Both k-quants and i-quants (two quantization algorithms) require matrix row widths to be divisible by 256. Nemotron's weights don't satisfy that condition, so the tool silently swapped in a 32-block type while keeping the filename unchanged.
His solution is called ShimQuant—it pads the row width to 256 before compressing, achieving a final 3.07 bit, 11.77 GB model with 260k token context support. HumanEval (a code capability benchmark) scored 91.5%, on par with the official 19.65 GB version—7 GB smaller.
What This Is
In plain terms: 16GB GPUs that couldn't previously run Nemotron can now run it, but there's a hard condition—ShimQuant only works inside a patched llama.cpp. LM Studio, Ollama, and other "download-and-go" clients won't work; they throw errors on load.
This is a "runs but is hard to use" solution. The developer himself didn't claim this was optimal—it's just the only currently working option under 18GB.
Industry View
The optimistic take: this is exactly how open source should work—one person digging deep enough catches a bug the manufacturer missed. We expect a "patch culture" like ShimQuant to push the entire quantization toolchain toward maturity.
But we see warning signs. First, Josh Bolding had to audit 443 files to find one obvious bug, which tells us this toolchain lacks standardized auditing. Second, if "fake compression" happened with other models, users may never have noticed—thinking they were running low-VRAM versions while actually burning through VRAM. Third, his method is 7 GB smaller than the official version, but HumanEval only matches—there's no proof that smaller means "good enough"; it's just "runs."
A longer-term concern: as "on-device LLM" and "AI PC" become marketing slogans, regular users and enterprise IT departments will find it increasingly hard to distinguish what's "actually runs" from what's "labeled to run."
Impact on Regular People
For enterprise IT departments: the "16GB minimum" claims around local LLM deployment can't be taken at face value yet—we'd recommend having the team test before procurement.
For individual professionals: anyone hoping to replace paid APIs with local AI to save money still needs to prepare for "compile it yourself, debug it yourself"—"download and go" isn't there yet.
For the consumer market: products labeled "AI PC" or "on-device LLM" are still safer running the vendor-tuned versions in the short term; DIY headroom is limited.