This week, Austria's ISTA-DASLab lab compressed a 27B (27-billion-parameter) Alibaba Qwen model into 10GB — the size of a standard USB stick — while nearly matching the original's quality. The "good-enough bar" for local AI just dropped another notch.

What this is

The bigger an AI model, the smarter it gets — but also the more space it eats. A 27B-parameter model in its native form weighs roughly 54GB and requires enterprise-grade GPUs to run. Quantization is a "slimming" technique that compresses a model's internal high-precision numbers (16-bit) into low-precision representations (2–3 bit), shrinking file size by 5–10x — but traditional approaches make the AI dumber.

This new quantized version applies two techniques: Gumbel-Softmax (lets the AI learn how to compress itself most efficiently) and RCO (allocates more precision to critical layers while keeping total size fixed). The result: a model file of 8.4 to 10.1GB that achieves a perfect score on math competition sets (AIME25) and posts 84.57 on a coding benchmark (LiveCodeBench) — within 1 point of the original BF16 version.

Worth flagging: its average zero-shot score (75.70) actually edges past the original (74.34). That suggests quantization isn't purely "cutting things away" — it may reshape certain behavioral properties of the model.

Industry view

The reaction has been largely positive. Hugging Face and the r/LocalLLaMA community have circulated this widely, calling it "the strongest quantized version at this size." Open-source enthusiasts see it as proof that local AI is starting to catch up with cloud-level performance.

But stepping back, several caveats deserve mention:

  • "Zero-shot beating the original" sounds counterintuitive — it usually reflects quirks of specific benchmarks and shouldn't be casually equated with "more useful."
  • The 10GB file may be small, but running it smoothly requires at least 16GB of VRAM (e.g., an RTX 4090, priced around RMB 10,000+). Integrated graphics on regular laptops still struggle.
  • ISTA-DASLab is a European academic lab; the work is paper-stage. Whether industry adopts it and when it commercializes remains unclear.

Our deeper read: the open-source ecosystem is producing a new trend — "good enough" models are becoming reality. This puts potential pricing pressure on Nvidia (sells the compute) and cloud providers (AWS, Alibaba Cloud).

Impact on regular people

For enterprise IT: buy a single server or workstation with 16GB of VRAM, and you can deploy an AI roughly on par with cloud offerings — no need to upload customer data or contract documents to OpenAI or Alibaba Cloud. Compliance and privacy costs drop.

For professionals: a mid-to-high-end gaming laptop (RTX 4060 or above) can now run a usable local code assistant or translation tool. For lawyers, doctors, and consultants handling sensitive documents, this is a "good enough and stays local" option.

For the consumer market: when open-source models are "good enough," the subscription pricing power of cloud AI gets diluted. The premium for products like ChatGPT Plus and Claude Pro is likely to face further compression.