This week, an unassuming post in the Reddit local-AI community surfaced a striking number: 384K context, 90 tokens/second — running on nothing more than two retired crypto-mining GPUs (CMP 170HX, each with 64GB HBM). A side-by-side test of Alibaba's Qwen3.8-Flash-Next produced comparable throughput. The open-source releases from two of China's leading model labs now run on "non-mainstream" hardware — and that tells us the cost frontier of self-hosted large models is quietly being redrawn.

What this is

The author used Zhipu's GLM-5.3-Flash (community version), a 320-billion-parameter / 18-billion-active Mixture of Experts (MoE) model — the architecture carries a huge total parameter count, but only activates a small fraction per token, which keeps inference costs low. Quantized to an average of 3.05 bits per weight (EXL3 quantization, which compresses 16-bit parameters down to ~3 bits to save VRAM), the model weighs in at roughly 125GB — a near-perfect fit for the 128GB of VRAM across the two mining GPUs.

For comparison, Alibaba's Qwen3.8-Flash-Next (community version) used a different quantization stack (AWQ INT4 + FP8 PLE) on the vLLM inference engine and delivered stable throughput as well. The implication: roughly ¥100,000–200,000 (around $15,000–$20,000) in hardware can now stand up a "self-hosted ChatGPT" — long documents stay off the cloud, data never leaves the premises.

How the industry sees it

Skeptics-turned-believers will tell you: 384K context plus 90 tok/s is enough for real workloads (the author used the DSH benchmark suite to evaluate programming and Agent tasks — i.e., letting AI autonomously invoke tools to complete multi-step operations). For enterprises with large internal document corpora and compliance constraints, this door has been cracked open.

The counter-arguments are equally clear:

First, this is a single data point, not a benchmark. The author himself admits GLM and Qwen used different engines and different speculative decoding (a speedup technique where a small model drafts tokens first and the large model verifies them) setups.

Second, 3.05 bpw quantization carries real quality loss. EXL3 uses trellis quantization (which assigns precision by tensor importance to preserve sensitive layers), but the numbers the user posted: against the BF16 original, top-1 agreement is roughly 81.6% and KL divergence sits near 0.28 — usable, but not "near-lossless."

Third, mining GPUs are a special case. The CMP 170HX has no video output and only PCIe Gen2 x8 bandwidth — it works only because its large VRAM and the MoE architecture's sparse activation keep per-token compute low. Switching to regular consumer cards, two used 64GB units would still cost ¥10,000+ (around $1,400) each. This is not a story of "anyone can now run large models locally."

What this means for regular people

For enterprise IT: long-document summarization, internal knowledge bases and similar workloads have crossed the technical-feasibility bar for self-hosted solutions — but you still need 1-2 engineers who understand inference optimization, which often costs more than the hardware itself.

For individual professionals: no direct short-term impact. Firing up ChatGPT, ERNIE Bot, or Tongyi in a browser tab is still the most cost-effective option. But if your company comes to you about a "self-hosted AI" data-governance initiative down the road — this post is the basis for judging whether to nod along or push back.

For the consumer market: the signal matters more than the immediate impact. The willingness of Zhipu and Alibaba to keep maintaining releases whose capability approaches closed-source models is the supply-side guarantee that API prices will continue falling. The few hundred yuan you're paying monthly for subscriptions today will most likely be half that in three years.