We noticed a set of numbers: GitHub user curvedinf used four used AMD MI100 GPUs (whole rig about $6,500) to run Alibaba's Qwen3 27B model, pushing token generation from stock vLLM's 15 token/s to 972 token/s—roughly 65x; batch processing hit 5,680 token/s. This isn't enterprise engineering; it's a personal fork project. Our read: the signal here isn't "yet another vLLM fork (an independent branch built off open-source code)," but that it has sharply lowered the hardware floor for "running a respectable large model."
What this is
vLLM is today's mainstream open-source large-model inference framework, but the author points out its INT8 (a low-precision compute format using 8-bit integers in place of 16/32-bit floats—faster, smaller, and barely supported on older hardware) support is "full of holes." curvedinf's work is patching each of those holes: an INT8 matrix multiplication library, INT8 KV cache (the intermediate storage a model uses during inference to "remember" context), INT8 versions of the attention mechanism (the core module that decides "which part of the input to look at"), custom cross-GPU communication—the full pipeline running on a single 4-MI100 machine.
The MI100 is an AMD accelerator card from about four years ago, with no native FP8 (a more advanced 8-bit float format) support. The author's approach is essentially "making hardware without FP8 still work well," and explicitly states the code is portable to MI50, MI210, and some older Nvidia cards.
Industry view
Supporters will treat this as proof that "AI is no longer locked to a few chip vendors." For budget-sensitive enterprises and industries with strict data-compliance requirements (finance, healthcare, government), running a 27B model locally, cheaply, and at batch scale is something that was previously unthinkable.
We also want to flag the flip side: this is code maintained by a single person, with no commercial support; it's only been "carefully polished" on Qwen3 so far—other models are only theoretically portable; a good benchmark (a controlled performance test) doesn't equal production-grade SLA (service-level agreement—a vendor's written commitment on stability and response time). Community voices also note that while INT8's precision loss under long context and complex reasoning is small, the quantization data is thin. Worth getting excited about, but still some distance from "ready to deploy to production."
Impact on regular people
For enterprise IT: "Building your own inference cluster" used to mean a seven-figure budget. You can now stand up a workable 27B deployment node for under RMB 100,000—but only if you have engineers willing to read source code, troubleshoot, and tune.
For individual careers: "Running a large model locally" is migrating from a geek toy to a mainstream technical skill. People who understand this full pipeline will become increasingly valuable inside enterprises; conversely, the premium on roles that just call APIs will shrink.
For consumer markets: Short term, end users won't feel much—they'll keep using the cloud. Medium to long term, the two-tier pattern of "small models locally, large models in the cloud" will sharpen, and the hardware cost of local AI assistant products will continue to drop.