218 tokens/sec — this week, a developer on Reddit's LocalLLaMA community pushed Alibaba's Qwen team's 27B-parameter model (Qwen3.8-27B) to this single-request decoding speed using two consumer-grade RTX 3090s (24GB VRAM each, ~¥3,000 / ~$420 used). What's worth noting: workloads that previously required H100 (NVIDIA's data-center GPU, ~¥250,000 / ~$35,000 per card) clusters to run smoothly now run on two consumer cards.

What this is

The poster quantized the model to INT4 (4-bit integer storage that trades a bit of precision for size and speed) and used vLLM (open-source LLM inference framework) plus DFlash2 (a speculative decoding draft model: a small model guesses tokens first, the large model verifies them in batches) to accelerate. Peak VRAM hit 22.3 GB per card, with total usage around 45 GB — pushing the two 3090s to their limit. Prefill speed (how fast the model "reads" the prompt) reached 1342 tokens/sec on 10,000-token inputs. Notably, the DFlash2 draft model itself eats about 13.5 GB of VRAM — that space was carefully carved out.

Industry view

Optimists see this as a signal that LLM "democratization" is accelerating: many enterprise inference workloads can be served by hundreds of thousands of consumer-grade GPUs, not million-dollar H100 clusters. But we'll pour some cold water here — the poster had to fork vLLM and apply a patch to make this work. This is essentially a tinkerer's achievement, not an out-of-the-box solution. INT4 quantization sacrifices precision, so it may not suit rigorous scenarios like finance, law, or healthcare. Another overlooked risk: the two 3090s rely on the PCIe bus and patched P2P communication. The poster didn't disclose how stable that setup is under sustained high load.

Impact on regular people

For enterprise IT: For tasks with higher precision tolerance — internal document Q&A, summarization — validating on two 3090s first before going to the cloud can save money and sidestep compliance risks from data leaving the premises.

For individual professionals: Handling sensitive information like client contracts or internal data, local 27B models are starting to become viable — provided you're willing to invest in hardware and debugging time. For those who don't want to tinker, cloud solutions remain the more practical path.

For the consumer market: Once hardware costs drop, next-generation home AI boxes and AI workstations will keep getting cheaper, and monthly AI subscription pricing will continue to be pushed lower.