This week we noticed a post on Reddit's r/LocalLLaMA — a user ran Alibaba's open-source Qwen3.8 27B model on two RTX 3090 GPUs (gamer-grade hardware, roughly 4,000–5,000 RMB on the secondhand market), hitting 143 tokens per second (tokens are the smallest units of text a model processes, roughly equivalent to one Chinese character). After a water-cooling mod, temperatures dropped from 70°C to 35°C. His own verdict: "I feel like I've finally graduated."
What this is
This event is a small slice of a broader trend: the barrier to running LLMs locally is falling fast.
Running large models locally used to be a developer's hobby — you needed to know Python, CUDA (Nvidia's parallel computing framework), and model quantization (compressing a model to reduce VRAM usage), while tolerating output speeds of a few characters per second. The user's previous setup with LM Studio was fine for tinkering, but visibly struggled with 27B-parameter models (more parameters generally means a smarter model, but requires far more compute).
vLLM is an open-source inference engine out of academia that boosts throughput via techniques like PagedAttention (managing GPU memory the way an OS uses paging to manage RAM). Community developer Syv-ai tuned it specifically for the RTX 3090 (24GB VRAM, a 2020 release), giving old hardware a second wind.
Industry view
Beyond the Reddit thread's chorus of congratulations, the cooler voices are worth hearing. Our read: the real significance isn't the number 143 — it's that the combo of "old hardware + Chinese open-source model + community optimization" ran end-to-end for the first time.
The optimistic camp sees this as proof that Qwen and other Chinese open-source models are penetrating the global developer community. Alibaba's investment in open-source LLMs is paying off; overseas developers are willingly adapting and optimizing the stack — a sharp contrast to six months ago, when Qwen was mostly a domestic talking point.
But the pushback is clear. First, cloud APIs now process hundreds of thousands of tokens for a dollar; hardware depreciation and electricity for local rigs don't always pencil out. Second, the 3090 has been out of production for nearly three years — building an ecosystem on legacy silicon carries sustainability risk. Third, what actually blocks enterprises was never inference speed, but the "soft" costs of data compliance, system integration, and long-term maintenance. One commenter's line was the truest: "Just because you can run it doesn't mean you should — depends on the use case."
Impact on regular people
For enterprise IT: The open-source model + consumer-GPU combo offers SMEs a "skip the cloud" alternative worth evaluating — especially in data-sensitive verticals like healthcare, legal, and manufacturing.
For individual professionals: No need to panic — local AI is nowhere near replacing your daily workflow. But knowing names like Qwen and vLLM puts you on stronger footing when talking to IT departments or outside vendors.
For the consumer market: "Local AI" may become a fresh marketing hook for high-end PCs and gaming laptops, but in the short term it remains an enthusiast conversation that won't change what most people buy.