RTX 3090 Single Card Doubles Qwen 27B Inference — Local LLM Playbook Rewritten

What this is

A developer has built a custom CUDA kernel dubbed megakernel — a GPU compute unit tightly tailored to a specific model — that pulls Tongyi Qianwen's Qwen3-27B on a single consumer-grade RTX 3090 from 73 tokens/sec to 140 tokens/sec on code-generation inference. That's nearly a 2x jump. The KL divergence (a measure of distance between two distributions; lower = more similar) against standard-inference output sits at just 0.0009 — quality loss is essentially negligible.

The code is open-sourced on GitHub as open-jet. The author, L-Forster, is candid about the limits: the kernel is tuned only for the 3090; the 4090, 5090, and other cards have not yet been ported.

Industry view

Two things stand out to us.

First, the hardware cost of running LLMs locally is sinking. A "good enough" 27B-parameter class now hitting usable speeds on a consumer card redraws the private-deployment budget line for mid-size companies. Workloads that previously required at least an A6000 or a rented cloud GPU may now be handled by a secondhand 3090.

Second, community effort can still beat mature frameworks head-on. llama.cpp is the product of dozens of contributors and years of maintenance — and one independent developer's hand-written CUDA kernel beat it by close to 2x. A genuinely rare "small-shop vs. incumbent" case in this industry.

Three caveats warrant cool-headedness. LLM inference optimization is a domain of diminishing marginal returns: megakernel's edge collapses to only ~1.4x in 30K-token long-context scenarios. The author concedes there are no task-level pass-rate evaluations yet — low perplexity does not equal real task performance. And the kernel has only been tuned on a single card, so generalizability remains an open question.

Impact on regular people

For enterprise IT: Mid-size private-deployment hardware budgets warrant a fresh look. Consumer GPUs are now viable, but teams must weigh long-term stability and community-support risk before locking in.

For individual professionals: People with engineering chops can start seriously experimenting with local AI assistants (data never leaves the machine). For most white-collar workers, cloud services remain the lower-friction choice.

For the consumer market: The AI-productivity profile of consumer GPUs is being redefined. A secondhand RTX 3090 is starting to look like the entry ticket for individuals and small studios running local AI.