We've spotted a set of numbers: across 35 "one-shot" tasks (give one prompt, get a finished output), three versions of Alibaba's Tongyi Qianwen (Qwen) 27B model scored on average 2.46, then 2.74, then 3.00. The improvement between generations is modest, but the direction is stable—open-source large models haven't hit a wall.
What This Is
The test comes from Reddit's LocalLLaMA community, comparing three versions of Qwen 27B (27 billion parameters, runnable on a single consumer GPU) on their ability to produce usable code or imagery from a single prompt.
Specifically: the older versions couldn't draw a passable pelican, the newer one can; older versions couldn't write a playable 2048 game, the newer one can; older versions couldn't render a simple 3D scene in the style of Wolfenstein, and the newer version can roughly reproduce one. Scoring was done by Claude-family models.
These tasks may sound like toys, but one-shot competence is a hard metric for enterprise IT—clients won't wait five rounds of debugging.
Industry View
The supportive view: continued progress at the 27B size shows open-source large models haven't hit a ceiling, Alibaba's iteration cadence is steady, and it's good news for companies unwilling to hand their data over to closed-source APIs.
The critical view also deserves airtime: 35 tasks, scoring one large model with another large model—the sample size and methodology are both rough. Is the jump from 2.46 to 3.00 a real improvement, or noise in the scoring model? Some practitioners privately argue that "using Claude to evaluate Qwen" embeds a systematic bias. The Qwen 3.5 / 3.6 / 3.8 version labels floating around the community aren't Alibaba's official releases either—the rigor deserves a discount.
Impact on Regular People
For enterprise IT: 27B-class models can be deployed on a company's own hardware, keeping data in-house. For finance, healthcare, and manufacturing, that's a more reassuring option than calling a closed-source API in the cloud. This news says "close to usable"—not yet "safe to depend on."
For individual professionals: the one-shot success rate for coding and prototyping work is slowly climbing. You may no longer need to spend an afternoon debugging the AI's output—but don't expect it to replace programmers just yet.
For consumer markets: in the short term, the AI assistant and customer service bot on your phone won't visibly change—most still run on smaller or older models. The lag from a model upgrade to consumer-facing deployment is typically 6–12 months.