43%. In a test comprising 202 real GitHub tasks, Qwen3.8 27B solved 43% of them, while Qwen3.6 27B solved just 17%. We note that these results reflect a reality more closely than conventional coding leaderboards: whether a model is genuinely usable often depends on how it performs in enterprises’ less common languages and legacy systems.
What this is
The testing team selected 13 languages that rarely appear in mainstream coding benchmarks, including Elixir, OCaml, Haskell, Clojure, Kotlin, Swift, Julia, and R, then evaluated the models using tasks from real GitHub repositories. Qwen3.8 27B was benchmarked against Qwen3.6 27B, and the results can also be compared horizontally with Kimi K3, DeepSeek, GLM, and closed-source models.
These tests assess a model’s ability to locate problems within existing codebases, make changes, and deliver results—not merely write isolated code snippets. For teams using local deployments, they offer a more relevant reference for day-to-day work than aggregate benchmark scores.
Industry view
We believe the rise from 17% to 43% is significant: it suggests that smaller models may be rapidly improving at handling code in specific technology stacks. The test also enables comparison with closed-source models, showing that competition has shifted from parameter count toward completion rates and deployment costs in concrete use cases.
The risks are equally clear. Qwen3.8 participated in task selection, introducing some bias into the results; project size, test design, and compute budget can also affect the conclusions. A 43% solve rate also means that more than half the tasks remained incomplete, so the model cannot be considered ready to replace engineers outright. More importantly, a community test is not a rigorous, unified industry benchmark.
Impact on regular people
For enterprise IT: Local models may become more valuable for niche languages, legacy code maintenance, and privacy-sensitive projects. Enterprises can begin by assigning models verifiable work such as test coverage and defect identification, then decide whether to expand their use.
For individual professionals: Developers working in less mainstream languages may gain access to capable local coding assistants sooner. Model selection should still be based on performance in real repositories, hardware costs, and error rates—not a single benchmark score.
For the consumer market: As local models gain support for more development tools, privacy-first applications that do not require ongoing subscriptions will increase. At this stage, however, their capability gains remain concentrated in verifiable technical tasks.