This week, an unassuming experiment on the LocalLLaMA forum caught our editorial eye: someone fed the same 330-line HTML/JS code snippet to both Qwen 3-5B and Gemma 2-6B — two open-source models with comparable parameter counts — and the tokenization (splitting text into the smallest units a model can understand) results were 1,609 and 4,258 respectively, a nearly 2.6x gap.
What this is
At its core, this reveals that the two models' tokenizers (the component that decides how a model breaks human text into recognizable chunks) took fundamentally different paths from day one. Qwen's tokenizer, when facing code, clearly defaults to a code-aware mode with chunkier cuts, treating whole structural blocks as single units. Gemma, by contrast, uses an approach close to natural-language tokenization, breaking code apart like ordinary English. The result: the same code looks like 1,609 meaningful "building blocks" to Qwen, but 4,258 fragmented tokens to Gemma.
This directly affects two things: how efficiently the context window gets used, and how the model "perceives" code structure. Finer granularity isn't necessarily bad — but when the same code is split into 2.6x more tokens, Gemma is forced to process significantly more noise during both understanding and generation.
Industry view
Plenty of evidence supports this observation. The community has long held an impression: Qwen series has a stable reputation for leading on coding tasks, while Gemma is more of a "language model that's good at chatting." One counterpoint comes from teams like LiquidAI, which are working on retraining more efficient tokenizers for existing models — if they succeed, Gemma's weakness could be patched. But developers on the other side point out that the tokenizer is only the starting point; what ultimately determines a model's ceiling is the pretraining data distribution and post-training (the fine-tuning and alignment stage after a model's base training is complete) strategy. Swapping in a new tokenizer won't necessarily let Gemma catch up to Qwen's coding ability overnight. Our view: the tokenizer difference is a necessary but not sufficient condition — but it has been a chronically underestimated variable.
Impact on regular people
For enterprise IT: If you're choosing open-source models for code assistants or internal tools, Qwen's coding strength isn't just "better training" — tokenizer efficiency also means lower inference costs and more room in the context window.
For working professionals: Regular employees using AI to write code don't need to understand tokenizers, but they should remember this: feeding the same prompt to different models can produce wildly different outputs. That's not because your prompt is bad — the underlying mechanisms are simply different.
For the consumer market: Chinese model companies are putting Qwen forward as their "open-source flagship," reflecting Chinese teams' grip on engineering details. This is worth noting for decision-makers in traditional industries: open source isn't charity — it's competition.