This week we noticed a counterintuitive benchmark report from the Reddit LocalLLaMA community: running Alibaba's Qwen3.8 27B open-source model on 4× RTX 3090 delivers 33%–41% lower performance than using just 2×, and the gap widens with concurrency (number of simultaneous requests).
What This Is
The tester deployed Qwen3.8 27B across 4 consumer-grade GPUs, each with 24GB VRAM. The RTX 3090 was NVIDIA's 2020 flagship and remains a workhorse on the second-hand market for SMBs running large models locally.
The key finding: when all 4 cards cooperated on inference — tensor parallelism of 4 (TP=4, meaning multiple GPUs compute the same layer together) — inter-card communication became the bottleneck. In this rig, GPUs within each pair were wired directly via NVLink (a high-speed GPU-to-GPU interconnect), but the two pairs could only communicate over PCIe (the standard motherboard slot, much slower). For every generated token (the smallest unit of text an AI processes; one Chinese character is roughly 1–2 tokens), each layer of the model required one data sync across the two pairs — an overhead that scales with concurrency.
Measured data: starting at 12 concurrent requests, TP=2 (using only one pair of cards) delivered 428–587 tokens/sec, clearly faster than TP=4's 320–418 tokens/sec. Running two TP=2 instances in parallel pushed aggregate throughput to 1,175 tokens/sec.
Two other findings are worth flagging: the model's built-in MTP — a "guess multiple tokens, then verify" acceleration trick — actually slowed things down by 27–42% on the 3090; and prompt processing (the model's "reading the question" stage) is a single-threaded bottleneck that queues up regardless of how many requests you throw at it.
Industry View
We see this benchmark as a direct challenge to the "more cards, more speed" arms-race logic. For enterprise IT teams planning to self-host large models, the takeaway is clear: before stacking GPUs, run nvidia-smi topo -m to see the actual physical topology — don't spec the rig based on vendor talking points.
But the counterargument stands: the test hardware was the Ampere-architecture (NVIDIA's 2020 GPU microarchitecture) 3090. The latest Hopper and Blackwell architectures fully interconnect all GPUs via NVLink Switch, eliminating the inter-card communication bottleneck. MTP acceleration hurt performance on the 3090, but official figures show positive gains on H100 and H200. In other words, the conclusion has a limited scope — it cannot be casually extrapolated to cloud or newer hardware.
Another signal worth watching: open-source large models increasingly deserve serious benchmarking. Qwen3.8 is Alibaba's Tongyi team's flagship open-source release; the fact that the community can stress it on consumer hardware is itself evidence that Chinese large models have reached engineering maturity comparable to Llama and DeepSeek.
What It Means for Regular People
For enterprise IT: Run small-scale benchmarks before procuring AI servers — measured benchmarks beat vendor slides every time. "Double the cards, double the performance" is an illusion.
For individual careers: Hybrid engineers who understand hardware topology, model quantization (compressing models via lower precision), and inference frameworks will command a premium; pure API-wrapper roles will be displaced by tools.
For the consumer market: Alibaba getting a 27B model to run on consumer GPUs means on-device AI on phones and laptops will keep getting stronger — cloud AI subscription prices won't keep climbing indefinitely.