A developer-published benchmark shows that BigMoeOnEdge achieves roughly 35 tokens/s when running Qwen3.6-35B on an 8GB GPU, ahead of Colibri at roughly 23 tokens/s; on a 16GB MacBook, it reaches about 10 tokens/s with a 125B-class model. Our judgment is that it lowers the hardware barrier to running large models locally—on your own devices rather than cloud servers—but the results remain project-specific and cannot yet be treated as a general-purpose solution.
What This Is
BigMoeOnEdge is a loading solution for the llama.cpp inference framework, the open-source tool for running large models on local devices. Its developer says it can load any sparse Mixture-of-Experts model supported by llama.cpp without repackaging or format conversion. MoE, or Mixture of Experts, activates only part of a model’s parameters for each inference pass, reducing computational work.
The tests covered NVIDIA, AMD, Intel, and Mac hardware, as well as phone and desktop CPUs. The developer reports that running Qwen3.6-35B on an 8GB GPU takes about five seconds to generate the first token, compared with nearly two minutes for Colibri. However, on a dual-RTX 5060 Ti configuration optimized for a specific model, Strata remains about 1.3 times faster.
Industry View
The most notable development is that “run any model directly” means developers no longer need to maintain custom compute kernels for every new model, reducing the time cost of deploying new models locally. Automatic adaptation to hardware parameters also makes it easier to move the same quantized model across devices.
The risks are equally clear: these figures come from tests conducted by the project’s author, and no independent reproduction is yet available. Model choice, quantization method, context length, and system settings all affect performance. More importantly, specialized solutions such as Strata still lead on their target hardware, showing that broad compatibility does not equal peak performance. For ordinary users without a GPU, the size of models that can run on a Mac with 16GB of memory or on a phone—and whether their speeds are acceptable—still needs separate validation.
Impact on Regular People
For enterprise IT: Small teams can use a workstation to experiment with internal models and reduce reliance on cloud APIs, but they still need to account for concurrency, latency, maintenance, and compliance costs before production deployment.
For individual professionals: Those with a GPU or a high-memory computer can process sensitive documents locally, reducing the need to upload materials to the cloud. Ordinary PC users, however, may not gain usable office-model performance as a result.
For the consumer market: Lower barriers to running local models could drive more privacy-first AI assistants and compact AI PC products, but we remain a long way from everyone being able to run large models smoothly.