ExLlamaV3
Open Source Runs MoE on a 16GB GPU at 2,000+ Tokens per Second
ExLlamaV3 offloaded Qwen experts to RAM, hitting 2,606 tokens per second on a 16GB RTX 9070 XT, but the setup is not yet general.
4 articles tagged with this topic
ExLlamaV3 offloaded Qwen experts to RAM, hitting 2,606 tokens per second on a 16GB RTX 9070 XT, but the setup is not yet general.
A developer achieved about 1,200 prefill tokens per second with 128K context on a 16GB RX 9070 XT using MoE offloading and quantization.
Tests across 35 weight quantizations, 3 engines, and 64K-token contexts show 8-bit KV cache halves memory with little quality loss, while 4-bit drifts
Two 30B MoE models now run on a 128GB AMD Strix Halo mini PC at 30-44 tok/s. Models that once needed multiple H100s now fit a $1,400 desktop.