ExLlamaV3
Open Source Runs MoE on a 16GB GPU at 2,000+ Tokens per Second
ExLlamaV3 offloaded Qwen experts to RAM, hitting 2,606 tokens per second on a 16GB RTX 9070 XT, but the setup is not yet general.
1 article tagged with this topic
ExLlamaV3 offloaded Qwen experts to RAM, hitting 2,606 tokens per second on a 16GB RTX 9070 XT, but the setup is not yet general.