FreeToken
FreeToken Beats llama.cpp 7x on VRAM Overflow, Loses 3x When Model Fits
Reddit benchmark: llama.cpp is 2-3x faster when models fit in RTX 3090 VRAM; FreeToken is 7x faster when a 120B model overflows at 32 users.
2 articles tagged with this topic
Reddit benchmark: llama.cpp is 2-3x faster when models fit in RTX 3090 VRAM; FreeToken is 7x faster when a 120B model overflows at 32 users.
FlashML's open-source FreeToken tool hits 110 token/s on a Qwen3 35B model using a 16GB consumer GPU. If reproducible, local AI could start rivaling c