An open-source project called FreeToken has sparked discussion in Reddit's local-LLM community: a user running a Qwen3-series 35B model (about 20GB) on a consumer-grade RTX 5080 with 16GB of VRAM hit roughly 110 token/s, using a mix of VRAM and system memory to fit the weights.
What this is
FreeToken is open-sourced by the FlashML team. Local inference — running AI models on your own device instead of calling a cloud API — is nothing new; llama.cpp, vLLM, and MLX have all been pushing it forward. But traditional paths usually demanded datacenter cards like the A100 or H100, or delivered sluggish speeds on small-VRAM consumer hardware.
What sets FreeToken apart: it is specifically optimized for NVFP4 (NVIDIA's 4-bit quantization format) and aggressive memory offloading, letting a 16GB consumer card hit triple-digit token/s. Conversations that previously required paying OpenAI per token may now run at comparable speed on an ordinary gaming desktop.
Industry view
Supporters frame this as a watershed moment for local AI: if the numbers hold up, it is especially valuable for data-sensitive industries — healthcare, legal, consulting — where customer data no longer has to leave the premises.
We would counsel caution. This is a single data point from a Reddit user, with no independent third-party reproduction. NVFP4 is an NVIDIA-proprietary format, so AMD and Intel GPUs are out of luck. The "35B-A3B" model designation is also confusing — "A3B" usually denotes a Mixture-of-Experts architecture where active parameters are far smaller than 35B, and high throughput would not be surprising. Declaring that "the cost story has been rewritten" is premature.
Impact on regular people
For enterprise IT: We would not recommend replacing production cloud APIs in the short term, but a small-scale pilot that moves non-sensitive internal AI workflows on-prem is worth running.
For individual professionals: Practitioners in data-sensitive fields — lawyers, accountants, consultants — are seeing rising feasibility for handling client material locally. Worth tracking closely.
For the consumer market: Consumer GPU vendors and AI-PC makers (Lenovo, ASUS) will likely seize on tools like this as marketing ammunition. "The laptop that runs large models" is shaping up to be its own product category.