This week we spotted a local AI signal worth unpacking: a Reddit user ran a quantized 8B model on a 96GB M5 Ultra Mac Studio, hitting a prefill speed of 3191 tokens/sec — 2.2x faster than llama.cpp, another mainstream local-inference framework. But a 107GB download and ¥40,000+ hardware tell us "anyone can run a big model at home" is still far off.
What this is
Original poster Dankpaws took a model called Swift 1.5 (a fine-tune of Alibaba's Qwen3 8B) and converted it into a 4.7 bits-per-parameter MLX-quantized build — a compression format optimized for Apple silicon — specifically tuned for the 96GB M5 Ultra Mac Studio. The model file weighs 107GB and was tested with a 179,000-token context window.
Key benchmarks: on a 25,000-token prompt, prefill hit 3191 tok/s, decode (token-by-token generation) hit 113.7 tok/s, and output matched the original BF16 (16-bit floating point, uncompressed precision) at 91% consistency. Side-by-side against llama.cpp's IQ3_XXS (a more aggressive compression scheme), it's both faster and more accurate.
Industry view
The optimistic read: Apple silicon's AI inference potential is finally being unlocked — 3,000 tok/s means long-document processing is basically seamless. Local LLMs offer data privacy, offline availability, and long-term cost advantages cloud APIs can't match.
The more measured judgment: this is a single-machine community benchmark with no generalizability. 8B parameters counts as "small" by 2025 standards — orders of magnitude behind cloud flagships running hundreds of billions of parameters. A 107GB download is a real barrier for normal users' bandwidth and patience. Most telling: the author said he self-quantized only because "off-the-shelf models weren't good enough" — the local-LLM toolchain (from quantization to deployment) is far from mature. At ¥40,000+ starting price, the M5 Ultra Mac Studio is an "AI power-user club," not "AI democratization."
Impact on regular people
For enterprise IT: in the short term, watch on-premise deployment demand from data-sensitive sectors (finance, healthcare, government). SMB IT teams shouldn't rush in — off-the-shelf cloud APIs still win on price-performance.
For working professionals: white-collar users are better served by ChatGPT, Claude, Kimi, Wenxin Yiyan, and other cloud products. Running a model locally requires hardware, downloads, and full debugging capability — not cost-effective for non-technical roles.
For the consumer market: Apple is positioning the Mac as an "AI creator workstation." Over the next 1-2 years, content creators and indie developers will find local models increasingly smooth. But "everyone runs a big model at home" is still 2-3 years away, at minimum.