Qwen
Qwen 8B Hits 400 Tokens/Sec — Self-Hosted AI Cost Inflection Point
Qwen3-8B-Flash hits 400 tokens/sec on an RTX 6000; 8K prefilling nears 14K. The self-hosted AI affordability threshold is arriving for enterprises.
Back to home
local-inference
10 articles tagged with this topic
Splash
Splash 1.1: Apple Silicon Now Runs 27B AI Models Smoothly
Splash 1.1.0 runs 27B models locally on top-spec Macs at ~50 tokens/sec. Local AI inference becomes practical—enterprises can skip cloud fees.
Zhipu
256GB Memory Finally Enough for Local AI—But M5 Ultra Hits GPU Bottleneck
Reddit M5 Ultra user tested Zhipu's GLM-5.3-Flash: 256GB RAM handles local AI, but GPU is now the bottleneck—the memory wall is breaking.
Qwen
Two DGX Sparks Hit 181 tok/s Concurrent: Local Multi-Agent Is Now Viable
Two NVIDIA DGX Sparks + Qwen models hit 181 tok/s concurrent across 9 parallel AI Agents. Local multi-Agent moves from demo to real work.
YOLO
Local AI Cuts Basketball Edits From 2 Hours to 30 Minutes
A basketball fan built a free YOLO-based tool cutting 2-3 hours of manual highlight editing to 30 minutes — no cloud AI required.
llama.cpp
llama.cpp Fork Turns Retired AMD Server GPUs into AI Inference Rigs
Reddit user milpster and Zhipu AI's GLM team release a llama.cpp fork optimized for AMD GFX906, letting Mi50, Mi60, and Radeon VII run LLMs locally.
Qwen
Radeon 7600 Runs 35B MoE Model — Local AI's Hardware Barrier Just Dropped
A Reddit user ran Qwen's 35B MoE model at 21 t/s on a ~$280 Radeon 7600 + 64GB DDR4 + Ryzen 5600. Local AI's hardware barrier is dropping fast.
DeepSeek
DeepSeek V4 Hits 70 tok/s on a 32GB Laptop — Local LLMs Finally Fit on One Machine
A developer got DeepSeek V4's massive MoE running on a 32GB laptop at 70 tok/s input by resharding weights to SSDs and speculative expert prefetch — p
llama .cpp
llama.cpp Adds Audio Processing Support via Gemma-4 E2 A/E4A Models
ll ama-server now supports speech-to-text via Google's Gemma-4 E2A and E4A multimodal models.
Apr 123 min readopcnew Editorialwww.reddit.com
Gemma 4
Gemma 4 Local CUDA Setup: Precision Traps and Real Benchmarks
Running Gemma 4 locally on CUDA requires strict dtype matching at KV cache boundaries or output degenerates silently.
Apr 72 min readopcnew Editorialreddit.com