local-inference
7 articles tagged with this topic
Two DGX Sparks Hit 181 tok/s Concurrent: Local Multi-Agent Is Now Viable
Two NVIDIA DGX Sparks + Qwen models hit 181 tok/s concurrent across 9 parallel AI Agents. Local multi-Agent moves from demo to real work.
Local AI Cuts Basketball Edits From 2 Hours to 30 Minutes
A basketball fan built a free YOLO-based tool cutting 2-3 hours of manual highlight editing to 30 minutes — no cloud AI required.
llama.cpp Fork Turns Retired AMD Server GPUs into AI Inference Rigs
Reddit user milpster and Zhipu AI's GLM team release a llama.cpp fork optimized for AMD GFX906, letting Mi50, Mi60, and Radeon VII run LLMs locally.
Radeon 7600 Runs 35B MoE Model — Local AI's Hardware Barrier Just Dropped
A Reddit user ran Qwen's 35B MoE model at 21 t/s on a ~$280 Radeon 7600 + 64GB DDR4 + Ryzen 5600. Local AI's hardware barrier is dropping fast.
DeepSeek V4 Hits 70 tok/s on a 32GB Laptop — Local LLMs Finally Fit on One Machine
A developer got DeepSeek V4's massive MoE running on a 32GB laptop at 70 tok/s input by resharding weights to SSDs and speculative expert prefetch — p
llama.cpp Adds Audio Processing Support via Gemma-4 E2 A/E4A Models
ll ama-server now supports speech-to-text via Google's Gemma-4 E2A and E4A multimodal models.
Gemma 4 Local CUDA Setup: Precision Traps and Real Benchmarks
Running Gemma 4 locally on CUDA requires strict dtype matching at KV cache boundaries or output degenerates silently.