Local Inference
20 articles tagged with this topic
Qwen 27B hits 50 tok/s: 16GB consumer GPUs can now run large models locally
A Reddit user ran Alibaba's Qwen 27B on a 16GB consumer GPU, hitting 50 tok/s generation + 100k context. Local LLMs are leaving the geek circle behind
$35K AMD DIY Rig Runs Qwen 122B Locally, 2x Faster — Private AI Now PC-Tier
A Reddit user built a $5K AMD-based local AI workstation (Framework Strix Halo + R9700) running Qwen 122B at 2x speed. The real story: workstation-gra
Apple's M5 Ultra Hits 1.2TB/s Bandwidth — Local LLMs Cross the Practical Threshold
Apple's M5 Ultra hits 1.2TB/s memory bandwidth—the first chip making local LLM inference practical, reshaping how knowledge workers handle sensitive d
Qwen 27B Long-Context Speed Drops 65% — Local Players Distrust Benchmarks
AMD's 51.8 tokens/sec Qwen3 27B claim is ideal-only, Reddit user finds; real long-context drops 75→26 (–65%). Local LLMs now judged on stability.
16GB GPU Hits 110 token/s — Local AI Finally Matches Cloud Speed
FlashML's open-source FreeToken tool hits 110 token/s on a Qwen3 35B model using a 16GB consumer GPU. If reproducible, local AI could start rivaling c
Qwen Runs 45 Tokens/Second Locally — Apple Silicon's Silent Win for Open LLMs
Reddit user benchmarked Qwen 3.8B at 45+ tok/s on Apple silicon — a record on consumer hardware. Local LLM inference shifts from geek toy to viable op
Qwen 27B Hits 99 tps on a $200 RTX 3090 — Personal Local AI Arrives
GitHub user syv-ai got Alibaba's Qwen 27B running at 99 tokens/sec on a used RTX 3090 (24GB VRAM, ~$200) — consumer GPUs can now smoothly handle mid-s
llama.cpp Adds Adaptive MTP — One Fewer Knob for Local LLM Users
llama.cpp adds adaptive MTP — the model picks its own token depth. Coding up to 2x faster, prose ~3% slower. A shift from manual tuning to self-tuning
Qwen 27B Hits 19 tok/s on a Laptop — Open-Source LLMs Are Finally Usable Locally
Alibaba's open-source Qwen 27B hits 19 tok/s on a 128GB ROG laptop, autonomously building an HTML flight simulator via Agent mode.
RTX 3090 Hits 82 tok/s on a 27B Model — Time to Retire the 'Cloud-Only' Myth
Dev squeezed a 27B Qwen model into 14GB VRAM on a 2020 RTX 3090 at 82 tokens/sec. Real signal: local AI hardware costs are now directly competing with
Local Coding Agents on RTX 4060 Are Finally Usable — No Cloud Required
Open-source tool runs AI coding on 8GB GPUs; cloud models plan. Near-zero inference—relevant for enterprise IT and indie devs.
5060 Ti Generates 60s AI Songs Locally: Music AI Hardware Barrier Hits Desktop
RTX 5060 Ti 16G + int8 quantized MiniMax Music-3 now generates 60s vocal AI songs locally. AI music's hardware barrier has crashed to desktop.
Mac Local LLM Inference: A Complete Mess, a Full Generation Behind NVIDIA
A Reddit developer tested all major Mac LLM frameworks for two weeks. Verdict: Apple's AI software is fragmented, a generation behind NVIDIA.
LocalLLaMA Posts 'My Turn! Drop It' — The Signal Says More Than the Weights
r/LocalLLaMA (700K-subscriber open-source LLM dev community) sees a nearly title-only "My turn! Drop it!!!" post — the bare-bones gesture itself signa
Qwen3.8 27B Runs 200 Tokens/Sec on a Single GPU, Closing Gap with Cloud APIs
Qwen3.8 27B with NInfer hits ~200 tokens/sec on a consumer RTX 5090. Local AI hardware barriers are falling fast, with real implications for enterpris
RTX 5080 Runs 30B Model in 2 Minutes—Local AI Catches Up to the Cloud
A consumer GPU runs a 30B model 15x faster than the cloud at near-cloud quality. Local AI is becoming a real enterprise option for data-sensitive indu
Kimi K3 Trimmed 33% for English-Only — Local LLMs Are Getting Quietly Affordable
Reddit user hellohazime trimmed Moonshot's Kimi K3 from 711GB to 478GB by stripping multilingual weights. A 2-bit variant reportedly beats its predece
Viral RTX 3090 Refurb Guide: Geeks Fix GPUs for Cheap Local AI Compute
A viral RTX 3090 refurb guide highlights a key trend: tech teams dodge steep cloud bills by using secondhand consumer hardware to run local AI models.
Minimax 2.7 Update Anticipated by Local LLM Community
Reddit's LocalLLaMA community signals anticipation for Minimax 2.7, but details remain sparse.
Best Local LLM for Agentic Coding on a Single RTX 4090
A 4090 owner benchmarks GLM-4.7, Nemotron-30B, and Qwen3-Coder for local agentic coding via llama.cpp.