Quantization
11 articles tagged with this topic
Alibaba Qwen 27B Squeezed to 10GB, Matches Original Quality — Local AI Advances
Austria's ISTA-DASLab squeezed a Qwen 27B to 10GB, matching original quality. Local AI advances — capable, private deployment without cloud uploads.
16GB VRAM Runs 200K Context: Local LLMs Cross a Real Threshold
A Reddit user ran a 27B-parameter Qwen model at 200K-token context on a laptop with an eGPU. Mid-size AI just got much closer to a regular desktop.
Qwen 27B Compressed to 1-2 Bits — Local AI Memory Savings, Quality Wavers
Reddit's jojohai quantized Qwen3.8-27B to Q1-Q2 with MTP baked into weights. Lower memory than external MTP, but the model "gets stupid" off thinking
Qwen 27B Crushed to 1 Bit — Runs on 8GB VRAM, Output Is Brain-Dead
Reddit user crushed Qwen 27B to 1-bit, ran it on an 8GB laptop — output was gibberish. We dig into what this reveals about local AI limits.
Qwen 27B Tests: KV Cache Precision Trumps VRAM for Long-Text Tasks
Reddit LocalLLaMA tests show Qwen 27B with f16 KV cache outperforms q8_0 at 120K-token contexts. Quantization isn't just about saving VRAM.
NVIDIA Compresses 66GB LLM to 22GB, 4x Faster — Inference Cost Story Rewritten
NVIDIA's Nemotron 3.5 Lightning gets NVFP4: 66GB to 22GB, 4x faster, near-lossless. The inference cost story just got rewritten — on-prem AI is now ch
27B Model Squeezed Into 16GB GPU — Local AI Begins Eating Into Cloud APIs
Viral Reddit post: Qwen 27B compressed via IQ4_XS quantization now runs on 16GB consumer GPUs. Local AI economics are being rewritten.
AI at 1/4 Size, 96% Capability — Local AI Cost Tipping Point Is Here
A 3.3GB small model jumped from 28.9 to 69.5 on reasoning via precision allocation — usable local AI may cost less than we thought.
Undergrad's Rented 4090 Beats Six Quantization Teams; Qwen 9B Near-Lossless
An undergrad with a rented 4090 quantized Alibaba's Qwen3.5-9B to near-lossless, beating six established compression teams 31-3-0.
llama.cpp llama-bench Adds -fitc and -fitt Benchmark Flags
llama-bench gains -fitc and -fitt flags from build b4679, enabling finer control over benchmark timing output.
APEX Quantization vs K-Quants: Why MoE Coding Models Need Different Compression
APEX quantization targets MoE architecture coherence layers at Q8, outperforming generic K-quants for multi-file coding agents.