vLLM
19 articles tagged with this topic
vLLM Goes Dual-Track: LLM Inference Admits Performance and Portability Don't Mix
vLLM hardware-agnostic layer costs 3.4% throughput, but peak performance and portability can no longer share code. Paradigm shift for self-hosted AI.
Engineer Pushes Qwen to the Limit: 260K Tokens Is Local AI's Hard Ceiling
Engineer pushed Qwen to extremes: context over 100K tokens drops generation 75%. Long-context remains local AI's hard ceiling—proof enterprises can't
Million-Token Context on Two 5090s — Amateur Dev Shatters Enterprise AI Myth
Reddit developer NInfer hits 1.04M token context on consumer RTX 5090s at 119 tok/s — 2.8x faster than vLLM with a 27B Qwen model.
Speculative decoding is becoming standard — open-source LLMs now predict ahead
Reddit users spotted speculative decoding working on local GPUs—AI instantly outputting phrases via MTP. The local inference cost curve is being quiet
How LRU Cache Became LLM Infrastructure's Secret Backbone
LeetCode #146 solution went viral this week. That hashmap+linked-list algorithm underpins LLM KV Cache, vector DBs, and Agent memory—AI's speed/cost b
$6,500 Used GPUs Hit Top AI Speed — Solo Dev Rewrites Inference Economics
GitHub's curvedinf pushed Qwen3 27B from 15 to 972 token/s (65x) on $6,500 of used AMD MI100 GPUs. Enterprise AI's hardware bar just dropped.
100B-AI Training Pipeline Goes Open Source — Under 10 Chinese Teams Can Run It
Engineers stitched K8s, Megatron-LM, and vLLM into a 100B-parameter training pipeline. Side-by-side docs help—but under 10 Chinese teams can run it.
Qwen Runs Faster and Cooler on 3090 — Local LLMs Are Finally Real Tools
Reddit user runs Qwen 27B on two RTX 3090s at 143 tokens/sec, dropping temps from 70°C to 35°C. Local LLMs cross from hobby to usable tool.
Qwen 27B Hits 138 Tokens/Sec on a Single RTX 3090 — Local AI Costs Crater
Qwen 27B hits 138 tokens/sec on one RTX 3090; 2nd-turn latency from 23s to 1s. Not a model breakthrough — open-source is flattening local LLM costs.
Qwen 27B Hits 218 Tokens/sec on Two RTX 3090s — Local AI Cost Curve Drops
Developer hits 218 tokens/sec running Alibaba's 27B Qwen on two consumer RTX 3090s. H100-cluster workloads now run on consumer GPUs. Local AI hardware
RTX 3090 Hits 82 tok/s on a 27B Model — Time to Retire the 'Cloud-Only' Myth
Dev squeezed a 27B Qwen model into 14GB VRAM on a 2020 RTX 3090 at 82 tokens/sec. Real signal: local AI hardware costs are now directly competing with
Two Flags Nearly Double Small Model Throughput — But the Hidden Compatibility Trap Matters More
InclusionAI's Ling-3.0-flash INT4 hits 38.7 tok/s on DGX Spark with two config tweaks — but default vLLM silently breaks V3 architecture, producing fl
vLLM V1 Skews RL Results: Why Inference Correctness Beats Speed
Upgrading vLLM from V0 to V1 causes output inconsistencies in RL. If inference frameworks trade accuracy for speed, dependent models silently drift.
llama.cpp MTP Hits Beta: Local LLM Inference Speed Gap Narrowing
llama.cpp MTP beta supports Qwen3.5. With tensor parallelism maturing, the local-cloud inference speed gap is narrowing, making local LLM deployment m
Qwen 3.6 Wins Benchmarks, Fails Reality: Benchmaxing Distorts AI Perception
Qwen 3.6 won benchmarks but lost to Gemma 4 in practice, burning 8000+ tokens in a loop. Benchmaxing distorts AI perception; firms must shift to real-
Single 3090 Runs Qwen3 Natively on Windows: Local LLMs Drop Linux Requirement
Developers ran Qwen3.6-27B natively on Windows at 72 tok/s. This slashes deployment barriers—enterprises can run LLMs on existing GPUs without Linux.
Local LLMs Lose Tool Call Accuracy After 8–9 Chained Calls
Qwen 32B, Gemma 9B, and Command R 32B all fail similarly after 8+ tool calls — attention dilution, not context limits.
Running Gemma 4 26B-A4B on vLLM: Community Troubleshooting Notes
Developers report mixed results deploying Gemma 4 26B-A4B on vLLM, with INT4 quants too slow on DGX Spark GB10.
Agent Swarms + Continuous Batching Cut LLM Task Time 36x
Running 50 parallel agents on Qwen 27B drops a 42-minute research job to 70 seconds using continuous batching.