Basalt 0.4.0 runs Qwen 3.8 27B on a single RTX 5090, reaching up to 776 token/s for structured output. The project’s developers also report a maximum consistency rate of 92.8% with the original BF16 model. We believe the barrier to local large models is shifting from “Can it run?” to “Is it fast and stable enough to fit into real workflows?”
What This Is
Basalt is an open-source project for running large models locally. This update introduces quantization for the first dense version of Qwen 3.8 27B. Quantization saves models at lower precision to reduce VRAM usage and improve speed, while a dense model activates all parameters during every inference rather than calling on only a subset, trading some efficiency for more consistent quality and performance.
The new release also adds support for multiple models and automatic routing, with DeepSeek V4.1 Flash among the first supported models, along with chat file and video input capabilities and related fixes. The developers report single-stream conversation speeds of 262 token/s for Qwen 3.8 27B, prose generation at 194 token/s, input prefill speeds of up to 12,000 token/s, and support for contexts of up to 262K tokens.
Industry View
These figures show that high-performance consumer GPUs can already handle some workloads that previously required data centers. For enterprises, privacy-sensitive, latency-sensitive, and relatively stable usage patterns will make local deployment more attractive. Model runtimes will also increasingly become interchangeable infrastructure components, much like databases and office software.
The risks are just as clear. These results come from the project’s own tests and cannot be treated as equivalent to performance across every hardware configuration, language, context length, and real-world task. A 92.8% consistency rate also does not mean the quantized model is identical to the original. Moreover, multi-model routing and video support have only just been added, leaving their real-world stability and maintenance costs to be validated by more third parties. Open-source projects can also be affected by version changes, driver compatibility, and the strength of community maintenance.
Impact on Regular People
For enterprise IT: When handling internal documents, code, or business data, local models may offer lower latency and stronger control over data. However, deployment, access management, and continuous upgrades still require specialized expertise, and costs cannot be estimated from GPU benchmarks alone.
For individual professionals: Local models are poised to become private tools for document organization, writing support, and information retrieval. For now, they are better suited to users willing to configure their environments and verify results than to those seeking a completely frictionless product.
For the consumer market: Faster local inference will drive growth in “offline AI assistants” and small AI devices. Whether they reach the mass market, however, will depend on installation and maintenance, model quality, and the experience across devices.