The open-source project RustingStrata has enabled a 249GB DeepSeek-V4.1-Flash model to run on a single RTX 3060 12GB GPU with 30GB of memory, generating roughly 2–3.6 tokens per second. The result matters: the barrier to running extremely large models locally is moving from professional servers toward consumer hardware.
What this is
RustingStrata is a local model inference tool written in Rust. It does not load the entire model into VRAM. Instead, it uses tiered storage: frequently used expert parameters are placed in an approximately 9GB VRAM cache, while the remaining parameters are read sequentially from memory and SSD. The GPU handles operations such as attention computation. A token is the basic unit into which a model breaks down text for processing.
This approach depends on the operating system's page cache, which temporarily holds frequently accessed data in memory. Because the model is far larger than the available 30GB of RAM, SSD read speed and cache-hit rates will materially affect output speed. The project also currently supports only greedy decoding, meaning it selects the highest-probability token at every step. Tool calling and multiple sampling methods have not yet been completed, and the context limit is 4,096 tokens.
Industry view
This shows that local deployment need not be limited to model compression or purchasing high-VRAM servers. For organizations that value data control, offline use and fixed costs, “tiered model storage plus on-demand loading” may provide an additional option, particularly for infrequent calls, batch tasks and use cases tied to specific devices.
The risks are equally clear. At 2–3.6 tokens per second, performance remains insufficient for most everyday interactions, and page caching can cause substantial speed fluctuations between runs. The project produced results consistent with mainstream llama.cpp in 189 of 192 cases for short prompts and 21 of 23 cases for long prompts, but these are specific tests and do not establish that the model's capabilities remain unchanged. The absence of sampling, tool calling and stable serving capabilities also means that it is currently more of a technical proof of concept than an enterprise solution ready for procurement.
Impact on regular people
For enterprise IT: The value of keeping data on-device is rising, and technically constrained teams with limited budgets can experiment with running selected models locally. Production deployments must still account for SSD reads, concurrency and maintenance costs.
For individual professionals: Sensitive materials can be processed locally on a personal computer, reducing the need to upload them to the cloud. However, current performance is closer to an interactive demonstration than a high-frequency writing or programming assistant.
For the consumer market: Beyond large-model vendors, open-source tools are bringing inference capabilities to ordinary graphics cards. Hardware makers may need to reconsider how VRAM capacity and software coordination shape the user experience.