A multi-user inference server built around 4 Radeon AI PRO R9700 GPUs, with a total cost of about C$13,000, delivered roughly 80 to 225 output tokens per second while serving 8 users with 128K-token contexts. These figures matter because the barrier to deploying AI locally is moving from “an experiment for a few people” toward “a solution small teams can evaluate.”
What this is
The poster assembled the inference server with an AMD EPYC 7402P, 128GB of memory, and 4 R9700 GPUs, then used Radiance software to run Qwen 3.8 Flash Next. The model was quantized with a mix of FP8 and 4-bit precision, primarily to reduce GPU and memory overhead.
Testing covered contexts from 32K to 128K tokens and 1 to 8 concurrent users. Prefill speed was approximately 6,800 tokens per second. For ordinary Agent follow-up responses—where the model continues generating an answer within an existing context—time to first token was about 0.2 to 2.5 seconds. But when 8 users simultaneously submitted new 128K-token requests, the first token took about 2.5 minutes to arrive.
Industry view
We believe these results send a pragmatic signal: small and medium-sized teams can build local inference services with consumer-grade or professional Radeon GPUs without buying expensive dedicated AI accelerators. For sensitive-data industries such as healthcare, legal services, and finance, “data never leaves the internal network” is no longer merely a principle; it can also become a quantifiable cost proposition.
That said, this is still an initial test on a single machine, the 4 graphics cards are not from the same vendor, and the power and cooling setup required custom work. Critics would point out that latency in the most extreme long-context concurrent scenario reached 2.5 minutes, which remains far from the ideal for interactive services. If a business depends on continuous conversational interaction, this configuration may not be sufficient. The more prudent approach is to validate software compatibility, long-term stability, and total cost of ownership together.
Impact on regular people
For enterprise IT: More sensitive material can remain inside the enterprise, and AI assistants may expand beyond public-cloud APIs into local deployments. Before procurement, teams still need to account for the full cost of GPUs, memory, operations, and model tuning.
For individual careers: More professionals will need to understand server and model deployment, while companies may create roles responsible for local AI environments. Beyond assembling the hardware, however, stability, monitoring, and security still require specialist expertise.
For the consumer market: This kind of setup will not immediately become standard in ordinary PCs, but it will continue to lower the barrier for enterprises using local models. More software and services will first deploy models on customers’ own devices.