What This Is
A 10-person US startup publicly shared its local AI hardware configuration this week: 4 AMD R9700 AI Pro GPUs (128GB total VRAM), paired with a Threadripper 9970X processor and 128GB of DDR5 ECC memory — the full system costing $18,000.
The machine runs a fork of the open-source inference engine Radiance, simultaneously serving 16 employees on open-source models including Qwen 3.8 Next Flash 27B. In a 48K context scenario, aggregate decoding speed hits roughly 538 tokens/sec (tokens being the smallest unit of text large models process), with a single user topping out at 80 tokens/sec. Short version: adequate, not slow.
Industry View
The bullish take argues this marks local inference entering the "small companies can afford it" phase. Hardware setups that only big tech or cloud providers could previously justify are now compressed into workstation-class server territory. The AMD R9700 is pitched for inference, not training, priced well below the Nvidia H100, and is being positioned as a viable alternative to cloud APIs.
But the counterarguments deserve airtime. First, the $18,000 entry threshold is still steep for many 10-person outfits, and hidden costs — power, cooling, platform ops — aren't in that number. Second, AMD's ROCm software ecosystem (analogous to Nvidia's CUDA development platform) still trails CUDA, and model compatibility regularly hits snags. Third, if a company's monthly AI call volume is modest, pay-as-you-go cloud pricing can actually come out cheaper. This setup isn't a universally better answer.
Another detail that caught our editorial eye: they're running Qwen 3.8 — an open-source model released by Alibaba. A US team using a Chinese open-source model as a core business engine is a signal that Qwen's engineering maturity has cleared the bar.
Impact on Regular People
For enterprise IT: In industries with strict data compliance mandates (finance, healthcare, government), local deployment has shifted from "theoretically viable" to "engineering-viable" — a real option on the table.
For individual careers: Over the next 12–24 months, expect more mid-sized companies to stand up similar clusters internally, with employees using in-house local models instead of external ChatGPT for sensitive data.
For consumer markets: Individual users can't build this stack in the short term, but as open-source models get validated by enterprises, the capability will trickle down into more consumer apps — local AI capability is the trend.