What this is

This week we noticed in Reddit's LocalLLaMA subreddit a long post from an IT architect who describes himself as "not a developer, not a programmer." He did two things:

First, he ported Ninfer — an inference engine that originally only supported RTX 3090/4090 (software that runs large models efficiently on local hardware, in the same category as llama.cpp and vLLM) — onto the mining card CMP170HX. Mining cards were originally built for crypto mining and have no display output, but since they're "close cousins" of the RTX 3090 (same chip family), the hardware structure is mostly compatible.

Second, he didn't write the code himself — he used Hermes (an AI agent) and a locally running Qwen3.8-27B as the "programmer," with himself only handling architecture guidance and verification.

The result: Qwen3.6-35B on this sub-$1,000 mining card delivers double the performance of an RTX 3090, and still handles a 260,000-character context window (using int8 compression, it occupies 26GB of VRAM).

Industry view

The optimistic voices come mainly from the local AI community: this case proves "cloud GPU isn't the only answer" — mining cards cost 1/5 to 1/10 of professional cards, and combined with open-source inference engines, the cost of local large-model deployment can be significantly lowered. At the same time, "AI agents doing low-level CUDA work" means the technical bar is being flattened by AI itself.

But the objections are worth listening to:

  • This is a one-person experiment with no reproducible benchmark. The llama-swap config screenshots he posted are not a substitute for standardized benchmarks.
  • The hidden costs of mining cards are underestimated: high electricity bills, no display output requiring remote maintenance, poor ecosystem compatibility — ordinary enterprise IT teams can't handle it.
  • The quality of "AI-written CUDA code" is not controllable. Hermes's self-report includes cudaErrorCooperativeLaunchTooLarge error handling — this kind of low-level debugging still cannot do without a human.
  • Cloud vendors have barely responded. AWS and Alibaba Cloud won't cut prices because of this case — their customers are buying "no need to run your own ops."

Our judgment: this is a signal, not a trend. The direction it points to (cheap hardware + AI agents lowering the bar) is real, but there's still a gap between here and "enterprises can actually ditch the cloud."

Impact on regular people

  • For enterprise IT: next time you evaluate local AI deployment options, second-hand mining cards can be on the table. But look at ops costs and stability — don't get pulled in by the "low unit price" alone.
  • For individual careers: non-coders doing low-level technical projects is already happening. If your work involves technical decisions but you don't code yourself, "using AI as a co-pilot" may be more doable than you think.
  • For consumer market: the cost of running large models locally keeps falling, but it's still far from "plug it in and use" — underneath sit software compatibility, VRAM management, model distillation, and a stack of other issues.