AI company Heidi Health, which processes 2.4 million clinical conversations per week, this week cut the GPUs running its speech recognition from 16 to 4, saving 75% of compute—the latest signal that the AI industry's center of gravity is shifting from models to inference cost.

What this is

Heidi is a clinical-scribing AI company processing 2.4 million clinician-patient conversations weekly across 190 countries. Its speech-to-text (ASR, automatic speech recognition) pipeline uses NVIDIA's fine-tuned Parakeet TDT 0.6B V2 model running on L40S GPUs. A single recognition request uses only 15–20% of GPU compute; the remaining 80% sits idle. The default time-slicing approach queues multiple requests to share the GPU in rotation, with extra overhead from context switching. AWS and NVIDIA's solution: MPS (Multi-Process Service)—letting one GPU serve multiple processes simultaneously, paired with the Triton Inference Server for request scheduling. At the same concurrency level, GPU count dropped from 16 to 4, with average latency under 650 ms and the 99th percentile under 1 second.

Industry view

AWS and NVIDIA stated it plainly in a joint blog: this is an engineering win, not an algorithmic one. Its significance: when model capabilities homogenize, whoever cuts inference cost wins a few extra margin points on the contract. But we also see the limits: MPS shares the GPU across processes at the cost of weaker isolation—one stalled process can ripple into others. In stability-sensitive domains like healthcare, deployments need more complex fault-tolerance design; the savings get partially offset by engineering complexity. The bigger trend: the industry's center of gravity is shifting from "can we train good models" to "can we run them cheaply." We judge that over the next 2–3 years, optimizations in infrastructure layers—inference frameworks, GPU scheduling, quantization, and compression—will more directly affect AI project ROI (return on investment) than model progress itself.

Impact on regular people

  • For enterprise IT: AI compute budgets can be more aggressive—but only if the team has people who understand GPU scheduling and inference optimization. Procurement alone won't solve this.
  • For individual careers: engineers skilled in Triton, quantization, and deployment are commanding a clear premium. Traditional backend developers pivoting to AI inference roles is one of today's highest cost-of-effort transition paths.
  • For consumer markets: end prices for AI customer service, AI healthcare, AI meeting notes, and similar products may ease as inference costs drop.