The same matrix multiplication — the most compute-hungry operation in any AI model — can run tens to over a hundred times faster in an engineer's carefully tuned version than in one knocked out casually. Same algorithm, dramatically different efficiency. We've watched the large-model race move past "whose model is smarter" into a new phase: it's now about who can run AI cheaper and faster. That edge depends on a group of engineers who almost never make the headlines — operator engineers (the people who build the basic compute units of neural networks) and AI Infra engineers (the people who build the systems that actually run the models).
What this is
Think of it this way: an AI model is the recipe, operators are the "chop, stir, sauté" of cooking, and AI Infra is the kitchen, the serving system, and the supply chain. Researchers design the recipe; operator and Infra engineers make sure the kitchen actually runs.
What really decides how fast AI runs is "how you schedule the work." FlashAttention, the famous technique, is a textbook example. It didn't change the model's math — it simply reordered the computation so data stays inside the GPU's on-chip cache as much as possible, avoiding repeated reads and writes to VRAM. Result: training accelerated several-fold, VRAM usage dropped dramatically. The approach is called operator fusion — combining multiple compute steps into one.
Operator authors come in four camps: hardware vendors (Nvidia's cuBLAS and cuDNN libraries), framework developers (PyTorch ships with thousands of built-in operators), specialist kernel engineers (hand-writing and tuning in CUDA, Triton), and AI compilers (torch.compile, TVM, and others — aiming to automate the process).
Industry view
Optimists argue this lane is exploding: bigger models, more inference, and whoever drives per-call cost the lowest wins pricing power. Nvidia's moat isn't just its GPU silicon — it's decades of accumulated operator libraries. That's the hardest thing for latecomers to replicate.
But we see an underestimated risk: operator engineering is brutally empirical, unglamorous grind. The same logic, written differently, can perform a hundred times worse. That kind of craft only comes from repeatedly polishing on specific hardware. The biggest bottleneck for domestic AI chips right now is "operator adaptation" — no matter how good the chip's specs look on paper, if the hundreds or thousands of operators aren't tuned for it, mainstream large models either won't run or won't run fast.
A more sober voice also exists: once critical optimizations like FlashAttention become widespread, the marginal gains of the next generation keep shrinking. What decides the next winner may no longer be the operator layer, but data, algorithms, and application scenarios.
Impact on regular people
Enterprise IT: When procuring AI services, don't stare only at model benchmark scores. Look at requests per second, cost per inference, and latency in milliseconds — these numbers are set by the underlying infra.
Individual careers: Operator and Infra engineers are an underestimated branch of technical roles, in long-term undersupply. Application-layer practitioners who grasp a bit of the underlying logic will communicate far more smoothly with algorithm teams.
Consumer market: AI products will keep getting cheaper. Today an AI conversation may cost a few cents; in three to five years that may compress to a fraction of a cent. The biggest hero behind cost reduction isn't the model — it's these invisible "kitchen renovations."