A 27B model has already been loaded into 96GB of GPU memory, yet the service has not necessarily become faster: users may still wait a long time for the first token, and subsequent text may also arrive slowly. Our view is that the central constraint in large-model inference is shifting from “whether it fits” to “whether data can reach the compute units in time.” GPU procurement cannot focus only on card count and peak compute.
What this is
The original article uses public specifications to break down four factors that affect inference speed: memory capacity determines whether the model and request state can fit; memory bandwidth determines how much data can be read per second; compute determines how many operations the compute units can complete per second; and arithmetic intensity indicates how much computation can be completed per byte read (the number of arithmetic operations ÷ the amount of data read).
For a 27B model stored in BF16 precision, the weights occupy roughly 54GB. Two 96GB GPUs may appear to have ample room, but the KV Cache—the attention key-value states retained during model generation—along with temporary workspaces, the runtime, and concurrent requests also consume memory.
In an extremely simplified single-request generation scenario, if generating each token requires reading all 54GB of weights, a GPU with a theoretical bandwidth of 4.0TB/s has a maximum step rate of about 74 per second. This is an instructional example, not a measured result, and it does not account for caching, scheduling, or communication overhead.
Industry view
Our view is that once a model enters production, the bottleneck is usually not a single hardware parameter. It is whether the hardware, data movement, and software scheduling are properly matched. Higher concurrency allows the same set of weights to serve multiple requests, amortizing the cost of reading them. But larger batches also increase pressure on memory, computation, and queues, so concurrency doubling does not automatically mean speed doubling.
Compute claims also require careful comparison. The published peak of 1,979 TFLOPS FP16/BF16 for the NVIDIA H100 SXM includes gains from structured sparsity. Estimated on a consistent dense-compute basis, its peak is about 989.5 TFLOPS. Structured sparsity zeros out selected weights according to fixed rules and requires support from both the model and the software; a peak figure therefore does not equate to production performance.
One point deserves particular attention: performance models such as Roofline are useful for identifying theoretical limits, but they cannot replace real-world load testing. Request length, attention computation, KV Cache behavior, cross-GPU communication, and inference-engine implementation can all change the final result. If a vendor markets only peak compute, memory capacity, or a single tok/s metric, buyers should still ask for time to first token and throughput under defined model, context-length, and concurrency conditions.
Impact on regular people
For enterprise IT: When evaluating AI infrastructure, memory, bandwidth, concurrency, and real-world latency need to appear in the same table. Whether existing compute is sufficient also depends on whether the inference engine can manage memory and batching efficiently.
For individual professionals: Large-model response speed will shape the continuity of everyday use. When selecting tools, watch time to first token, output stability, and performance during peak periods rather than focusing only on parameter count.
For the consumer market: As concurrency improves, prices for AI video, search, and productivity products may fall. But if service bottlenecks remain unresolved, what users will continue to experience is waiting—not simply a gap in model capability.