What this is

Last week we spotted a screenshot: when running Qwen3-32B inference on two cards, half the wall-clock was spent waiting for cross-card communication to sync. This week's open-source project, graphsignal-top, is the tool that puts this "waiting for replies" silent overhead on the table. Once you wrap vLLM, SGLang, and other inference engines (the middleware that deploys a large model as an accessible service) into a graphsignal-run launch, it surfaces in real time: per-GPU utilization, memory, power draw, throttling status, request throughput, and time-to-first-token — plus a human-readable ranking of the most expensive CUDA kernels (the smallest executable compute unit on a GPU) over the last 5 seconds.

What's worth noting — monitoring tools of this kind used to mostly serve training. The "wait cost" on the inference side is being seen this clearly for the first time.

Industry view

On Reddit's r/LocalLLaMA, most developers recognize the tool's value: it lets a team decide on the spot whether to adopt NVLink (a direct card-to-card interconnect with higher bandwidth and lower latency than PCIe), whether to enable CUDA Graph replay, and how coarse the tensor parallelism (splitting one large model across multiple cards to compute in parallel) shard size should be.

But we also hear a sober note: a single screenshot does not yield a general conclusion — the test only covers one configuration: Qwen3-32B, two RTX PRO 6000 cards, and PCIe. Switch to NVLink or a larger card cluster and the bottleneck location may be entirely different. To treat it as a universal "inference cost diagnostic," more benchmark data is required.

Impact on regular people

  • For enterprise IT: if you buy or rent GPUs to run large models, this is a checklist you can pull out immediately to troubleshoot "why aren't my cards fully utilized" — idle cards are burning money.
  • For individual professionals: when negotiating with cloud vendors or evaluating "private deployment vs. calling an API," you can now ask the follow-up: "are the other side's cards spending most of their time waiting on communication?" — one more factor to keep in mind.
  • For the consumer market: when AI apps broadly raise prices or throttle, it may not be that compute is genuinely scarce — it could be that the communication links aren't connected. This will gradually show up in the response-speed and price curves.