inference-optimization

Speculative decoding is becoming standard — open-source LLMs now predict ahead

Reddit users spotted speculative decoding working on local GPUs—AI instantly outputting phrases via MTP. The local inference cost curve is being quiet