AMD
9070 XT Runs 128K Context: Local LLMs Cross the Usability Threshold
A developer achieved about 1,200 prefill tokens per second with 128K context on a 16GB RX 9070 XT using MoE offloading and quantization.
1 article tagged with this topic
A developer achieved about 1,200 prefill tokens per second with 128K context on a 16GB RX 9070 XT using MoE offloading and quantization.