RustingStrata
Fitting a 40B Model in 12GB VRAM Still Leaves a High Barrier
RustingStrata ran a quantized 40B model on an RTX 3060 at 64.6 tokens per second, about 33% faster than llama.cpp.
1 article tagged with this topic
RustingStrata ran a quantized 40B model on an RTX 3060 at 64.6 tokens per second, about 33% faster than llama.cpp.