llama.cpp
llama.cpp Adds FP8 Cache, Cutting Local Model Operating Costs
llama.cpp now supports FP8 KV-cache, reducing memory use for longer local-model conversations and larger deployments on constrained hardware.
3 articles tagged with this topic
llama.cpp now supports FP8 KV-cache, reducing memory use for longer local-model conversations and larger deployments on constrained hardware.
LingBot-Video 8-step DMD keeps 30B MoE params. FP8 runs faster yet uses more VRAM—verify weights, sampling, and hardware when picking video models.
FP8/BF16 users horrified by IQ1_S 1-bit quantization on r/LocalLLaMA. No middle ground between precision purists and accessibility pragmatists.