Spectra-Global dropped a number last week: swapping AdamW for FFT cuts VRAM for finetuning 8B to 70B models by nearly half, but each training step is slower. This is an open, non-peer-reviewed experiment on the r/LocalLLaMA Reddit community, and the poster is actively offering free Colab invites for reproduction.
What this is
The biggest bottleneck in LLM finetuning (secondary training of a pretrained model on specific data) is GPU VRAM — the massive gradient states stored inside the AdamW optimizer (the algorithm controlling how the model learns) consume huge amounts. Spectra-Global's approach: convert these gradient states to the frequency domain, discard low-energy "noise," then compress them back.
Sounds clever, but we noticed that experienced Reddit users quickly raised several questions:
- No training loss curves or convergence rates provided — just "VRAM halved."
- No comparison against established approaches like quantization (lower precision to save VRAM) or gradient checkpointing (skip storing intermediate results to save VRAM).
- FFT isn't new in signal processing, but applying it to optimizer state with genuine theoretical guarantees remains unverified.
Industry view
Supporters' logic: existing VRAM compression methods each have side effects — quantization can damage training, checkpointing slows things down — so frequency-domain compression, if it preserves training direction, is genuinely worth exploring.
But skepticism and reservations dominate. Developers on r/LocalLLaMA widely caution: a single-point claim is not a result. The poster proactively offering "free Colab + baseline weights" can be read either as open-source goodwill or as self-promotion without peer review.
The most critical, overlooked point: they haven't disclosed how the trained model performs. If VRAM is halved but model capability drops, the method has zero commercial value.
Impact on regular people
For enterprise IT: if the method holds up, the most direct beneficiaries are mid-sized companies wanting to finetune LLMs on industry data but can't afford A100/H100s — consumer GPUs will be able to do more.
For individual careers: regular employees don't need to worry about this yet — FFT and AdamW are engineer concerns. But this reaffirms a trend: AI training costs keep falling fast, and the "compute barrier" continues to loosen.
For consumer markets: indirect tailwind. The trajectory of falling training costs hasn't changed — custom enterprise AI models will keep getting cheaper. What costs 500K today might be 50K in three years.