What this is

Training a 7B model demands 160GB of VRAM — more than a single top-tier GPU can hold. That single fact turns the AI industry into a big-players-only game.

Where does the number come from? Parameters take 14GB. Backprop needs another 14GB for gradients. The Adam optimizer (the dominant training algorithm) keeps three FP32 buffers per parameter — momentum, variance, and a weight copy — totaling 84GB on their own. Add forward-pass activations (intermediate computation results) and you land at 142–162GB.

Microsoft's DeepSpeed ZeRO and Meta's FSDP (Fully Sharded Data Parallel) are fundamentally about splitting the bill: slicing parameters, gradients, and optimizer states across multiple GPUs. ZeRO's three stages (1/2/3) carve up more and more, and at Stage 3, per-GPU memory drops from 112GB to roughly 14GB on an 8-GPU node.

We see this clearly: it's the entry ticket for the large-model industry. Technical capability is just one card — capital and compute are the harder threshold.

Industry view

The bull case: this infrastructure stack has fueled NVIDIA's seller market and underpins the AI training services of AWS, Azure, Alibaba Cloud, and Tencent Cloud. "VRAM economics" is the main story behind cloud providers' recent earnings beats.

The other side — and the one we care about more: as the training bar rises, independent research teams and mid-sized companies are being pushed off the table. So-called "AI democratization," on the training end, is mostly marketing. Fine-tuning and inference (running a trained model to answer questions) are genuinely accessible — but "training from scratch" remains a game for the few. There's a hidden cost too: running this many cards in parallel means electricity and cooling bills at astronomical scale.

Impact on regular people

For enterprise IT: thinking of deploying your own large model? Get one thing straight — building in-house training capacity starts in the tens of millions. Without scale advantage, use APIs or managed services.

For individual careers: you don't need to understand the tech. The key judgment: AI services aren't "expensive" because vendors are gouging you — the underlying training and inference costs are simply that high.

For consumer markets: understand this and you finally see why phone AI assistants and smart speaker upgrades charge monthly fees. The hardware doesn't make the money — the AI compute bill gets passed straight to the consumer.