Local Player's New Approach: Context as Scratchpad, Storage Outside the Model

What This Is

This is a post from a local player on Reddit's r/LocalLLaMA: he uses a 32GB VRAM home PC as his main machine, plus an older machine running a 9B-parameter (parameters are the "brain capacity" unit of a model) Qwen small model as a "memory curator." The core idea: the context window (the length of text a model can "see" at once) is not working memory, but scratch paper for each inference step. The actual working state is written into external documents, continuously compressed and organized by the small model, with Git used for version rollback.

The total cost is one consumer-grade GPU (around ¥15,000 for an RTX 5090 or a used 4090), not an enterprise-grade 8x H100 cluster.

Industry View

This player's approach is essentially the "poor man's version" of big-company Agent (AI that autonomously plans multi-step tasks) architectures — treating the context window as RAM and external documents as disk. Both Anthropic's and OpenAI's Agent products already use a layered "short-term context + long-term memory" design.

But the objections are clear. Context compression inevitably loses information. One AI engineer pointed out in the comments that a 9B model acting as "memory curator" is likely to grind down key details during compression; Git seems to allow rollback, but retrieval costs will explode as documents grow. More critically, the big labs are charging toward "million-token context" (a token is the smallest unit a model processes, roughly 0.7 Chinese characters; Google Gemini 1.5 already supports 1 million tokens), betting that "the larger the context, the more the model seems to think" — the exact opposite of this local player's approach.

Our judgment: this path isn't wrong, but it suits individuals and small teams. Once scale increases, the maintenance cost of an external memory system will almost certainly exceed just buying a larger context.

Impact on Regular People

For enterprise IT: no need to chase million-token flagship models right away — try running a layered setup with existing GPU clusters + small models first, then decide whether to upgrade hardware.

For working professionals: when using ChatGPT or Claude to process long documents daily, learn to feed "the current task" and "background material" separately, instead of dumping everything into the dialog box — this is a simplified version of this player's approach.

For the consumer market: 32GB VRAM-class hardware may become the new entry threshold for local AI players, and demand for DIY rigs and AI mini-PCs will gradually pick up.