Last week an old client asked me: "Xiaolin, can you run my contract draft through local AI? I don't want my client list uploaded to foreign servers." I said sure. Half an hour later, Ollama spat out something that read like a high schooler padding their word count — I wanted to crawl into a hole on the spot.
What it is + who's using it
Running a large model locally means installing AI on your own machine — data never leaves. Common free tools: Ollama, LM Studio, Jan. Pick one. My friend Chen jie (a therapist in Beijing, started 2024) uses it to process anonymized session notes; A Liang, a designer in Shenzhen, runs light tasks on it while drafting logos for clients.
But if yours feels "dumber than ChatGPT," it's almost always one of these three switches:
- Quantization level: The more the model file is compressed, the smaller and faster — but also "dumber." Q4 is faster than Q8 but dumber; Q8 is slower than FP16. Most people sit stuck at Q4 and wonder why nothing works.
- Context window: Default might be 2044 (how many tokens you can feed it). You toss in a 5,000-character contract and it "zones out." Bumping to 4096 or 8192 usually makes it smarter instantly.
- Prompt format: Every model has its own preferred chat template (Llama 3, Qwen, Mistral all differ). Use the wrong one and it starts babbling nonsense.
Replication cost today
Money: $0 – $420 (0 – 3000 RMB). Mac M1 and up is basically free (Ollama just runs); Windows users want at least an RTX 3060 12GB — about $280 used.
Time: 30 minutes – 2 hours. 10 min installing software, 20 min tuning three switches, 1 hour iterating on results. I got stuck here for an entire afternoon before I figured it out.
Technical barrier: You can install software. No code needed.
First step: Open Ollama or LM Studio → find "Context Length" → change from default 2044 to 4096 → ask the same question again. You'll feel the difference immediately.
Advice by stage
If you're just starting / no clients yet: Don't bother with local yet. Free ChatGPT, Claude, Gemini will carry you six months. Get the business running first, then optimize cost. I spent three months fiddling with local before sheepishly going back to ChatGPT for my main drafts.
If you have 1-2 clients / worried about data: Local AI is worth trying. Run Qwen 2.5 7B (most Chinese-friendly) at 4096 context in Ollama, then toss sensitive files in. That's how I cleaned up that contract draft.
If you're scaling / 5+ people: Just buy ChatGPT/Claude Enterprise (with data isolation guarantees), or deploy on your company intranet server. Local AI fits individuals and small teams — maintenance gets expensive fast with more people.