$807, 5 days, one H100: Unbounded Labs trained Bart, an LLM that only knows pre-1931 English — and it suggests "small and specialized" may be a better bet than "pile on the parameters."
2.8 billion parameters, 20.1 billion tokens of training data (a token is the basic unit of text processing). The project took 3 months and burned $807 (excluding labor). The team also open-sourced the first "vintage model" evaluation benchmark, Vintage CORE.
What this is
Bart's positioning is clear: it's a "time capsule" LLM trained only on pre-1931 English text. The team cleaned and compressed the Harvard institutional library dataset from 242 billion tokens down to 23 billion, then combined it with other historical corpora to reach 20.1 billion tokens of training data.
Even more interesting is the supporting engineering: since no "vintage LLM benchmark" existed, they built their own — Vintage CORE (20 evaluation items) — and also curated and released a 416,000-row annotated Q&A dataset. All code, data, and training logs are open source.
The project draws inspiration from a question posed by Demis Hassabis, founder of DeepMind: can LLMs derive new conclusions the way great scientists throughout history have? "General relativity is definitely not in the budget," the team wrote, "but the direction is worth pushing."
Industry view
Supporters see this as a clean demonstration of the "verticalization route": rather than spending hundreds of millions of dollars to push parameter counts, you use curated data to train a small model for a specific domain. Lower barrier to entry, faster iteration, friendlier to SMBs. The open-source benchmark and data also push the broader small-model research community a step forward.
But we also have to look at the other side. The model Bart benchmarks against, "GPT-1900," is another miniature model — not a real frontier LLM. "Leading" in a niche with virtually no competition deserves a discount. Old English is far too narrow a domain to become a commercial product anytime soon, and the fundamental question Hassabis raised — can LLMs produce original thought — this piece doesn't answer, and can't.
In addition, the team is openly seeking investment and compute donations. The project itself is cheap, but if they want to scale up, where the next round of money comes from is another matter.
Impact on regular people
For enterprise IT: vertical-domain model costs have dropped to the thousand-dollar range, which means enterprises no longer have to put blind faith in general-purpose LLMs — domain-specific models for legal, medical, manufacturing, and other industries may land sooner than expected.
For individual careers: people who hold industry-specific data can now participate in building custom knowledge assistants at a low entry cost, instead of waiting for big companies to release tools.
For consumer markets: no direct impact in the short term — a pre-1931 English model means nothing to consumers. But if the "small and focused" approach proves out, we may see more lightweight AI products targeting specific scenarios in the future, rather than ever-bloated general-purpose assistants.