A knowledge base with over a million document chunks takes several hours to rebuild from scratch and requires double the disk space. The most counter-intuitive judgment here: the cost peak of RAG deployment never falls on the day the library is built—it falls on every document update that follows.

What is this

RAG (Retrieval-Augmented Generation) is a technical approach that lets large models "scan through" a company's own document repository before answering questions. Policy documents, product manuals, customer service scripts—all can be fed in.

The problem is: documents aren't static. Parameters change, policies get deprecated, products go offline—but the "index" in the AI knowledge base (think of it as a library's card catalog) doesn't update itself. It still looks the way it did on the day the library was built.

So there are two paths:

  • Full rebuild: every document gets reprocessed. The logic is simple, but once the library scales up, machines run at full capacity for hours, and you still need double the disk space.
  • Incremental update: only process the changed portions. Fast, but you have to solve problems like "was the old version fully deleted?" and "does the new version conflict with existing content?" yourself.

The author's plain judgment: for small libraries (a few hundred internal documents), scheduled full rebuilds are the least hassle; for large libraries (millions of document chunks), incremental updates are mandatory, but you need three things in place—stable chunk IDs, a document-to-chunk mapping table, and batch writes.

Industry View

Proponents argue that incremental updates are the only path to scale. Customer service systems add or remove thousands of products daily; compliance libraries batch-update regulations quarterly. In these scenarios, full rebuilds are neither realistic nor cost-effective. The key is to build the document-to-chunk mapping table properly—the old version must be fully deleted. Otherwise, retrieval will surface both old and new content simultaneously, and the model has no way to judge which is current. In customer service, finance, and compliance contexts, the cost is real.

But the dissenting voice deserves equal attention. A senior architect points out: many teams underestimate the cost of "switching Embedding models." Embedding is the tool that converts text into vectors. Once you switch, the existing and new vectors exist in different semantic spaces, and all historical data must be rebuilt from scratch—there is no second option. The cost of that model-selection decision is far higher than imagined.

There's another overlooked complexity: BM25 keyword indexes (traditional retrieval based on word frequency) and vector indexes are two different systems, and knowledge graphs (networks that structure entity relationships) get layered on top. In a serious RAG system, there may be three to four systems that need to update in sync with documents, each with its own update logic.

Impact on Regular People

For enterprise IT departments: when evaluating AI knowledge base projects, don't just look at "how much to build the library"—calculate "how many times documents are updated per year × compute cost per update." The maintenance bill could be several times the build bill.

For individual careers: the "maintenance role" for internal enterprise knowledge bases will become concrete and scarce—it's neither a pure algorithm engineer nor a traditional document administrator. It's a new hybrid position.

For consumer markets: when using AI customer service to check orders or ask about policies, if the response is off-topic or contains outdated information, don't rush to blame the agent—it's very likely that the upstream knowledge base update hasn't kept pace.