What This Is
Alibaba Cloud EMR (big data processing platform) head Li Yu made a judgment call this week at Apache Flink Forward Asia 2026: the next battleground for AI data infrastructure is not models, but whether underlying data can be consolidated into "one copy." We note that most production-grade RAG (letting models retrieve before answering) and Agent (AI that autonomously calls tools to complete tasks) systems today run a "dual-system architecture"—on one side a data lake (a massive warehouse for raw data), on the other a vector database (a specialized store that converts data into mathematical coordinates and retrieves by similarity)—stitched together by a long chain of ETL (scheduled extract-transform-load) pipelines. It works in the early days, but once an Agent does sustained read-write and frequent updates, problems surface: any data change triggers a full index rebuild, online and offline views may diverge, and even cold data has to stay resident on the cluster at full cost.
Industry View
The consensus is that storage, governance, and retrieval layers are converging onto a single copy of the data. The disagreement is over who leads. Databricks expands from the Lakehouse (unified lake-warehouse architecture) side outward, Zilliz (the company behind Milvus) expands from the vector database side outward, while Alibaba Cloud's Paimon × Milvus path emphasizes lake-format neutrality and engine replaceability, sidestepping the touchy "who owns the data" question. But we want to raise a counterpoint: treating the lake format as the unified foundation means Spark, Flink, and retrieval engines must all be deeply coupled, and engineering complexity won't necessarily drop; meanwhile vector retrieval chases extreme low latency, and binding it to lake storage may trade performance for generality—a trade-off that still lacks large-scale production validation.
Impact on Regular People
For enterprise IT: data teams may shift from "operating two systems" back to "managing one copy of data." Governance cost drops but selection complexity rises—the lake-format war over the next two years will directly shape purchasing decisions.
For individual careers: people who understand the full "lakehouse + vector retrieval" stack will become scarce; traditional ETL engineers need to build AI engineering skills, and the premium for narrow specializations is starting to compress.
For consumer markets: infrastructure consolidation will transmit to the application layer, potentially accelerating the rollout of multimodal retrieval (search that simultaneously understands text, images, and video) and long-term-memory AI products—but how much cheaper, and when, remains unclear.