Back to home

Compare

Comparing: AMNH Runs 3PB on JuiceFS: Research Storage Shifts to Low-Cost Combos & 美国自然历史博物馆用 JuiceFS 承载 3PB 数据 — 科研存储开始转向低成本组合

AEN
JuiceFSAMNHObject Storage·

AMNH Runs 3PB on JuiceFS: Research Storage Shifts to Low-Cost Combos

The American Museum of Natural History's scientific computing system now carries roughly 3PB of research data, with 95% of traffic belonging to read-intensive workloads, and it has run for four years without any data integrity issues detected. We don't read this as a story about "storage costs being eliminated outright." Rather, it's research infrastructure starting to adopt a more flexible combination: the institution retains ownership of its data while handing off hardware operations to an open storage network provider.

What this is

AMNH works with an Open Storage Network (OSN — a partner that manages local hardware collaboratively and provides remote operations), connecting standard servers and disk arrays to S3-compatible storage, then mounting the object storage as a file system through JuiceFS Community Edition, so compute clusters can access remote data the same way they access ordinary disks.

This architecture serves large-volume reference data such as genome sequences and astronomical survey data: data is written once and then read repeatedly, with little modification. AMNH places data consistency ahead of performance, repeatedly tests during the proof-of-concept phase, and continuously monitors logs and metrics.

Worth noting: write operations against object storage did produce timeout warnings, but no data corruption resulted. JuiceFS handles exceptions through a unified retry mechanism, avoiding part of the risk. As for write-back (writing first to a local cache, then syncing asynchronously), it improves write throughput but raises the requirements placed on local disks and backups.

Industry view

We believe the most valuable part of this case isn't a single piece of software, but its replicability: object storage carries capacity and hardware management, the file system layer provides compatible access, and the research institution still controls the data and the underlying devices. This model is attractive to universities, museums, and labs with limited budgets that still need PB-scale storage.

But the risks are equally clear. S3-compatible does not mean every storage backend behaves identically on timeouts, retries, and consistency; write-back can also widen the window for data loss. AMNH's current workload is predominantly read-only, so this combination cannot be taken as direct proof that it suits write-heavy operations, real-time collaboration, or workloads with strict job scheduling. Remote operations by an OSN provider also mean the institution must define fault response, data migration, and takeover responsibilities if the vendor exits.

Impact on regular people

For enterprise IT: Companies can reassess the cost boundary between self-built storage and cloud object storage, using the file system as a transitional layer — but they must load-test against real read/write ratios and define data consistency SLAs (specific commitments on service availability and data reliability).

For individual careers: Data platform, infrastructure, and research support roles will keep accumulating the combined skill of "storage architecture + reliability validation"; knowing how to configure object storage alone may not be enough to stay competitive.

For the consumer market: There won't be direct product changes for ordinary consumers in the short term, but lower-cost data preservation and compute may progressively underpin larger-scale online model training, scientific retrieval, and generative AI services.

Source: juejin.cn
BZH
JuiceFS美国自然历史博物馆AMNH·

美国自然历史博物馆用 JuiceFS 承载 3PB 数据 — 科研存储开始转向低成本组合

美国自然历史博物馆的科学计算系统如今承载约 3PB 研究数据,且 95% 的流量属于读密集型负载,运行四年未发现数据完整性问题。我们认为,这不是一条“存储成本被彻底消除”的新闻,而是科研基础设施开始采用更灵活的组合:机构保留数据所有权,同时把硬件运维交给开放存储网络服务商。

这是什么

AMNH 与开放存储网络(OSN,合作管理本地硬件并远程运维)合作,将标准服务器和磁盘阵列接入 S3 兼容存储,再通过 JuiceFS 社区版把对象存储挂载为文件系统(让计算集群能够像使用普通磁盘一样访问远程数据)。

这套架构服务基因组序列、天文巡天等大体量参考数据:数据写入后被反复读取,很少修改。AMNH 将数据一致性置于性能之前,并在概念验证阶段反复测试,持续监控日志与指标。

值得一提的是,写入对象存储时曾出现超时警告,但没有造成数据损坏。JuiceFS 通过统一重试机制处理异常,避免了部分风险。至于 write-back(先写入本地缓存、再异步同步的机制),它能改善写入吞吐,却会提高对本地磁盘和备份的要求。

行业怎么看

我们认为,这个案例最有价值的部分不是单一软件,而是“可复制性”:对象存储承担容量与硬件管理,文件系统层提供兼容访问,科研机构仍能控制数据和底层设备。这种模式对预算有限、又需要 PB 级存储的大学、博物馆和实验室有吸引力。

但风险也相当明确。S3 兼容不等于所有存储后端在超时、重试和一致性上完全相同;write-back 还可能扩大数据丢失窗口。AMNH 当前以只读负载为主,不能直接证明这套组合适合写入频繁、实时协作或任务调度严格的业务。OSN 远程运维也意味着机构必须明确故障响应、数据迁移和供应商退出后的接管责任。

对普通人的影响

对企业 IT: 企业可以重新评估自建存储与云端对象存储的成本边界,把文件系统作为过渡层,但要按真实读写比例做压测,并明确数据一致性 SLA(服务可用与数据可靠的具体承诺)。

对个人职场: 数据平台、基础设施和科研支持岗位会继续积累“存储架构+可靠性验证”的复合能力;仅会配置对象存储,竞争力可能不够。

对消费市场: 短期内不会有面向普通消费者的直接产品变化,但更低成本的数据保存与计算能力,可能逐步支撑更大的在线模型训练、科学检索和生成式 AI 服务。

Source: juejin.cn