Back to home

Compare

Comparing: NGINX Hits #7 on GitHub Trending: Streaming AI Stalls at the Proxy Layer & NGINX 登上开源热榜第七,流式 AI 的延迟常卡在代理层

AEN
NGINXGitHub TrendingSSE·

NGINX Hits #7 on GitHub Trending: Streaming AI Stalls at the Proxy Layer

NGINX climbed to No. 7 on GitHub Trending's daily chart on November 20, 2026. What deserves attention is that delays or choppy output in streaming AI responses often originate not with the model, but with the reverse proxy between the model and the browser.

What this is

NGINX is a widely used Web server and reverse proxy, commonly deployed between model services and front ends to forward requests, hold data temporarily, and manage connections. Two easily confused mechanisms are involved in its streaming configuration: response buffering collects upstream data before sending it, and disabling it can reduce waiting; backpressure, meanwhile, makes the data source wait when downstream consumers read too slowly, preventing memory usage from growing without bound.

The article also distinguishes reloading, retries, and resumption: reloading a configuration does not migrate existing connections to a new process; failed requests can be redirected to another server, but that does not guarantee the business operation will execute only once. Therefore, “the API can retry” and “the generated result will neither duplicate nor be lost” are not the same thing.

Industry view

We believe this shows that competition over the AI product experience is moving into the infrastructure layer. When model output speeds are similar, whether the proxy forwards data promptly, how slow clients exert backpressure, and how failed connections are handled may influence perceived performance more than adding more model parameters.

Disabling buffering, however, is not a universal solution. The impact on a slow client is transmitted more directly upstream, tying up connections and computing resources at the model service; the proxy still needs working memory to absorb data chunks. Nor does this mean every model token corresponds to one network send. Uniformly switching every API to “no buffering” may simply trade display latency for capacity pressure.

The risks are equally clear. Allowing applications to control the proxy through response headers makes troubleshooting harder, while retaining old long-lived connections after a configuration reload can lead teams to mistakenly believe a new policy is already in effect. If enterprises monitor only front-end time to first content and fail to track upstream connections, memory usage, and failed retries, they may mistake a localized improvement for a system-wide optimization.

Impact on regular people

For enterprise IT: When deploying AI applications, teams should include the proxy in end-to-end call-chain observability and separately record model generation, proxy buffering, and browser rendering times rather than attributing all latency to the model.

For individual professionals: When evaluating AI tools, users should pay attention to whether long answers continue streaming and whether content is regenerated after an error. These details often reveal service stability better than the speed of a single response.

For the consumer market: Users will increasingly struggle to distinguish “the model is slow” from “the network is stalling.” If companies promote only model capabilities while neglecting the delivery chain, even a smooth experience may remain behind the competition.

Source: juejin.cn
BZH
NGINXGitHub TrendingSSE·

NGINX 登上开源热榜第七,流式 AI 的延迟常卡在代理层

NGINX 在 2026 年 11 月 20 日登上 GitHub Trending 日榜第 7 位。值得关注的是,流式 AI 回答出现延迟或成段跳出,问题常常不在模型,而在模型与浏览器之间的反向代理。

这是什么

NGINX 是广泛使用的 Web 服务器和反向代理,常被放在模型服务与前端之间,负责转发请求、暂存数据和控制连接。它的流式配置涉及两个容易混淆的机制:响应缓冲会先收取上游数据再发送,关闭后可减少等待;背压则是下游读取太慢时,向数据来源施加等待,避免内存被无限堆积。

文章还区分了重载、重试与续跑:重新加载配置不会把旧连接迁移到新进程;失败请求可以改投另一台服务器,但无法保证业务只执行一次。因此,“接口能重试”和“生成结果不会重复或丢失”不是同一回事。

行业怎么看

我们认为,这说明 AI 产品的体验竞争正在进入基础设施层。模型输出速度相近时,代理是否及时转发、慢客户端如何反压、连接失败后如何处理,可能比继续增加模型参数更能决定用户感知。

但关闭缓冲并非通解。慢客户端的影响会更直接地传回上游,占用模型服务的连接与计算资源;代理仍需工作内存承接数据块,也不意味着一个模型词元对应一次网络发送。把所有接口统一改成“不缓冲”,可能只是把显示延迟换成容量压力。

风险同样明确。允许应用通过响应头控制代理,会增加排障难度;配置重载后沿用旧长连接,还可能让团队误判新策略已经生效。企业若只看前端首屏时间,而不监测上游连接、内存和失败重试,仍可能把局部改善当成系统优化。

对普通人的影响

对企业 IT:部署 AI 应用时,应把代理纳入完整调用链观测,分别记录模型生成、代理暂存和浏览器呈现时间,而不是把延迟全部归因于模型。

对个人职场:评估 AI 工具时,可关注长回答是否持续输出、异常后是否重复生成;这些细节往往比单次回答速度更能反映服务稳定性。

对消费市场:未来用户会越来越少区分“模型慢”和“网络卡”,企业若只宣传模型能力,却忽略传输链路,流畅体验仍可能落后。

Source: juejin.cn