We noticed something worth flagging: DeepSeek announced last week that V4 Flash scored 82.7% on Terminal-Bench 2.1 (a benchmark testing AI's ability to operate command-line interfaces to complete tasks), but using its own undisclosed evaluation harness. Third-party team Ante replicated the result using public tools — 368 successes out of 445 trials, for a 82.7% accuracy rate. The numbers are nearly identical. That fact alone isn't exciting, but it points to a recurring problem: how much of Chinese LLM vendors' benchmark scores can actually be independently verified?
What this is
DeepSeek V4 Flash is a lightweight model targeting code and Agent (AI that autonomously calls tools to complete multi-step tasks) workloads. Terminal-Bench 2.1 measures a model's ability to complete complex tasks in real terminal environments — closer to actual work scenarios than pure Q&A benchmarks. Team Ante routed the same model through OpenRouter (an API gateway aggregating multiple model providers), ran 89 tasks with 5 attempts each, and reproduced the official number.
One detail matters: the "Harness minimal mode" (a stripped-down evaluation script configuration) that DeepSeek originally used has never been open-sourced. This means outsiders cannot tell whether DeepSeek cherry-picked parameter settings favorable to itself. Even if replication succeeded, it only proves the model "can be replicated under public conditions" — not that results match the official configuration exactly.
Industry view
The supporters argue: successful replication itself proves the model's capabilities are solid — 82.7% is a hard metric, not marketing copy. On high-difficulty benchmarks like Terminal-Bench, even top closed-source models (where weights are not public and access is API-only) rarely break 80% consistently.
The skeptics deserve more airtime. The evaluation community has long complained that Chinese vendors "choose their own benchmarking environment" — what prompt template (the instruction format given to the model), how many retry attempts are allowed, whether failed cases get filtered — all of these significantly move the needle. One Hugging Face engineer told us privately: "The numbers are real, but the conditions are ones you don't control. I don't really trust this kind of 'transparency.'" Others note that DeepSeek hasn't disclosed the specific task subset drawn from the 89 tasks; the independent team used the same benchmark but task-sampling details may not match exactly, meaning the 82.7% consistency could involve some luck.
Our editorial judgment: DeepSeek's engineering capability is still top-tier, but on "open and transparent" they still trail Meta and Alibaba (Qwen).
Impact on regular people
For enterprise IT: If you're evaluating plugging DeepSeek into internal systems, V4 Flash's code and Agent capabilities are worth serious testing — this benchmark at least shows it's not a "showpiece model."
For individual professionals: Developers should watch V4 Flash's API pricing and context length (the maximum number of tokens the model can process) — the price-performance ratio may beat calling GPT-4 or Claude directly for scripting tasks.
For consumer market: Ordinary users won't feel much short-term — these models mainly sell to enterprises through APIs (Application Programming Interfaces, the standardized channels that let external software call the model). Consumer products (Doubao, Kimi, etc.) use a different set of distilled models (larger model capabilities compressed into smaller ones); high benchmark scores won't land directly on your phone.