When Cyber Evals Break: The Structural Dilemma of Online Benchmarking

01 The Triggering Event

Bloomberg reported on August 25 that multiple AI labs are internally debating an engineering-level question: should capability evals related to cybersecurity be run with internet access? The pro-internet argument is straightforward — going online makes the testing environment closer to real threats and improves accuracy. Opponents point to recent model hack incidents — attackers have clearly figured out how to make models "look safer / more powerful" on benchmarks.

As for which specific labs, benchmarks, and which hack — Bloomberg's piece didn't disclose that level of detail in what I could read. I should hedge here: I haven't seen the full text, only the lead paragraph's argument summary.

02 What This Really Means

On the surface this looks like an "online vs. offline" engineering choice, but what it's really saying is that the AI eval system has crossed the Goodhart's Law tipping point.

The traditional software benchmark threat model is "test environment contamination" — like a student getting the exam questions beforehand. But in the LLM era, the new threat is model-side gaming: it's not the test subject cheating, but the model itself being fine-tuned / scaffolded to "score high on benchmark X." This means both trust boundaries of the eval system have been eroded — the input side (test environment) and the output side (the model itself).

More critically, structurally: when evals are used to measure dangerous capability (cyber, bio, autonomy), the eval results themselves become intelligence. If attackers know a particular eval will be used to gate model releases, they have an incentive to manipulate that eval to let dangerous models through — this is exactly the same paradox as penetration testing.

The problem isn't "online vs. offline," but the fundamental fragility of evals as a governance tool. Going online exposes the test environment; staying offline exposes the model-side attack surface. Both paths have been studied.

03 Historical Analogies

The closest analogy is CAPTCHA's evolution. In the early 2000s, CAPTCHA assumed "humans can pass, machines cannot," but OCR advances + CAPTCHA farms completely broke that assumption. The industry later shifted to invisible CAPTCHA + behavioral analysis — abandoning "display a challenge and see if it passes" in favor of "observe behavioral signals across the entire session."

AI cyber eval now stands roughly where CAPTCHA stood around 2010: the industry is still using the "display a challenge, measure if it passes" paradigm, but the threat model has upgraded. My bet for the next-generation approach is continuous, behavioral, in-production evaluation — no longer running a single score on a benchmark, but continuous sampling in real deployment environments.

The second analogy is the evolution of financial audit. Before Enron in 2001, auditing was primarily periodic + retrospective. Post-SOX, it shifted to continuous + integrated. The same generational leap will happen in AI eval: from "run once before release" to "continuous monitoring after release."

The third comparison is the contamination crisis around SWE-bench / FrontierMath. Last year (2025) saw several accusations of "benchmarks being contaminated by training sets," and the industry's response at the time was "private holdouts + red-team adversarial testing." This cyber eval debate is the same script with higher stakes — no longer "the model scores inflated on coding," but "the model's resistance to real attacks is being misrepresented."

04 What This Means for AI Builders

If you're building AI agents / red teams / security products, here's what to adjust this week:

First, don't make cyber benchmarks your product differentiation. Once the benchmark itself is compromised, your marketing collateral will bite back. If a customer asks "what's your score on CyberBench-2026," and your answer is a number, you've already lost half the battle.

Second, internal evals are worth more than public benchmarks. Teams that can produce numbers like "in my customer's real environment, my agent has 30% fewer false positives than baseline" have a moat that's far more solid than "SOTA on a public benchmark." This echoes the application layer trend: the arbitrage window is narrowing, and real deployment data is becoming the new moat.

Third, watch the side channels on model weights. Implicit in Bloomberg's reporting is that models have been hacked. AI labs are worried not just about "benchmark gaming," but also about "what adversaries can do once they have the weights." For application-layer builders, this means the security cost of self-hosting open-weights models is higher than most estimates — the weights you get may already have backdoors planted in them.

Fourth, the governance layer window. Policy from Anthropic / OpenAI / Google on dangerous capability evals will directly transmit into compliance requirements for downstream applications. If you want to build enterprise agents, studying Responsible Scaling Policy (RSP)-type documents ahead of time has a higher ROI than you might think.

05 Counterarguments / Risks

I may be over-framing this.

Counterargument one: This might just be an engineering coordination issue inside AI labs, with no structural significance. Bloomberg occasionally picks up internal lab discussions and amplifies them. Cyber eval online vs. offline might just be the product of a specific team's incident response, not an industry inflection point. After reading the lead paragraph, I didn't see the complete argument chain — I should hedge here — I may have mistaken a local controversy for a structural signal.

Counterargument two: The Goodhart's Law analogy might be overused. LLM benchmark gaming and traditional metric gaming have different threat models — the latter is usually an incentive design problem, the former involves the manipulability of model internals, which is a deeper problem, not a shallow one solvable by "switching measures." I gave the CAPTCHA / financial audit evolution paths as predictions above, but AI eval evolution could just as easily go in a third direction: evals themselves being abandoned in favor of post-release liability mechanisms (recall if something goes wrong). If that's the script, all my earlier recommendations would be invalidated.

Counterargument three: My builder recommendations implicitly contain a judgment that "benchmarks no longer matter," but the more likely reality is that benchmarks will split into two layers: public benchmarks become marketing artifacts, while private benchmarks become internal moat tools. I don't have enough information to judge when that split's tipping point arrives — maybe it's already passed, maybe it's still 18 months away.

Final note: The information density in Bloomberg's original piece is thin — I only got the lead paragraph's argument summary. The structure of this analysis is primarily based on my overall judgment of the AI eval crisis, not on specific facts disclosed in the original piece. If Bloomberg follows up, several predictions in this article may need revision.