A recent cross-entropy explainer on Lobsters has drawn attention—it's episode two of the "Compression is Intelligence" series. But we believe the video's real point isn't the formula itself, but the research program Sutskever has repeated for years: "compression is intelligence." That thread is the key to understanding why today's large models "seem to think."
What this is
Cross-entropy (a metric measuring how far a model's predictions deviate from the true answer) is a statistics term: it measures how far apart two probability distributions are; the larger the value, the greater the gap. During model training, the more accurate the predictions, the lower the cross-entropy, and the "smarter" the model is considered. The video lays out this technical layer clearly.
But what it really wants to convey is the framework Sutskever has returned to in recent years: if you can compress massive amounts of text well, you've captured the statistical patterns behind language; capturing those patterns means some degree of "understanding." This is also the intellectual starting point behind his later founding of Safe Superintelligence.
Industry view
Supporters argue that this thread explains exactly why scaling works—bigger model capacity means stronger compression, which means higher intelligence. Yann LeCun also partly agrees with "prediction is intelligence," but he insists that language prediction alone is far from enough.
Pushback is equally clear: the equivalence between compression and intelligence has no rigorous mathematical proof to date; it's more an engineer's intuition and metaphor. Symbolic AI researchers like Michael Wooldridge argue that true "understanding" requires causal reasoning and world models—compressing statistical patterns alone is far from sufficient. We believe this debate will run for years; but as a "thinking scaffold" for industry judgment, it remains useful—especially when deciding whether an AI company is doing research or telling stories.
Impact on regular people
For enterprise IT: understanding the upper limit of a large model's "compression ability" is closer to the truth than simply comparing parameter counts—this is a more reliable yardstick for evaluating AI products.
For individual careers: if you're considering a move into AI-related work, foundational courses like this can't be skipped; but beware circles where "knowing a concept is license to lecture on it"—depth matters more than breadth.
For consumer markets: no direct short-term impact on consumers, but this theory is the explanatory framework for why products like ChatGPT "seem to be thinking"—worth keeping in mind when choosing an AI assistant.