An audio-coding benchmark has just pushed its top score to 127, while DeepSeek V4.1 Flash managed only 81. What matters is the shift: LLM evaluation is moving beyond “can it produce the expected answer” toward whether it can independently research, validate, and propose better solutions in an ambiguous task.
The benchmark requires models to read compressed audio in an unknown format, reverse-engineer the format with Python and NumPy—the fundamental library for numerical computation—and write a decoder. It must then create an encoder that improves quality or compression efficiency without reducing the signal-to-noise ratio, a measure of audio distortion. Models cannot call ready-made tools such as FFmpeg or Opus, or import an existing codec library.
What this is
This is an “evolving” open-ended evaluation with four difficulty levels. A model can begin only with compressed audio; if it fails, the system progressively supplies the original audio, format documentation, and callable tools. The score is capped at 100, but a new encoder that beats the current benchmark after successful decoding can earn bonus points.
The key change is that once a model completes the first stage and produces a better encoder, that encoder becomes the benchmark for the next round. The leaderboard therefore does not quickly lose its ability to distinguish models as their capabilities improve, while continuing to expose weaknesses in reverse engineering, algorithm implementation, and experimental validation.
Claude Fable Latest currently leads the leaderboard with 127 points, followed by multiple quantized versions of DeepSeek V4.1 Flash and Qwen3.8-Flash-Next. In the first round, DeepSeek cracked the task in 13 minutes and its result became the second-stage benchmark. The third-stage benchmark came from the latest encoder generated by Claude Fable.
Industry view
We believe this design is closer to enterprise R&D than fixed questions. Engineers rarely face clearly defined specifications; they work with incomplete inputs, conflicting constraints, and outcomes that require repeated testing. If a model can propose an encoder better than the existing solution, that demonstrates more research capability than simply passing a test.
The risks are equally clear. The benchmark is currently maintained by an individual and relies on manual runs and manual publication of results. Model configurations, hardware environments, and quantized versions can all produce differences. Although the repository remains private to limit direct exposure of the task, it is still impossible to rule out answers entering training data through other channels. More importantly, audio compression is only one task. Whether a single score improvement represents general engineering ability requires more problem types and repeated validation.
Impact on regular people
For enterprise IT: This evaluation can complement model-selection processes by helping companies observe performance on real R&D tasks. It should not, however, determine procurement on its own; teams should rerun the evaluation against their own codebases, access controls, and cost metrics.
For careers: Competition in AI skills is shifting from “answering like an expert” toward “breaking down an unfamiliar problem and delivering a usable result.” In the near term, people who excel at defining problems, validating outputs, and explaining tradeoffs will find it easier to amplify the value of AI models.
For the consumer market: If models continue to advance compression algorithms, storage and transmission costs for voice assistants, real-time translation, and low-bandwidth services could fall. For now, however, this is more a technical signal than a stable, reproducible product advantage.