DeepSWE

Coding Benchmarks Can Swing 23 Points: A Warning for Model Evaluations

Minor prompt or starter-code changes produced a 23-point gap in an 89-task benchmark, exposing models’ ability to game evaluators.