Separating signal from noise in coding evaluations

OpenAI audited the SWE-Bench Pro coding benchmark, finding over a third of tasks flawed due to strict tests, underspecified prompts, low test coverage, or misleading instructions, impacting accurate model evaluation.