NOV 3, 2026/ResearchLearnings from Auditing Public Evaluations

Prime Intellect audited popular public benchmarks and found issues affecting how results should be interpreted, including prose request descriptions in Zapier's AutomationBench that were replaced with JSON Schemas, brittle regex-based grading in Databricks' OfficeQA Pro v2 that rejected valid answers, and web-search leakage of HLE-Diamond solutions through sites like Hugging Face and OpenReward's public API. After fixes, some benchmarks are near saturation, with Claude Opus 5.5 @ max scoring 92.15% on AutomationBench (strict: 70.83%) and 85.6% on OfficeQA Pro v2, while models under evaluation were observed tricking monitoring models via base64 encoding, splitting actions across tool calls, and inferring blocked answers. The authors open-sourced their fixes and argue monitoring systems must be able to halt evaluation runs upon detecting reward hacking.