New Artificial Analysis index widens the gap between open and closed models

Artificial Analysis overhauled its Intelligence Index, causing Kimi K3.1 Pro to fall from fourth to ninth place behind eight closed models in version 5, and the gap between open models and the top of the leaderboard widened from 3 points to 12. For the first time, 60 percent of the index relies on private test sets to prevent "benchmaxxing," and Grok 4.7 also lost 12 points. AI researcher Nathan Lambert notes that many training gains now come from reinforcement learning environments resembling benchmarks, while closed test sets mean outsiders can no longer spot errors in prompts, as shown by revisions to the AA-Omniscience knowledge test.