A new benchmark, ARC-AGI-3, developed with a $2 million investment, has starkly revealed the immense gap between the ambitions of "human-like" AI creators and current reality. Leading models, including Gemini 3.1 Pro Preview, GPT 5.4, Opus 4.6, and Grok-4.20, performed dismally, failing to achieve even 1% success rates on the benchmark's challenges. Notably, ordinary humans can complete these tasks with ease and without prior training, while AI agents proved incapable.
The ARC-AGI-3 benchmark presents models with interactive game environments. Within these simulations, AI is expected to explore independently, formulate hypotheses, and plan its actions without explicit instructions. Developers intentionally restricted the models' access to vast knowledge bases and the ability to simply extract patterns from static data. The core objective is to train AI to autonomously understand rules, set goals, and achieve them in dynamic conditions, serving as a test of adaptability and genuine autonomy rather than mere information retrieval.
To provide a more objective assessment of AI's lag behind human capabilities, the metric RHAE (Relative Human Action Efficiency) was introduced. This metric considers not only the final outcome but also the number of steps taken to achieve it. If an AI takes ten times more actions than a human to reach the same goal, its score is automatically reduced to a nominal 1%. This approach effectively devalues brute-force methods and penalizes inefficiency. Furthermore, to ensure rigor, advanced levels were given significant weight, and strict limits were imposed on the number of attempts allowed.
What this means for you, CEO: You should not expect current AI models to perform miracles in tasks requiring flexibility, intuition, and independent decision-making in unpredictable, unstructured environments. It is prudent to focus on narrow, established applications where AI can deliver measurable benefits. The pursuit of artificial general intelligence remains, for now, the domain of visionaries.
Why this matters: Current AI models are not ready for complex, autonomous decision-making in unpredictable scenarios. Focus your AI investments on well-defined tasks where demonstrable value can be achieved now, rather than chasing the elusive goal of general intelligence.