ARC-AGI Benchmark: The Test Built to Measure Real Reasoning

Most AI benchmarks eventually fall. A new model arrives, the score jumps, the test is declared saturated, and everyone moves on. The ARC-AGI benchmark was built to be the exception. Created by François Chollet in 2019, it measures something models are supposed to be worst at: solving genuinely novel puzzles they could not have memorized. For five years it resisted every large language model thrown at it. Then in late 2024 one system finally cracked the first version, and within months a harder second version pushed the machines right back to the floor. This is the story of what ARC-AGI tests, why it keeps humbling frontier AI, and what the scores really say about progress toward general intelligence.

What the ARC-AGI benchmark actually tests

Chollet introduced ARC-AGI in his 2019 paper "On the Measure of Intelligence," which argued that most AI evaluation measures the wrong thing. A model that has read the whole internet can answer almost any factual question, but that is memory, not intelligence. Chollet defined intelligence instead as skill acquisition efficiency: how quickly a system learns to do something it has never seen before. The benchmark is designed to isolate exactly that ability.

Each ARC-AGI task is a small visual puzzle made of colored grids. You are shown a handful of example pairs, each an input grid next to its transformed output grid. Your job is to figure out the hidden rule from those few examples, then apply it to a fresh input to produce the correct output. The rules involve things like symmetry, counting, filling shapes, moving objects, or completing patterns. Critically, every task uses a different rule, so there is nothing to memorize across tasks. A model that solved a thousand of these gets no head start on the next one.

The other deliberate constraint is what Chollet calls core knowledge priors. The puzzles rely only on cognitive basics that humans have very early in life: objectness, basic geometry, counting, simple cause and effect. They do not require language, cultural knowledge, or education. That is the point. By stripping out crystallized knowledge, ARC-AGI tries to measure raw reasoning rather than accumulated facts, which is why the ARC Prize team describes the tasks as easy for humans yet hard for AI. The full explanation lives on the ARC Prize site.

A concrete example makes the format click. Imagine three example pairs where each input grid contains a few scattered colored squares and each output grid has those same squares but with a line drawn connecting them. From just those three pairs a human infers the rule, connect the marked cells, then applies it to a new grid they have never seen. Another task might show that every blue shape in the input becomes red in the output while everything else stays put. The rules are never stated in words. You have to induce them from the examples alone, which is the entire challenge.

How the ARC-AGI benchmark is structured and scored

The benchmark is not a single pile of puzzles. It is split into several sets that keep the scoring honest. There is a public training set that anyone can study to understand the style of tasks, a public evaluation set for open testing, plus a hidden private set that produces the official numbers. Keeping a private set sealed is what stops a team from simply overfitting to the questions, a constant risk with public benchmarks where the answers leak into training data over time.

Scoring itself is strict but fair. A solver gets two attempts on each task, so the metric is pass at two: the task counts as solved if either attempt produces the exact correct output grid. There is no partial credit for getting a grid mostly right. Every cell has to match. The Kaggle competition that runs the ARC Prize adds another safeguard by running submissions in a sealed environment with no internet access, which blocks a model from quietly calling out to a giant hosted system or looking up an answer. That closed setup is part of why the competition scores tend to sit below the flashier numbers a lab can post when it runs a frontier model with unlimited resources.

This structure explains a subtlety that trips people up. You will sometimes see two very different ARC-AGI scores for what sounds like the same model. One may come from an unrestricted run on the public set, the other from the sealed competition on the private set. The gap between them is not a mistake. It is the difference between a controlled measurement and a best case demo.

Why the ARC-AGI benchmark is easy for humans but hard for machines

Show an ARC-AGI puzzle to almost any adult and they solve it in seconds, often without being able to explain how. That intuitive leap, seeing two examples and instantly grasping the rule, is the human superpower the benchmark targets. It is the same skill a person uses when they pick up an unfamiliar board game from watching one round.

Language models struggle because their default strategy is pattern completion over their training data, and these puzzles were built to have no useful pattern in that data. For years this produced a striking gap. Through 2020 to 2024, general purpose models scored close to zero on ARC-AGI while humans breezed through it. The benchmark became a favorite argument for skeptics who said that scaling up next token prediction, however impressive on other tests, was not producing the fluid reasoning that defines general intelligence. A model could ace a bar exam and still fail a puzzle a seven year old finds obvious.

The o3 moment that broke ARC-AGI-1

The long stalemate ended in December 2024. OpenAI's o3, a reasoning model that spends heavy compute thinking through a problem before answering, posted a step change on the first version of the benchmark. In a low compute configuration it scored 75.7%, and in a high compute configuration it reached 87.5% on the held out evaluation set. For a test where models had scored in the low single digits for years, this was a genuine breakthrough, and the ARC Prize team said so publicly.

The result mattered because 87.5% sits in the range the competition treats as human level. The organizers use roughly 85% as a human bar, based on studies where individual people solve well over half the tasks on their own while a panel of humans covers almost all of them. o3 had, for the first time, matched that territory on ARC-AGI-1. The skeptics' favorite counterexample had fallen.

There was a catch, and it became the theme of everything that followed. o3's high score came at enormous compute cost per task. Getting to human level accuracy on these puzzles required the model to burn far more money and time than a human needs. So the machine had matched human accuracy while being wildly less efficient at reaching it, and efficiency was the whole thing Chollet set out to measure. The benchmark had been beaten on the score, but arguably not on the deeper question it was asking.

How researchers try to beat the ARC-AGI benchmark

The methods teams use to attack ARC-AGI reveal a lot about why it is hard. Two broad families dominate. The first is program synthesis: rather than asking a model to output the answer grid directly, the system searches for a small program, often written in a custom domain specific language of grid operations, that transforms every example input into its correct output. If a program reproduces all the demonstration pairs, it is applied to the test input. This approach treats each puzzle as a tiny coding problem, and for years hand built search programs outperformed neural networks on the benchmark.

The second family is test time training. Here the system briefly fine tunes itself on the handful of example pairs for the specific task in front of it, adapting its weights to that one puzzle before answering. The winning entries in the 2024 competition leaned heavily on this idea, and it works because it forces the model to actually learn from the examples rather than pattern match against training memory. o3's breakthrough combined deep reasoning with sampling many candidate solutions and selecting among them, which is powerful but expensive, since generating and checking many candidates per puzzle is what drives the compute cost so high. Each of these routes chips at the problem from a different angle, and the fact that none has cracked ARC-AGI-2 cheaply is itself a strong signal about how far current methods are from human style learning.

ARC-AGI-2 resets the ARC-AGI benchmark gap

In March 2025 the ARC Prize team released ARC-AGI-2, and it moved the goalposts in a principled way. The second version keeps the same grid puzzle format but is deliberately built to defeat the strategies that let o3 through. It emphasizes tasks that resist brute force search and shallow pattern matching, the tactics a heavy compute reasoning model leans on. The design details are laid out in the ARC-AGI-2 paper by Chollet and colleagues.

The effect on the leaderboard was stark. The same frontier reasoning systems that had conquered version one dropped back to low single digit scores on ARC-AGI-2. Humans, meanwhile, stayed strong. The team calibrated the set so that a panel of people solved every task, with individual humans still performing far above the models. The gap that o3 had appeared to close reopened almost completely, which was exactly the intent. A durable benchmark should get harder as fast as models get better.

ARC-AGI-2 also promoted cost to a first class metric. A score no longer counts on its own. The benchmark reports accuracy alongside cost per task, so a system that only reaches a number by spending unlimited compute is scored honestly against one that reaches it cheaply. This directly answers the o3 critique. On ARC-AGI-2 you cannot buy your way to a high score and call it intelligence, because the price is on the scoreboard next to the accuracy.

The efficiency angle is easy to underrate, so it is worth dwelling on. A human solving one of these puzzles uses a trivial amount of energy and a few seconds of thought. A frontier model reaching the same answer may run through vast chains of reasoning tokens, sample dozens of candidate solutions, then verify each one, at a cost that can run to dollars per single puzzle. If the goal is a system that learns like a person, matching human accuracy at a thousand times the cost is a partial result at best. Chollet's framing insists that intelligence is about doing more with less, so a benchmark that ignored cost would let raw spending masquerade as reasoning. Putting price on the leaderboard forces the field to confront the part of the problem that scaling compute does not automatically fix.

The ARC Prize 2025 and its unclaimed jackpot

To push open research at the new target, the ARC Prize returned in 2025 as a public competition with $1 million in prizes, run on Kaggle. The headline grand prize, worth around $700,000, was set for any team that could reach 85% accuracy on ARC-AGI-2 at a cost under $0.42 per task. That single line captures the whole philosophy: high accuracy is not enough, it has to be efficient too.

The competition drew serious interest, with the ARC Prize 2025 technical report documenting 1,455 teams and more than 15,000 entries. Yet nobody came close to the grand prize threshold. The top public score on the ARC-AGI-2 private evaluation set reached just 24%, far below the 85% human level bar and far below the cost efficiency the prize demanded. The million dollar jackpot went largely unclaimed, and the full breakdown is in the ARC Prize 2025 report. For anyone tracking hype about imminent general intelligence, a field of thousands of teams topping out at 24% on a test humans find easy is a useful reality check.

ARC-AGI-3 makes the puzzles interactive

The ARC Prize team is not waiting for version two to be solved before building the next challenge. In 2025 they previewed ARC-AGI-3, which changes the format in a way that breaks AI's remaining crutch. Where the earlier versions are static, you see examples then answer, ARC-AGI-3 is interactive. The solver, human or machine, acts inside a small game like environment, observes what happens, forms a theory of the underlying rule, then tests it through more actions before committing.

This targets the exact weakness that pattern completion cannot cover. A fixed context window of examples is one thing. Learning a rule by experimenting, remembering what you tried, then adjusting, is another. Early reporting on the interactive benchmark found humans solving it at essentially 100%, while frontier models including strong reasoning systems failed almost everything, scoring under 1%. If ARC-AGI-2 reopened the gap, ARC-AGI-3 is designed to keep it open even as static reasoning improves.

What the ARC-AGI benchmark tells us about progress

The value of ARC-AGI is not that it produces a tidy score to brag about. It is that it keeps asking the one question the industry most wants to skip past: can these systems handle genuine novelty as efficiently as a person, or are they extraordinary memorizers wearing the costume of reasoning? The honest answer in 2026 is somewhere in between, and the benchmark's trajectory shows it. o3 proved that with enough compute a model can climb a reasoning ladder that once looked closed. ARC-AGI-2 and the unclaimed prize proved that efficient, cheap, general novelty solving is still out of reach.

That is why ARC-AGI is worth watching more than the benchmarks that saturate every few months. It is engineered to stay hard, to reward the right thing, and to embarrass premature declarations of victory. If you want a companion piece on how the other dominant test is graded, our explainer on the SWE-bench Verified coding benchmark covers a very different measurement problem. Read together, they map the two poles of how we judge modern AI: one on real software work it may have seen before, the other on pure novelty it definitely has not. When a single system finally tops both cheaply, the conversation about general intelligence will have earned a very different tone. Until then, the ARC-AGI benchmark remains the most useful reminder that a high score and real understanding are not yet the same thing.