Questions about Continual-ARC.
Short answers to what people ask about Continual-ARC, each drawn from the paper. The paper has the whole argument and every number.
What is Continual-ARC?
Continual-ARC is a benchmark for continual learning: it measures whether an AI system gets better at its work with experience. Continual-ARC serves a long stream of puzzles from the three editions of ARC-AGI. The puzzles need no knowledge of any field. Each ARC task is a hidden rule for turning one grid of colored cells into another, and each instance of a task is a new grid to apply the rule to. On the stream, the same tasks recur after gaps, each time with a new instance, while new tasks keep arriving. After the stream, the system stops learning and takes two tests. The memorization test serves new instances of the tasks the system met on the stream. The generalization test serves tasks the system never met. The benchmark is described in the paper Continual-ARC: Measuring Skill Acquisition over a Working Lifetime, by Daniel Lenton, Yusha Arif and Haris Mahmood.
What does continual learning mean for an AI agent?
Continual learning means an agent gets better at its job with experience, as a person does. The Continual-ARC paper splits that improvement into two kinds. Memorization is whether a skill the agent acquired on one job is still there the next time a similar job comes up. Generalization is whether the experience of past jobs makes the agent better at jobs it has never met.
Is there a benchmark that tests whether an AI agent gets better with experience?
Yes. Continual-ARC was built to test whether an AI agent gets better with experience. Continual-ARC serves the agent a long stream of puzzles in which the same puzzle tasks recur, each time with a new grid to solve. On each puzzle, the agent can pay for help: each worked example it asks for costs one point, and each wrong answer costs one point. An agent that fails a puzzle is charged the whole budget of 16 points. The score of each puzzle is the share of the 16-point budget the agent spent, so an agent that has learned from earlier puzzles spends less. To separate learning from what the agent could already do, the benchmark also runs a wiped copy of the agent: the same agent, with everything it has kept wiped before every puzzle. What the agent learned is measured by how much less the agent spends than the wiped copy spends. Agents that learn by keeping notes and agents that learn by training their weights face the same stream and the same tests.
How do you evaluate whether an agent learns on the job?
Continual-ARC evaluates learning on the job in four steps. First, the benchmark serves the agent a stream of tasks that recur, each time with a new input, so a remembered answer never fits. Second, the benchmark charges the agent for every worked example it asks for and every wrong answer, so an agent that has learned spends less. Third, the benchmark runs the same agent again with everything the agent kept wiped before every task, and compares the two runs. Fourth, the benchmark stops the agent learning and gives it two tests. The memorization test serves new inputs for the tasks the agent met on the stream. The generalization test serves tasks the agent never met. An agent that stores one program per task spends less on the memorization test but no less on the generalization test. An agent that has become a better solver of new tasks spends less on both tests.
How is Continual-ARC different from ARC-AGI?
ARC-AGI measures how well a system acquires a skill within a single episode: the system meets each task once, works out the task's rule from a few worked examples, and answers. Continual-ARC turns ARC-AGI's tasks into a long stream in which each task recurs after a gap, each time with a new input. The stream enables Continual-ARC to ask two questions that ARC-AGI does not ask. The first question is whether a skill the system acquired is still there when its task recurs. The second question is whether experience with some tasks makes the system better at tasks it has never met. Continual-ARC has a track for each of the three ARC-AGI editions. Continual-ARC is not affiliated with ARC Prize.
Does an agent's memory in its harness count as learning?
On Continual-ARC, mostly not. An agent's harness is the software around its model that runs the agent, and some harnesses keep notes between tasks as the agent's memory. The Continual-ARC paper found that keeping notes in the harness generally does not make an agent better at tasks it has met before, unless the harness is given explicit expert guidance. On every model the paper tested, even with expert guidance, most harness-based agents learned less than a simple baseline that only stores each task's worked examples and replays them. No harness-based agent got better at tasks it had never met.
Should an agent learn by updating its weights, or by keeping notes in its harness?
On Continual-ARC, no agent that learned by keeping notes in its harness got better at tasks it had never met. The agent whose weights were trained did get substantially better at those unseen tasks. In the Continual-ARC paper, that trained agent was Qwen3.5-9B, whose weights were trained to imitate GLM-5.3 Flash. The paper takes the trained agent's gains as opening the door to weight-based continual learning that could teach models to drive harnesses in new domains.
Can I run Continual-ARC on my own agent or harness?
Yes. The benchmark's code, the paper's agents and harness adapters, and the records of every run are open source under the Apache-2.0 license. The Run yours page shows how to run your own agent or harness on the public stream. To have a result verified, send your agent's code with the Verify my learner form, and unify reruns your agent on a private set of new tasks that nobody outside unify has seen. Any verified submission of a genuine learning algorithm goes on the leaderboard.
How should I cite Continual-ARC?
Cite the paper Continual-ARC: Measuring Skill Acquisition over a Working Lifetime, by Daniel Lenton, Yusha Arif and Haris Mahmood (arXiv:2611.14505), rather than this site or the code. The paper's BibTeX entry is at the foot of the Continual-ARC overview page.