Continual-ARCby unify

A continual learning benchmark · 114 runs

Continual-ARC

Measuring Skill Acquisition over a Working Lifetime

Daniel Lenton

See the resultsRead the postCode

Abstract

Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale.

We introduce Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with retries permitted on incorrect submissions, which are paid for. Each instance is scored based on the share of this feedback budget consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization.

We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. We also define a second regime in which one system learns from several users simultaneously.

Persistence in the harness generally does not improve memorization unless provided with explicit expert guidance. Even with this guidance, most harness-based methods still learn less than a floor that simply caches each task's demonstrations, across all models we tested. No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.

Solve rate, from the reset to the learner6075409025981560804095259915memorization: solve rate on the seen testgeneralization: solve rate on the held-out testQQAQEQTQTEQETEQPQDEQDE(2)QDE(4)QCtold the pageGGAGEGPGCAcross, the seen test, on a complementary log-log scale: click for a linear scalelogUp, the held-out test, on a complementary log-log scale: click for a linear scalelog
Solve rate increase, against the reset×1×2×4×1×2×4×0.5memorization: solve rate increase on the seen testgeneralization: solve rate increase on the held-out testQQAQEQTQTEQETEQPQDEQDE(2)QDE(4)QCtold the pageGGAGEGPGCAcross, the seen test, on a log scale: click for a linear scalelogUp, the held-out test, on a log scale: click for a linear scalelog

Bars: 95% simultaneous intervals within each comparison family, hidden where narrower than their point. Readout intervals are pointwise 95%. Points outside the main comparison have no bars.

Point at a learner, or tap it, to light it in both graphs and read its numbers here. Drag a box across a graph to zoom in, and click it to zoom out.

Q Qwen3.5-9B (student) plays in the harness (Prime Agent)G GLM-5.3 Flash (teacher) plays in it

T teacher writes the notesE expert guidance, for the letter before itD DAgger, imitates the teacher

P program library, in place of the harnessC demo cache, in place of the harness(n) n concurrent usersA asked to store

Figure 1. Every learner that keeps something between instances, frozen after the stream, at one user and DAgger also at two and four, whose users share one model and bring puzzles of their own. A square is a harness asked to store, which the paper's Figure 3 leaves to its appendix. The open models' left graphs are on complementary log-log axes, as the paper's are. Every learner's numbers are on the leaderboard.

Introduction

Learning over a working lifetime

Chollet (2019)[1] argued that intelligence should be measured by how efficiently a system turns priors and experience into skill, on tasks that lie largely outside its experience and have a verifiable answer. ARC is the concrete instance of that argument, and on its current edition the gap between what a person solves in a minute and what a model solves from the same demonstrations is still significant[2]. For agent systems that continually operate for days and weeks, a more pressing question is whether a skill acquired on one job is still there the next time a similar scenario is met, which we call memorization, and whether that experience makes the system better at similar jobs it has never met, which we call generalization. By design, ARC measures acquisition once, within a single episode, and does not ask either over longer horizons. An overview of our final results is shown in Figure 1 above, and its terms are explained below, from the benchmark to the learners, with every learner's numbers on the leaderboard.

The benchmarks that do ask it have three limitations.

  1. Each is built around one applied domain, such as coding or office work[3][4][5], in which the models under test already know a great deal from the pretraining data.
  2. Most serve a fixed set of instances, each seen once, so the same conceptual task never returns with a fresh input. This means a memorized answer cannot be told apart from a more abstract learned skill.
  3. Most assume a single family of learner, gradient updates for fine-tuning suites[6] and prompting and memory for agent benchmarks[7][8], so memory-based and weight-based systems are rarely compared on the same scale.

Continual-ARC is a lifelong stream built from ARC, where tasks need no knowledge of any domain beyond core-knowledge priors, such as objectness and counting[1], and the rule of each grid task has to be induced from a few demonstrations. A task can return after a dormant gap as often as the stream asks, each time with a new input that no cached answer fits. New tasks also keep arriving throughout the stream. Because a system may carry anything between instances, whether weights or notes, prompting, memory and weight updates all face the same stream and the same tests.

The benchmark

From one instance to a stream and two tests

One instance

An ARC task is a rule that turns an input grid into an output grid, and an instance of it is one input whose output the learner must produce. For each instance, the learner observes the task id, the input grid, and how many attempts and demonstration requests it has used on this instance, but it is told nothing about its position in the stream. It may submit a grid, and the environment answers only correct or incorrect. A correct submission ends the instance, and an incorrect one costs 1. It may instead request a demonstration, at a cost of 1, which returns one more input-output pair of the task, up to eight per instance. Failing to get the correct answer costs 16, even if there were demonstration requests to spare.

Step through a real one, or play it.

Test input

Demonstrations bought

None yet

Answers

None yet

DemonstrationsWrong answers0 of 16 spent

Hermes Agent meets task 46442a0e for the first time: the task's id and a test input, nothing more.

Hermes Agent on GPT-6 Luna, medium, step 21 of the ARC-AGI-1 stream. Open it in the replay

A stream of instances

The stream keeps a few tasks active at once, each worked in bursts with dormant gaps between them. A task can stay dormant for long enough to be forgotten, and since every instance is a fresh draw from the task's generator, no answer can be cached. The ARC-AGI-1 stream draws 25 tasks from the 400 and serves 200 instances, followed by two tests. Every system meets the same tasks in the same order, with the same instances.

In it, the task played above is drawn in bold.

workstreams125050100150
Point at a task’s row to draw it in bold and read when the stream met it, and click it to open the task.
an instance, in its task’s colortask active, in a spell

The score, and the reset

We score the instance by the share of this feedback budget it used, all of it if the instance fails[1]. Because a failure costs the whole budget however it came about, failing is never cheaper than asking, and a demonstration is paid for only by an instance it helped to solve. U, the mean percentage of budget used over the stream, mixes what a learner could already do before the stream with what it learns along the way. To measure the learning alone, we need one more run, the reset: the same learner on the same stream and the same tests, with its state wiped before each instance. The share learned is 0 for a learner that keeps nothing, 1 for one that answers every returning task at no cost, and negative for one that does worse than the reset.

Every instance of the run above, and of three learners beside it on the same stream, at GPT-6 Luna, medium.

U 28.1P 90.5

050100not solvedU 28.1050100150200

Hermes Agent, as shipped, is the run played above. Its 200 instances average u 20.0 at a task's first meeting and 29.2 when it recurs, so U 28.1, and it solved 90.5%. Point at a bar for one instance.

Two tests after the stream

The stream measures what a learner keeps while it learns. To measure what it has at the end, we freeze the learner and serve it two tests, each a single round of 100 instances. The seen test consists of four fresh instances of each of the stream's 25 tasks, at instance seeds the stream did not use, and measures memorization. In the held-out test, which measures generalization, 100 evaluation tasks never seen on the stream are served once each, to test whether the learnings carry over to new tasks. A learner that keeps one program per task gets cheaper on the seen test but no cheaper on the held-out test, whereas a learner that has become a better solver of new tasks gets cheaper on both. On each test we read the saving as a share of the reset's whole spend on the test's instances, so that a learner whose reset already spends little is not marked down for having fewer points to save.

The plotted intervals use paired task resampling, with Bonferroni correction across the main comparison's learners separately for each model, effort, user count, track, test and metric. They condition on one learned state: four test repetitions do not measure variation across independent learning runs.

Two regimes: the individual and the collective

A deployed system may instead serve many people at once, learning from all of them, and the collective regime tests such a system. A run has several users, each following a separate stream over 25 non-overlapping tasks, so that two users bring 50 unique tasks and four bring 100. A round serves one instance per user, concurrently, and the system may change its state only between rounds. All users share one state, both the harness's notes and the trained model's weights, so that whatever a learner keeps, it learns from every user's games.

The benchmark

A track for each ARC-AGI edition

The three editions rise in difficulty[9][10], to the point where a model costing a cent per task and a frontier model are best measured on different editions. Each track is played under the same rules and read with the same measures, and serves fresh instances from generators. Most of our experiments focus on ARC-AGI-1, and the leaderboard has the second and third tracks' first runs.

TrackThe stream's tasksThe held-out testUsersAt first sight
ARC-AGI-1[1]25 of its 400 training tasks, every instance fresh from Re-ARC's generators[11]100 of its evaluation tasks, with their own pairs1, 2 or 4Claude Opus 5.5 solves 98.5% of its public tasks
ARC-AGI-2[9]25 of the 117 training tasks new to the second edition whose ARC-GEN generator[12] passes our screen, every instance fresh from it100 of the 114 evaluation tasks new to it, with their own pairs1, 2 or 4Kimi K3.1 Pro solves 29% of 100 fresh instances of the stream's tasks
ARC-AGI-3[10]15 of its 25 games, a task being a game's rules at its first level and an instance a fresh level from a generator and solver we wrote for itfresh levels of the other 10 games1Kimi K3.1 Pro solves 25% of the held-out levels

The first two tracks support 1, 2 or 4 users, each with 25 non-overlapping tasks. The paper's Appendix K defines the second and third tracks in full.

The learners · Qwen3.5-9B

Notes or weights, with the same teacher

We ask whether learning should go into the notes that an agent harness keeps or into the weights of the model that drives it. Either can be taught by the model itself, from its own verdicts, or by a stronger model. We compare learners on one student model along these two choices, who teaches and where what is learned is kept. The student is Qwen3.5-9B and the teacher GLM-5.3 Flash, both open models. Two prompting floors set the level that any learning method has to beat, and one engineered reference, a program-induction learner shaped to the domain, shows that the floors can be beaten.

Who teachesKept in the harness's notesKept in the student's weights
Taught by the student
  • Qthe harness as shipped
  • QEgiven expert guidance, playing rules and the program library's procedure
    Taught by GLM-5.3 Flash
    • QTthe teacher plays the stream and writes the notes, and the student plays from them
    • QTEthe teacher given the guidance
    • QETEboth given it
    • QDEimitates the teacher's move in each state the student met, with GLM reading expert guidance withheld from the student

    In the methods where the teacher writes the notes, every game is played twice, once by each model. The teacher's demonstrations and verdicts on each instance go uncharged, so these methods are not learners the benchmark would verify: they are an experiment, an upper bound on what a teacher can pass on through a harness. The design is the paper's Table 2.

    See how each learner did

    Acknowledgements

    We thank Yusha Arif and Haris Mahmood for their work on the early stages of this project and their comments on a draft, and Yasser Mahmoud for comments on an early draft.

    References

    1. Chollet. On the measure of intelligence. ARXIV 1911.01547, 2019.

    2. Chollet et al. ARC Prize 2025: technical report. ARXIV 2601.10904, 2026.

    3. Wu et al. StreamBench: towards benchmarking continuous improvement of language agents. NEURIPS 2024.

    4. Zheng et al. LifelongAgentBench: evaluating LLM agents as lifelong learners. ARXIV 2505.11942, 2025.

    5. Joshi et al. SWE-Bench-CL: continual learning for coding agents. ARXIV 2507.00014, 2025.

    6. Guo et al. MLLM-CTBench: a benchmark for continual instruction tuning with reasoning process diagnosis. ARXIV 2508.08275, 2025.

    7. Shu et al. AgentCL: toward rigorous evaluation of continual learning in language agents. ARXIV 2606.02461, 2026.

    8. Yang et al. PATH-Bench: path-dependent evaluation of lifelong agents. ARXIV 2608.01149, 2026.

    9. Chollet et al. ARC-AGI-2: a new challenge for frontier AI reasoning systems. ARXIV 2505.11831, 2025.

    10. ARC Prize Foundation. ARC-AGI-3: a new challenge for frontier agentic intelligence. ARXIV 2603.24621, 2026.

    11. Hodel. Addressing the Abstraction and Reasoning Corpus via procedural example generation. ARXIV 2404.07353, 2024.

    12. Moffitt. ARC-GEN: a mimetic procedural benchmark generator for the Abstraction and Reasoning Corpus. ARXIV 2511.00162, 2025.

    Submit yours

    Have your learner verified

    Every task of the benchmark and every task's solver presented so far is public. External submissions are therefore verified by running the learner's code on a private set of new tasks for its track that only we have seen: on ARC-AGI-1 and ARC-AGI-2, 50 new rules, 25 for the stream and the seen test and 25 for the held-out test, and on ARC-AGI-3, 25 new games, 15 for the stream and 10 held out. A learner is verified automatically from an issue on the benchmark's repository. Continual-ARC is designed to explain how a learner learns rather than to purely rank learners, so we give no single score per se, and we present all verified submissions of genuine learning algorithms on the benchmark, novel or not.

    shell
    git clone https://github.com/unifyai/continual-arc
    cd continual-arc
    uv venv && uv pip install -e ".[dev]"
    uv run continual-arc data fetch
    uv run continual-arc schedule make --config arc1 \
      --seed 0 --out arc1-ws0.json
    How to run a learner, submit it and have it verified

    Cite

    BibTeX

    Continual-ARC: Measuring Skill Acquisition over a Working Lifetime. Daniel Lenton, unify, 2026.

    bibtex
    @misc{lenton2026continualarc,
      title  = {Continual-ARC: Measuring Skill Acquisition over a Working Lifetime},
      author = {Lenton, Daniel},
      year   = {2026},
      url    = {https://continual-arc.unify.ai}
    }