Continual-ARCby unify
Submit yours

A continual learning benchmark · 114 runs

Continual-ARC

Measuring Skill Acquisition over a Working Lifetime

Daniel Lenton, Yusha Arif and Haris Mahmood

See the resultsRead the postCode

Abstract

Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks that measure such systems have several limitations: they generally focus on a single applied domain, they usually serve each instance only once, and most assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale.

We introduce Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge and which the models under test solve only in part. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit an answer, or take actions inside the harness, with retries permitted on an incorrect submission. Each instance is scored based on the share of this feedback budget (demonstrations and failed attempts) consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization.

We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. We also define a second regime in which one system learns from several users simultaneously.

Across our experiments, we show that persistence in the harness saves nothing measurable on returning tasks when the model chooses what to store. Most of them learn less than a floor that simply caches each task's demonstrations. In contrast, a simple method that asks the LLM for one program per task and keeps following demo reproduction outperforms them by a wide margin. Neither gains on unseen tasks, where only weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.

Budget used, from the reset to the learner030608085705040200306080908570504015memorization: budget used on the seen testgeneralization: budget used on the held-out testQQAQEQTQTEQETEQPQDQD(2)QD(4)QCtold the pageGGAGEGPGCAcross, the seen test, on a log scale: click for a linear scalelogUp, the held-out test, on a log scale: click for a linear scalelog
Share saved, against the reset00.20.40.60.800.20.4memorization: share saved on the seen testgeneralization: share saved on the held-out testchance:Q innerG outerQQAQEQTQTEQETEQPQDQD(2)QD(4)QCtold the pageGGAGEGPGCAcross, the seen test, on a linear scale: click for a log scalelinUp, the held-out test, on a linear scale: click for a log scalelin

Point at a learner, or tap it, to light it in both graphs and read its numbers here. Drag a box across a graph to zoom in, and click it to zoom out.

Q Qwen3.5-9B (student) plays in the harness (Prime Agent)G GLM-5.3 Flash (teacher) plays in it

T teacher writes the notesE expert guidance, for the letter before itD DAgger, imitates the teacher

P program library, in place of the harnessC demo cache, in place of the harness(n) n concurrent usersA asked to save

Figure 1. Every learner that keeps something between instances, frozen after the stream, at one user and DAgger also at two and four, whose users share one model and bring puzzles of their own. Faint lines are 95 percent intervals. The gray bands are how far chance moves a learner that keeps nothing, widened for the number of learners on the panel, so a point outside a band is more than chance on that test for the panel as a whole. A square is a harness asked to save, which the paper's Figure 1 leaves to its appendix. Every learner's numbers are on the leaderboard.

Introduction

Learning over a working lifetime

Chollet (2019)[1] argued that intelligence should be measured by how efficiently a system turns priors and experience into skill, on tasks that lie largely outside its experience and have a verifiable answer. ARC is the concrete instance of that argument, and on its current edition the gap between what a person solves in a minute and what a model solves from the same demonstrations is still significant[2]. For agent systems that continually operate for days and weeks, a more pressing question is whether a skill acquired on one job is still there the next time a similar scenario is met, which we call memorization, and whether that experience makes the system better at similar jobs it has never met, which we call generalization. By design, ARC measures acquisition once, within a single episode, and does not ask either over longer horizons. An overview of our final results is shown in Figure 1 above, and its terms are explained below, from the benchmark to the learners, with every learner's numbers on the leaderboard.

The benchmarks that do ask it have three limitations.

  1. Each is built around one applied domain, such as coding or office work[3][4][5], in which the models under test already know a great deal from the pretraining data. A method's gains may come partly from drawing out that knowledge[6][7][8], and so may not carry over to a domain that the model is less familiar with.
  2. Most serve a fixed set of instances, each seen once, so the same conceptual task never returns with a fresh input. This means a memorized answer cannot be told apart from a more abstract learned skill.
  3. Most assume a single family of learner, gradient updates for fine-tuning suites[9] and prompting and memory for agent benchmarks[10][11], so memory-based and weight-based systems are rarely compared on the same scale.

Continual-ARC is a lifelong stream built from ARC, where tasks need no knowledge of any domain beyond core-knowledge priors, such as objectness and counting[1], and the rule of each grid task has to be induced from a few demonstrations. A task can return after a dormant gap as often as the stream asks, each time with a new input that no cached answer fits. New tasks also keep arriving throughout the stream. Because a system may carry anything between instances, whether weights or notes, prompting, memory and weight updates all face the same stream and the same tests.

The benchmark

From one instance to a stream and two tests

One instance

An ARC task is a rule that turns an input grid into an output grid, and an instance of it is one input whose output the learner must produce. For each instance, the learner observes the task id, the input grid, and how many attempts and demonstration requests it has used on this instance, but it is told nothing about its position in the stream. It may submit a grid, and the environment answers only correct or incorrect. A correct submission ends the instance, and an incorrect one costs 1. It may instead request a demonstration, at a cost of 1, which returns one more input-output pair of the task, up to eight per instance. Failing to get the correct answer costs 16, even if there were demonstration requests to spare.

Step through a real one, or play it.

Test input

Demonstrations bought

None yet

Answers

None yet

DemonstrationsWrong answers0 of 16 spent

Hermes Agent meets task 46442a0e for the first time: the task's id and a test input, nothing more.

Hermes Agent on GPT-6 Luna, medium, step 21 of the ARC-AGI-1 stream. Open it in the replay

A stream of instances

The stream keeps a few tasks active at once, each worked in bursts with dormant gaps between them. A task can stay dormant for long enough to be forgotten, and since every instance is a fresh draw from the task's generator, no answer can be cached. The ARC-AGI-1 stream draws 25 tasks from the 400 and serves 200 instances, followed by two tests. Every system meets the same tasks in the same order, with the same instances.

In it, the task played above is drawn in bold.

workstreams125050100150
Point at a task’s row to draw it in bold and read when the stream met it, and click it to open the task.
first meeting with the taskthe task recurstask active, in a spell

The score, and the reset

We score the instance by the share of this feedback budget it used, all of it if the instance fails[1]. Because a failure costs the whole budget however it came about, failing is never cheaper than asking, and a demonstration is paid for only by an instance it helped to solve. U, the budget used over the stream, mixes what a learner could already do before the stream with what it learns along the way. To measure the learning alone, we need one more run, the reset: the same learner on the same stream and the same tests, with its state wiped before each instance. The share learned is 0 for a learner that keeps nothing, 1 for one that answers every returning task at no cost, and negative for one that does worse than the reset.

Every instance of the run above, and of three learners beside it on the same stream, at GPT-6 Luna, medium.

U 28.1P 90.5

050100not solvedU 28.1050100150200

Hermes Agent, as shipped, is the run played above. Its 200 instances average u 20.0 at a task's first meeting and 29.2 when it recurs, so U 28.1, and it solved 90.5 percent. Point at a bar for one instance.

Two tests after the stream

The stream measures what a learner keeps while it learns. To measure what it has at the end, we freeze the learner and serve it two tests, each a single round of 100 instances. The seen test consists of four fresh instances of each of the stream's 25 tasks, at instance seeds the stream did not use, and measures memorization. In the held-out test, which measures generalization, 100 evaluation tasks never seen on the stream are served once each, to test whether the learnings carry over to new tasks. A learner that keeps one program per task gets cheaper on the seen test but no cheaper on the held-out test, whereas a learner that has become a better solver of new tasks gets cheaper on both. On each test we read the saving as a share of the reset's whole spend on the test's instances, so that a learner whose reset already spends little is not marked down for having fewer points to save.

Two regimes: the individual and the collective

A deployed system may instead serve many people at once, learning from all of them, and the collective regime tests such a system. A run has several users, each following a separate stream over 25 tasks of their own, so that two users bring 50 tasks and four bring 100. A round serves one instance per user, concurrently, and the system may change its state only between rounds. All users share one state, a harness's notes as much as a trained model's weights, so that whatever a learner keeps, it learns from every user's games.

The benchmark

A track for each ARC-AGI edition

The three editions rise in difficulty[12][13], to the point where a model costing a cent per task and a frontier model are best measured on different editions. Each track is played under the same rules and read with the same measures, and serves fresh instances from generators. Most of our experiments focus on ARC-AGI-1, and the leaderboard has the second and third tracks' first runs.

TrackThe stream's tasksThe held-out testUsersAt first sight
ARC-AGI-1[1]25 of its 400 training tasks, every instance fresh from Re-ARC's generators[14]100 of its evaluation tasks, with their own pairs1 to 16Claude Opus 5.5 solves 98.5% of its public tasks
ARC-AGI-2[12]25 of the 117 training tasks new to the second edition whose ARC-GEN generator[15] passes our screen, every instance fresh from it100 of the 114 evaluation tasks new to it, with their own pairs1 to 4Kimi K3.1 Pro solves 29% of the stream's tasks
ARC-AGI-3[13]15 of its 25 games, a task being a game's rules at its first level and an instance a fresh level from a generator and solver we wrote for itfresh levels of the other 10 games1Kimi K3.1 Pro solves 25% of the held-out levels

A user brings 25 tasks of their own, so a track serves as many users as its pool holds blocks of 25. The paper's Appendix K defines the second and third tracks in full.

The learners · Qwen3.5-9B

Notes or weights, with the same teacher

We ask whether learning should go into the notes that an agent harness keeps or into the weights of the model that drives it. Either can be taught by the model itself, from its own verdicts, or by a stronger model. We compare learners on one student model along these two choices, who teaches and where what is learned is kept. The student is Qwen3.5-9B and the teacher GLM-5.3 Flash, both open models. Two prompting floors set the level that any learning method has to beat, and one engineered reference, a program-induction learner shaped to the domain, shows that the floors can be beaten.

Who teachesKept in the harness's notesKept in the student's weights
Taught by the student
  • Qthe harness as shipped
  • QEgiven expert guidance, the program library's procedure
    Taught by GLM-5.3 Flash
    • QTthe teacher plays the stream and writes the notes, and the student plays from them
    • QTEthe teacher given the guidance
    • QETEboth given it
    • QDimitates the teacher's move in each state the student met, the teacher reading a page of playing rules

    In the rows where the teacher writes the notes, every game is played twice, once by each model. The teacher's demonstrations and verdicts on each instance go uncharged, so these rows are not learners the benchmark would verify: they are an experiment, an upper bound on what a teacher can pass on through a harness. The design is the paper's Table 3.

    See how each learner did

    References

    1. Chollet. On the measure of intelligence. ARXIV 1911.01547, 2019.

    2. Chollet et al. ARC Prize 2025: technical report. ARXIV 2601.10904, 2026.

    3. Wu et al. StreamBench: towards benchmarking continuous improvement of language agents. NEURIPS 2024.

    4. Zheng et al. LifelongAgentBench: evaluating LLM agents as lifelong learners. ARXIV 2505.11942, 2025.

    5. Joshi et al. SWE-Bench-CL: continual learning for coding agents. ARXIV 2507.00014, 2025.

    6. Yue et al. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? ARXIV 2504.13837, 2025.

    7. Shao et al. Spurious rewards: rethinking training signals in RLVR. ARXIV 2506.10947, 2025.

    8. Wu et al. Reasoning or memorization? Unreliable results of reinforcement learning due to data contamination. ARXIV 2507.10532, 2025.

    9. Guo et al. MLLM-CTBench: a benchmark for continual instruction tuning with reasoning process diagnosis. ARXIV 2508.08275, 2025.

    10. Shu et al. AgentCL: toward rigorous evaluation of continual learning in language agents. ARXIV 2606.02461, 2026.

    11. Yang et al. PATH-Bench: path-dependent evaluation of lifelong agents. ARXIV 2608.01149, 2026.

    12. Chollet et al. ARC-AGI-2: a new challenge for frontier AI reasoning systems. ARXIV 2505.11831, 2025.

    13. ARC Prize Foundation. ARC-AGI-3: a new challenge for frontier agentic intelligence. ARXIV 2603.24621, 2026.

    14. Hodel. Addressing the Abstraction and Reasoning Corpus via procedural example generation. ARXIV 2404.07353, 2024.

    15. Moffitt. ARC-GEN: a mimetic procedural benchmark generator for the Abstraction and Reasoning Corpus. ARXIV 2511.00162, 2025.

    Submit yours

    Have your learner verified

    Every task of the benchmark and every task's solver presented so far is public. External submissions are therefore verified by running the learner's code on a private set of 50 new rules that only we have seen: 25 for the stream and the seen test, and 25 for the held-out test. A learner that does not train its weights is verified automatically from an issue on the benchmark's repository, and one that does is run with us on GPUs we rent. Continual-ARC is meant to explain how a learner learns rather than to purely rank learners, so we give no single score, and we present any verified submission of a genuine learning algorithm, novel or not.

    shell
    git clone https://github.com/unifyai/continual-arc
cd continual-arc
uv venv && uv pip install -e ".[dev]"
uv run continual-arc data fetch
uv run continual-arc schedule make --config arc1 \
  --seed 0 --out arc1-ws0.json
    How to run a learner, submit it and have it verified

    Cite

    BibTeX

    Continual-ARC: Measuring Skill Acquisition over a Working Lifetime. Daniel Lenton, Yusha Arif and Haris Mahmood, unify, 2026.

    bibtex
    @misc{lenton2026continualarc,
  title  = {Continual-ARC: Measuring Skill Acquisition over a Working Lifetime},
  author = {Lenton, Daniel and Arif, Yusha and Mahmood, Haris},
  year   = {2026},
  url    = {https://continual-arc.unify.ai}
}