Leaderboard
Every learner after the stream, frozen, on the two tests: across on fresh instances of the tasks it met, which is memorization, and up on 100 tasks it never met, which is generalization, so the top right is better on both. Each line on the left runs from a learner's reset to the learner, and the right reads that change against the reset, so a stronger model earns nothing there for being stronger. The two show both what a system can do and how much it learns, and the interesting systems have something to show on both: a large share saved by a system already near the cap says little, and so does a small gain by one that can hardly play. Continual-ARC is meant to explain how a learner learns rather than to purely rank learners, so there is no single score.
Point at a learner, or tap it, to light it in both graphs and read its numbers here. Drag a box across a graph to zoom in, and click it to zoom out.
Q Qwen3.5-9B (student) plays in the harness (Prime Agent)G GLM-5.3 Flash (teacher) plays in it
T teacher writes the notesE expert guidance, for the letter before itD DAgger, imitates the teacher
P program library, in place of the harnessC demo cache, in place of the harness(n) n concurrent usersA asked to save
| Learner | Teacher | ||||||
|---|---|---|---|---|---|---|---|
| QD(2)DAgger, imitates the teacher, at 2 concurrent users | GLM-5.3 Flash | 0.55 0.46 to 0.64 | 0.40 0.34 to 0.47 | 82% → 37% | 86% → 51% | 18% → 62% | 0.47 0.38 to 0.56 |
| QD(4)DAgger, imitates the teacher, at 4 concurrent users | GLM-5.3 Flash | 0.51 0.45 to 0.57 | 0.39 0.33 to 0.46 | 78% → 38% | 86% → 52% | 18% → 63% | 0.53 0.46 to 0.60 |
| QDDAgger, imitates the teacher | GLM-5.3 Flash | 0.57 0.48 to 0.67 | 0.29 0.23 to 0.35 | 84% → 36% | 85% → 60% | 19% → 49% | 0.43 0.33 to 0.54 |
| told the pageuntrained, told the teacher's page of playing rules | none | 0.16 0.08 to 0.26 | 0.09 0.05 to 0.14 | 84% → 70% | 85% → 77% | 19% → 28% | – |
| GEgiven the expert guidance | none | 0.74 0.64 to 0.84 | 0.07 −0.02 to 0.16 | 18% → 5% | 18% → 17% | 97% → 97% | 0.62 0.49 to 0.75 |
| GPthe program library, in place of the harness | none | 0.81 0.59 to 1.00 | 0.05 −0.06 to 0.14 | 19% → 4% | 19% → 18% | 97% → 98% | 0.68 0.49 to 0.87 |
| QPthe program library, in place of the harness | none | 0.65 0.46 to 0.86 | 0.03 −0.02 to 0.09 | 65% → 22% | 76% → 73% | 29% → 33% | 0.43 0.25 to 0.64 |
| QTthe teacher writes the notes | GLM-5.3 Flash | 0.06 −0.01 to 0.14 | 0.03 −0.02 to 0.07 | 84% → 79% | 87% → 85% | 17% → 19% | 0.05 −0.02 to 0.14 |
| Gthe harness as shipped | none | 0.12 −0.03 to 0.27 | 0.00 −0.11 to 0.10 | 22% → 19% | 23% → 23% | 94% → 94% | 0.12 −0.16 to 0.36 |
| QAasked to save anything that could help | none | 0.02 −0.02 to 0.07 | −0.00 −0.05 to 0.04 | 84% → 82% | 87% → 87% | 17% → 17% | −0.05 −0.11 to 0.01 |
| Qthe harness as shipped | none | −0.01 −0.04 to 0.03 | −0.01 −0.05 to 0.03 | 84% → 85% | 87% → 88% | 17% → 16% | −0.02 −0.09 to 0.04 |
| QCthe demo cache, which shows again the examples it paid for, in place of the harness | none | 0.37 0.26 to 0.51 | −0.01 −0.05 to 0.03 | 67% → 42% | 73% → 74% | 32% → 33% | 0.31 0.19 to 0.45 |
| QEgiven the expert guidance | none | 0.03 −0.08 to 0.13 | −0.01 −0.05 to 0.03 | 74% → 72% | 80% → 81% | 25% → 23% | 0.05 −0.06 to 0.16 |
| QTEthe teacher writes them, given the expert guidance | GLM-5.3 Flash | 0.01 −0.05 to 0.07 | −0.02 −0.07 to 0.03 | 84% → 83% | 87% → 89% | 17% → 15% | 0.07 −0.03 to 0.16 |
| GCthe demo cache, which shows again the examples it paid for, in place of the harness | none | 0.44 0.34 to 0.57 | −0.02 −0.11 to 0.06 | 39% → 22% | 41% → 42% | 78% → 76% | 0.41 0.29 to 0.53 |
| GAasked to save anything that could help | none | 0.26 0.13 to 0.39 | −0.03 −0.14 to 0.06 | 22% → 16% | 23% → 24% | 94% → 94% | 0.31 0.20 to 0.39 |
| QETEthe teacher writes them, both given the expert guidance | GLM-5.3 Flash | 0.34 0.26 to 0.42 | −0.05 −0.10 to −0.00 | 74% → 49% | 80% → 83% | 25% → 21% | 0.29 0.19 to 0.40 |
| G0the harness, wiped before every instance | none | −0.05 −0.23 to 0.11 | 0.13 −0.01 to 0.26 | 21% → 22% | 25% → 21% | 93% → 96% | 0 |
| Q0the harness, wiped before every instance | none | 0.00 −0.05 to 0.07 | 0.03 −0.03 to 0.08 | 84% → 84% | 89% → 86% | 15% → 18% | 0 |
| GFthe model alone, keeping nothing | none | −0.06 −0.16 to 0.04 | 0.03 −0.10 to 0.14 | 38% → 40% | 42% → 41% | 79% → 78% | 0 |
| QP0the program library, wiped before every instance | none | 0.03 −0.07 to 0.12 | 0.01 −0.06 to 0.07 | 66% → 64% | 76% → 76% | 28% → 29% | 0 |
| QE0given the expert guidance, wiped before every instance | none | −0.06 −0.14 to 0.01 | 0.00 −0.05 to 0.06 | 72% → 76% | 80% → 80% | 25% → 25% | 0 |
| GE0given the expert guidance, wiped before every instance | none | −0.06 −0.24 to 0.07 | −0.01 −0.15 to 0.12 | 17% → 18% | 18% → 18% | 97% → 97% | 0 |
| QFthe model alone, keeping nothing | none | 0.06 −0.03 to 0.15 | −0.02 −0.10 to 0.05 | 69% → 64% | 72% → 74% | 34% → 31% | 0 |
| GP0the program library, wiped before every instance | none | 0.00 −0.16 to 0.13 | −0.05 −0.22 to 0.09 | 19% → 19% | 18% → 19% | 97% → 96% | 0 |
Every learner on view, sorted by the held-out test until another column is chosen. A share in gray cannot be told apart from zero once its interval is widened for the number of learners on the panel. The rows in light type keep nothing between instances, and each reads half its plays against the other half: how far chance moves a point. Every number is from the paper's own pipeline.
Verified submissions
Learners from outside unify go here, each placed the same way: the share of its reset's spend it saves on the seen test and on the held-out test, with the models it uses. We present any verified submission of a genuine learning algorithm, novel or not, beside the learner's own run on the public stream if it sent one. We verify a learner by running its code ourselves on Continual-ARC's private set: 50 new ARC-style rules that nobody outside unify has seen, each a generator of fresh instances, 25 for the stream and the seen test and 25 for the held-out test, played at a seed we do not disclose.
Each of its models plays the same test instances bare. A † beside a share marks a test on which its reset plays below one of them that costs no more to run, or below the cheapest of them, since a learner that played worse whenever its memory was empty would score for it, a ‡ a test on which the learner plays no better than one of them, since one that only switched to a stronger model once its memory was filled would score too, and a § a test on which the learner sends a larger share of its calls to a dearer model than its reset does, since a gain bought with a stronger model would score as well. Run yours says how to have a learner verified and what it may use.
None yet.
Submit yoursFurther introspection: the stream
The figure at the top reads each learner after the stream. These views read the stream itself, for GPT-6 Luna's runs, which cover three harnesses at three reasoning efforts: each learner's share learned while it played, against what its model calls cost per instance, then its budget used. They help explain a place in the figure rather than rank anyone.
Share learned on the stream, against spend
Each point is one learner at one effort on the stream: its share learned there against what its model calls cost per instance, joined from the least effort to the most. Higher and further left is better. A learner that keeps nothing between instances, the stateless floor or a wiped harness, is its own reset and sits at 0 by construction, so this chart leaves it out.
The cost is what it takes to serve an instance. A learner that trains its weights will show what its training cost, in dollars at the rental price of its GPUs, in a column of its own beside it, never added into this axis.
Hover a point for its numbers, and click it to replay the run. A hollow point is a harness with everything it wrote deleted before every instance, in the chart of the budget used below. Click a learner under either chart to hide or show it in both, or Option-click it to show it alone. The intervals switch above draws every point's interval.
Every learner at medium effort, on the stream
| Share learned | Stream | ||||
|---|---|---|---|---|---|
| # | Learner | ||||
| 1 | Program library | +0.60 ±0.13 | 11.5 ±4.0 | 94.5 | $0.005 |
| – | Demo-cache floor | +0.24 ±0.08 | 27.4 ±5.0 | 83.5 | $0.010 |
| – | Stateless floor | 0 | 35.1 ±4.5 | 84.0 | $0.009 |
| – | Prime Agentmemory wiped | 0 | 25.7 ±3.2 | 92.0 | $0.009 |
| 2 | Hermes Agentas shipped | 0.00 ±0.06 | 28.1 ±3.7 | 90.5 | $0.026 |
| – | Hermes Agentmemory wiped | 0 | 28.0 ±4.1 | 91.0 | $0.069 |
| – | OpenClawmemory wiped | 0 | 27.8 ±3.8 | 90.0 | $0.028 |
| 3 | Prime Agentas shipped | −0.04 ±0.05 | 26.6 ±3.5 | 92.5 | $0.010 |
| 4 | OpenClawas shipped | −0.15 ±0.10 | 31.5 ±4.3 | 87.0 | $0.028 |
L is the share learned on the stream: the share of the learnable budget a learner saves over itself reset, the same model in the same harness wiped before every instance, 0 for a learner that keeps nothing and 1 for perfect memory, so a stronger model scores no higher for being stronger. The demo-cache floor is read against the stateless floor, a harness against itself wiped and the program library against its programs wiped. A share in grey cannot be told apart from 0, since its 95 percent interval takes in 0. The floors, the model alone, take no rank, and neither does a learner that keeps nothing, the zero of its own scale. U is the budget used and P the share solved within the cap, both in percent, over the stream's 200 instances. Small type is the standard error over tasks, as in the paper's tables. Click a column to sort by it, and a number to replay its run. After its stream each workstreams run went on to some or all of the 100 held-out tasks while it could still learn, which the two frozen tests replace, so nothing from those tasks is here.
Budget used on the stream, against spend
The same learners on the paper's raw measure: the share of the feedback budget each used on the stream, against what its model calls cost per instance, joined from the least effort to the most. Lower and further left is better.
The budget used rewards the model and the harness as much as the learning, since a stronger model buys fewer demonstrations whether or not it learns. The share learned reads it against the learner itself reset to take that out; this chart shows what it starts from. Its whiskers are one standard error either side.
Harnesses asked to store, or told to keep a program library
These rows change what a harness is told, so the paper reports them apart, in its Appendix F. Asked to store, a harness hears after every verdict that it should save anything that could help with the task's next instances. Told to keep a program library, it carries out the program library's method with its own tools, keeping one tested program per task and running it first when the task returns; wiped, it does the same with everything deleted before every instance. The last row is the program library with nothing kept.
| Share learned | Stream | ||||
|---|---|---|---|---|---|
| # | Learner | ||||
| 1 | Prime Agenttold to keep a program library | +0.47 ±0.09 | 13.0 | 97.5 | $0.011 |
| 2 | OpenClawtold to keep a program library | +0.47 ±0.07 | 14.7 | 95.5 | $0.067 |
| 3 | Hermes Agenttold to keep a program library | +0.44 ±0.10 | 16.9 | 92.5 | $0.061 |
| 4 | Prime Agentasked to store | +0.32 ±0.08 | 18.4 | 95.0 | $0.010 |
| 5 | Hermes Agentasked to store | +0.28 ±0.07 | 21.0 | 90.5 | $0.039 |
| 6 | OpenClawasked to store | +0.03 ±0.08 | 27.2 | 90.0 | $0.043 |
| – | Prime Agenttold to keep a program library, memory wiped | 0 | 22.6 | 95.5 | $0.013 |
| – | Hermes Agenttold to keep a program library, memory wiped | 0 | 28.0 | 88.0 | $0.100 |
| – | OpenClawtold to keep a program library, memory wiped | 0 | 25.5 | 92.0 | $0.12 |
| – | Program library, nothing kept | 0 | 24.6 | 92.5 | $0.008 |
The paper reads these rows against each other, instance by instance, so their budgets carry no error bar here. Each one's share learned is read against itself reset, as every other row's is: a harness asked to store against the same harness wiped, one told to keep a program library against itself told the same and wiped.
Next: the second and third tracks
Everything above is on ARC-AGI-1, which leaves the strongest models little to learn, Claude Opus 5.5 solving 98.5 percent of its public tasks at first sight. ARC-AGI-2 and ARC-AGI-3 leave far more. So far we have played one model on each, Kimi K3.1 Pro at one user, in the two floors and one harness, below; comparing notes and weights on them is future work.
ARC-AGI-2
Point at a learner, or tap it, to light it in both graphs and read its numbers here. Drag a box across a graph to zoom in, and click it to zoom out.
K Kimi K3.1 Pro plays in the harness (Prime Agent)C demo cache, in place of the harness
ARC-AGI-3
Point at a learner, or tap it, to light it in both graphs and read its numbers here. Drag a box across a graph to zoom in, and click it to zoom out.
K Kimi K3.1 Pro plays in the harness (Prime Agent)C demo cache, in place of the harness
| stream | frozen tests | ||||
|---|---|---|---|---|---|
| track | row | solved | L | seen | held-out |
| ARC-AGI-2 | stateless floor | 61.0% | 0 | −0.01 −0.13 to +0.10 | −0.01 −0.05 to +0.03 |
| demo-cache floor | 71.0% | +0.35 +0.22 to +0.48 | +0.37 +0.26 to +0.48 | +0.01 −0.02 to +0.04 | |
| Prime Agent, as shipped | 64.5% | 0.00 −0.09 to +0.08 | +0.01 −0.06 to +0.09 | −0.02 −0.04 to +0.01 | |
| Prime Agent, wiped | 65.0% | 0 | −0.04 −0.21 to +0.09 | +0.01 −0.03 to +0.06 | |
| ARC-AGI-3 | stateless floor | 25.8% | 0 | −0.06 −0.12 to 0.00 | +0.01 −0.05 to +0.06 |
| demo-cache floor | 32.5% | +0.11 +0.01 to +0.23 | +0.13 +0.04 to +0.24 | −0.02 −0.10 to +0.04 | |
| Prime Agent, as shipped | 60.0% | +0.04 −0.07 to +0.14 | +0.03 −0.05 to +0.09 | −0.04 −0.11 to +0.02 | |
| Prime Agent, wiped | 52.5% | 0 | +0.01 −0.04 to +0.06 | +0.06 −0.03 to +0.17 | |