Run Continual-ARC on your own learner.

Step 01

Install the benchmark

You need uv and Python 3.10 or newer. The last command clones Re-ARC and ARC-AGI at pinned commits into ~/.cache/continual-arc unless CONTINUAL_ARC_DATA says otherwise.

git clone https://github.com/unifyai/continual-arc
cd continual-arc
uv venv && uv pip install -e ".[dev]"
uv run continual-arc data fetch

Step 02

A smoke run

arc1-smoke is a short stream, 128 instances of 16 tasks, and the learner here gives up on every instance, so it needs no model and no key and takes about a second. The runner writes the run directory: records.jsonl, one line per instance, with summary.json, schedule.json, config.json and meta.json.

uv run continual-arc schedule make --config arc1-smoke --seed 0 --out smoke.json
uv run continual-arc run --learner continual_arc.testing:GiveUpLearner \
  --schedule smoke.json --out runs/smoke
uv run continual-arc report runs/smoke

The report's score is 100 minus the budget used, so the leaderboard's U is 100 minus the score, or 100 times the mean cost over 16; its solve rate is P.

Step 03

Write a learner

A learner is any object with these methods, and whatever it keeps between instances is the method under test: notes, programs, a cache, or the weights of a model it trains. The runner calls reset wherever the schedule wipes the learner, which the reset reference does before every instance. What it keeps changes only in end_round, once a round is over, so every user of a round, and every instance of a frozen test, meets the state the round began with. This one keeps the demonstrations it pays for on each task, holding a round's new ones apart by user until the round ends, and asks solve for an answer; put your method there and it runs as it stands.

The learner sees only the instance: the task under a name drawn for the run, such as task-3f2a9c1e, the input, and its own attempts and requests against their caps. It is told nothing about the schedule, neither the phase nor how often the task has come before, and the run's records keep the real ids. The real id would be a lookup key into the public tasks and their solvers.

from continual_arc.env import DemosFeedback, GiveUp, LearnerObservation, RequestDemos, Submit
from continual_arc.learner import BaseLearner


def solve(pairs, grid):
    """Your method: an answer from the task's pairs so far, or None to pay for one more."""
    return None


class MyLearner(BaseLearner):
    def __init__(self):
        self.pairs = {}                  # what it keeps between instances, by task id
        self.bought = {}                 # this round's pairs, by user and task id

    def reset(self):                     # the reset reference wipes it before each instance
        self.pairs.clear()
        self.bought.clear()

    def act(self, obs: LearnerObservation):  # obs.task_id, obs.input, obs.user, obs.attempts_used, obs.demo_cap, ...
        pairs = self.pairs.get(obs.task_id, []) + self.bought.get((obs.user, obs.task_id), [])
        guess = solve(pairs, obs.input)
        if guess is not None:
            return Submit(guess)         # wrong costs 1; right ends the instance
        if obs.demo_requests_used < obs.demo_cap:
            return RequestDemos()        # one more pair of the task, costs 1
        return GiveUp()                  # ends the instance, charged as failed

    def observe(self, obs, action, feedback):
        if isinstance(feedback, DemosFeedback):
            self.bought.setdefault((obs.user, obs.task_id), []).extend(feedback.pairs)

    def end_round(self):                 # what it keeps changes only once a round is over
        for (_, task_id), pairs in self.bought.items():
            self.pairs.setdefault(task_id, []).extend(pairs)
        self.bought.clear()

    def resources(self):                 # cumulative counters, logged and never scored
        return {"pairs_kept": float(sum(map(len, self.pairs.values())))}

Run it from the folder that holds my_pkg/; PYTHONPATH=. lets the runner import it, or install your package into the same environment.

PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule smoke.json --out runs/my-smoke

Step 04

Or drive a harness

The baselines repository runs the paper's learners and drives agent harnesses through the benchmark's protocol. arc-baselines run starts a local proxy for the run and points OPENAI_BASE_URL, OPENROUTER_BASE_URL and ARC_LLM_BASE_URL at it, so every model call a harness makes is counted whatever the harness does, and the real key stays in the proxy. Each harness runs in a sandbox, bubblewrap on Linux and Seatbelt on macOS, that can write only its own state and reach the network only through the proxy, and it sees each task under an alias drawn for the run. The baselines need Python 3.12 or 3.13.

git clone https://github.com/unifyai/continual-arc
cd continual-arc/baselines
uv sync
uv run continual-arc data fetch
uv run arc-baselines run --learner continual_arc.testing:GiveUpLearner \
  --config arc1-smoke --seed 0

Adapters for Prime Agent, Hermes Agent and OpenClaw are in systems/, and scripts/launch-cell.sh launches any cell the paper reports, with the harness installed at the version of the paper's Table 7. This is OpenClaw as shipped on GPT-6 Luna at medium effort, playing the stream alone, as every cell does, so that its frozen tests can follow it as yours do below:

export OPENROUTER_API_KEY=...
bash scripts/launch-cell.sh acp workstreams 0 arc1 luna6-medium openclaw

For a harness of your own, subclass SystemLearner, which speaks the protocol, handles re-prompts, restarts and wipes, and leaves you five methods.

from continual_arc_baselines.systems.base import SystemLearner


class MyHarnessLearner(SystemLearner):
    """Everything the harness keeps between instances lives under self.state_dir."""

    name = "my-harness"

    def start(self):            # launch it, its model client on the proxy (self.environ)
        ...

    def stop(self):
        ...

    def ask(self, message, *, reprompt=False):  # one message in, its final reply out
        ...

    def new_session(self):      # a fresh conversation; what it keeps stays
        ...

    def wipe_state(self):       # delete everything it kept (called while stopped)
        ...

"end": "finish" ends an instance as the paper's harness rows did: the harness hears the verdict and nothing else, keeps whatever it chooses, and says when it is done. --llm-defaults sets the reasoning effort on any model call that leaves it unset, and --no-epilogue plays the stream alone, so that the frozen tests start from where it ends.

PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --config arc1 --seed 0 --no-epilogue --out runs/arc1/my-harness-ws0

A harness keeps one conversation, so it plays each frozen test one instance at a time. Before every instance arc-baselines test stops it and puts its state directory back as the stream left it, at the same path, so every path the harness wrote into its own files still names its own state. It builds the harness as the stream's launch did, so run it from the same folder, with the same key. The two tests play one after the other, each about as long as half the stream, and your harness's reset references, the same instances played with arc-baselines run, which wipes it before each, can play beside them. So can its reset on the stream, which wipes it before every instance of the same stream, from the schedule Step 05 makes.

uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --width 1 --out arc1-held-out0-1.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --width 1 --out arc1-seen0-1.json
PYTHONPATH=. uv run arc-baselines test runs/arc1/my-harness-ws0 --schedule arc1-held-out0-1.json \
  --out runs/arc1/my-harness-held-out
PYTHONPATH=. uv run arc-baselines test runs/arc1/my-harness-ws0 --schedule arc1-seen0-1.json \
  --out runs/arc1/my-harness-seen
PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --schedule arc1-held-out0-1.json --out runs/arc1/my-harness-reset-held-out
PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --schedule arc1-seen0-1.json --out runs/arc1/my-harness-reset-seen
# the stream with the harness wiped before every instance: arc1-ws0-reset.json from Step 05
PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --schedule arc1-ws0-reset.json --out runs/arc1/my-harness-ws0-reset

Each test's share saved and the share learned come from these runs, as Step 05 shows, with my-harness in place of my-learner.

Step 05

Run the stream and its tests

The track is arc1's stream at seed 0: 200 instances of 25 training tasks from Re-ARC's generators, and then two frozen tests. The held-out test serves one instance of each of 100 ARC-AGI-1 evaluation tasks the stream never served, and the seen test 100 fresh instances of the stream's 25 tasks, four of each. Each test is one round in which every instance is a user of its own, starting from where the stream left your learner, so nothing learnt on one test instance reaches another. The paper's rows are at seed 0.

uv run continual-arc schedule make --config arc1 --seed 0 --no-epilogue \
  --out arc1-ws0.json
uv run continual-arc schedule make --config arc1 --seed 0 --no-epilogue --reset-each \
  --out arc1-ws0-reset.json
uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --out arc1-held-out0.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --out arc1-seen0.json

Play the stream, then give each test its own copy of your learner as the stream left it, in a run directory that holds the stream's alias-salt, so that it meets the stream's tasks under the names it learnt them by. A learner that holds a model client or a process saves what it keeps instead, and loads it into a new instance for each test, and a harness plays its tests with arc-baselines test, as Step 04 shows.

import copy
import shutil
from pathlib import Path

from continual_arc.runner import run
from continual_arc.schedule import Schedule

from my_pkg.learners import MyLearner

learner = MyLearner()
run(learner, Schedule.load("arc1-ws0.json"), out_dir="runs/arc1/my-learner-ws0")
for test in ("held-out", "seen"):
    out = Path(f"runs/arc1/my-learner-{test}")
    out.mkdir(parents=True)
    shutil.copy("runs/arc1/my-learner-ws0/alias-salt", out)  # the names the stream gave its tasks
    run(copy.deepcopy(learner), Schedule.load(f"arc1-{test}0.json"), out_dir=out)

Each test is read as the share of its reset's spend your learner no longer pays, against itself reset before each of the same instances: the same model, in the same harness if it has one, which plays the test one instance at a time, --width 1, on its own. transfer prints the share with its 95 percent interval, T, the points of the budget saved, and the budget used and share solved on each side, the absolute view. learned prints the share learned over the stream, your run against your learner wiped before every instance of the same stream, the schedule --reset-each made: the share of the budget left to learn that it saves, from 0 for a learner that keeps nothing to 1 for perfect memory.

PYTHONPATH=. uv run python play.py
uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --width 1 --out arc1-held-out0-1.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --width 1 --out arc1-seen0-1.json
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-ws0-reset.json --out runs/arc1/my-learner-ws0-reset
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-held-out0-1.json --out runs/arc1/my-learner-reset-held-out
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-seen0-1.json --out runs/arc1/my-learner-reset-seen
uv run continual-arc transfer runs/arc1/my-learner-held-out runs/arc1/my-learner-reset-held-out
uv run continual-arc transfer --test seen runs/arc1/my-learner-seen runs/arc1/my-learner-reset-seen
uv run continual-arc learned runs/arc1/my-learner-ws0 runs/arc1/my-learner-ws0-reset

The bare models are no reference, but a verified result shows them beside the shares: the stateless floor of the baselines, each model alone with nothing kept, played on the two tests one instance at a time. A learner that played worse whenever its memory was empty would score for it against its own reset, so a test on which your reset plays below a bare model that costs no more is flagged, and so is one on which your learner plays no better than such a model, since one that only switched to a stronger model once its memory was filled would score too. We play them in a verified run. To see them beforehand, run them from the baselines:

export OPENROUTER_API_KEY=...
uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --width 1 --out arc1-held-out0-1.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --width 1 --out arc1-seen0-1.json
MODEL='{"model": "your/model", "max_tokens": 128000, "end": "finish"}'
EFFORT='{"reasoning": {"effort": "medium"}}'
uv run arc-baselines run --learner continual_arc_baselines.systems.bare:BareModelLearner \
  --learner-kwargs "$MODEL" --llm-defaults "$EFFORT" --schedule arc1-held-out0-1.json --out runs/arc1/bare-held-out
uv run arc-baselines run --learner continual_arc_baselines.systems.bare:BareModelLearner \
  --learner-kwargs "$MODEL" --llm-defaults "$EFFORT" --schedule arc1-seen0-1.json --out runs/arc1/bare-seen

The paper's harness runs played the stream, 200 instances, in 4 to 45 hours and 1 to 45 US dollars of model calls on GPT-6 Luna, as the tracking proxy counted them. The two tests add 200 instances to your learner's run, its reset on the stream 200 more, and its reset on the tests 200 more.

Step 06

Check your numbers

To check them, fork unifyai/continual-arc and copy every run directory the runner wrote to results/arc1/, each under a name of its own: your stream, your two tests, your learner reset on the stream and on each test, and the bare model's tests. Then add a submission.json to your stream's directory, beside its five files, naming your reset's stream as reset and each test's three runs, run, reset and bare, under tests.

{
  "name": "my-learner",
  "learner": "my-learner 1.0, with the pairs it buys kept between instances",
  "student": "openai/gpt-6-luna",
  "effort": "medium",
  "teacher": null,
  "code": "https://github.com/you/my-learner",
  "contact": "you@example.com",
  "reset": "results/arc1/my-learner-ws0-reset",
  "tests": {
    "held-out": {"run": "results/arc1/my-learner-held-out", "reset": "results/arc1/my-learner-reset-held-out",
                 "bare": "results/arc1/bare-held-out"},
    "seen": {"run": "results/arc1/my-learner-seen", "reset": "results/arc1/my-learner-reset-seen",
             "bare": "results/arc1/bare-seen"}
  },
  "notes": "anything a reader needs to know to read the number"
}

scripts/check_result.py rebuilds every run's schedule from its configuration, test and seed, and checks that its records cover every slot and that its summary is the one the records give. It checks that your tests were played under your stream's task names, that your reset is a run of your learner and that the bare runs are the bare model's. Then it prints your numbers on the public stream, both tests' shares with the budgets beside them, with a flag on any test where your reset plays below the bare model. It calls no model and no network.

mkdir -p results/arc1
cp -r runs/arc1/my-learner-* runs/arc1/bare-* results/arc1/
# write results/arc1/my-learner-ws0/submission.json, then
uv run python scripts/check_result.py results/arc1/my-learner-ws0

On their own these numbers are a claim, since the public tasks, their answers and the stream's solvers could be brought along, so nothing goes on the leaderboard until your learner is verified. To have them stand beside your verified result, open a pull request with the run directories and give it on the form in the next step. Questions go to the repository's issues.

Step 07

Have it verified

A verified result is one we have rerun ourselves on Continual-ARC's private set: 50 new ARC-style rules that nobody outside unify has seen, each a generator of fresh instances, and each checked against every public ARC task. 25 of them are streamed as arc1 streams its tasks, and 25 are held out for the frozen held-out test after it. Claude Opus 5.5 at its highest reasoning effort solves every one of them, nearly every time, and GPT-6 Luna fares on them at every effort as it does on the public stream. For now this verifies a learner at one user, and we will add rules for the collective regimes.

When your learner is ready, open an issue with the Verify my learner form. It asks for:

• The system. Its name and a contact.
• The code. Its public repository and the commit to run, and the learner's import path: a Python class that implements the benchmark's learner protocol, such as your SystemLearner adapter, installable with pip from the repository and built with no arguments, so a harness's settings, such as end, are its class's defaults. A harness depends on continual-arc-baselines @ git+https://github.com/unifyai/continual-arc@v0.6.0#subdirectory=baselines alone, which brings the benchmark with it.
• The models. Every published model it calls on OpenRouter, each with its reasoning effort: the one it plays with, the one that teaches it if one does, and any others.
• A budget. What its model calls may spend, in whole US dollars from 5 to 1,000, and the country you pay from, with the name on the invoice if you want one.

Once the issue is checked we post an invoice on it, to pay by card on Stripe's page: the budget, OpenRouter's 5.5 percent fee on buying that credit, and Stripe's fee. We make a key for the run on our own OpenRouter account, capped at the budget, which pays for your learner, its reset and each model you name played bare, and your code never holds it. Once it is paid and a runner is free, the run starts, one at a time on each runner.

We rerun your learner on the private set at a fresh seed we do not disclose, in a sandbox on a machine of its own, which has no public network and holds neither the rules nor the key, with every model call going through our tracking proxy. After the stream, your learner plays the two frozen tests from where the stream left it, fresh instances of the 25 rules it met and one instance of each held-out rule, one instance at a time: before each one we restart the machine it runs on and put its working files back as the stream left them, so what it carries from the stream to the tests is what it keeps as files. Your learner reset plays the same stream wiped before every instance, for the share learned, and the same test instances one at a time, wiped before each, for each test's share. Each of its models plays the test instances bare, and a test is flagged when your reset plays below one of them that costs no more, or below the cheapest, when your learner plays no better than one that costs no more than it, or when your learner sends a larger share of its calls to a dearer model than its reset does. A run takes hours to days, as on the public stream.

The result goes on the issue and on the leaderboard, marked verified: both tests' shares with their 95 percent intervals and the budgets beside them, the share learned on the held-out stream, and what the model calls cost. Its intervals are wider than the paper's rows, which average four plays a side where a verified run plays each side once, and its rules are the private set's, so its point is read beside the public stream's, never as the same measurement. Shares are rounded to a hundredth, points to a whole point and dollars to a whole dollar. Then we delete the key.

You pay ahead for the model calls, and we pay for the machines. What the run does not spend goes back to your card, less Stripe's fee. The paper's harness runs on GPT-6 Luna cost 1 to 45 US dollars on the public streams, and the same harnesses wiped before every instance 2 to 37, so a budget of about 100 dollars suits a model of Luna's class, and a stronger model needs more. A person, and a card, may have 10 runs in a season.

A learner that trains its weights is verified by arrangement, its training on GPUs we rent, since a machine of your own would keep the instances it trained on. We cannot verify a system whose code cannot be shared.

The rules

What a learner may use

The stream is public, and the answers to its tasks are too: Re-ARC ships a solver for every ARC-AGI-1 training task, and the evaluation tasks' outputs are published. A learner that brought them along would use none of the budget without learning anything. So a number on the public stream is a claim, and it becomes a result only once we have rerun its code on the private set, whose rules are not public.

• Nothing about ARC brought in. No ARC, Re-ARC, ARC-GEN or other task data, answers or solvers inside the learner, its prompts or its starting memory, and no rebuilding of the schedule. What a model learned in pretraining is its own prior, which your learner reset on the same model already carries.
• Published models that start as released. Every model your learner uses is a published model, used as released and named exactly: an OpenRouter model id, or open weights as their Hugging Face repository at a commit, org/name@commit. A model made to start badly would save a large share of its reset's spend just by recovering, and its absolute budget would show it ending where the published model starts.
• Every model named. Every model the learner calls is named in the submission and reached through the baselines' tracking proxy. Its shares are read against your learner reset, which calls the same models, and each model plays the tests bare beside them. A model that only teaches it, such as a stronger model labelling its games for training, is part of its training: those calls count in its training compute, shown beside the score.
• Learning only from what it was shown. A learner learns from its observations and feedback. The run's records, with their real task ids and exposures, are the benchmark's, not the learner's.
• Anything else is allowed: memory, programs, tools, a cache, training the weights. Compute is logged and shown beside the score, never in it, and a learner that trains shows its training compute apart from what it costs to serve an instance.