QC
Qwen3.5-9B: the demo cache, which shows again the examples it paid for, in place of the harness.
- Stream, share learned
- 0.31 0.19 to 0.45
- Seen test, share saved
- 0.37 0.26 to 0.51
- Held-out test, share saved
- −0.01 −0.05 to 0.03
The stream
Its runs go public with the paper, and its stream is drawn here once they do. Its numbers on the two frozen tests are below.
The two frozen tests
The learner is frozen and plays two tests, each a single round, and each is read against its reset on the same instances: the seen test, fresh instances of the tasks the stream serves, which is memorization, and the held-out test, 100 of ARC-AGI-1’s evaluation tasks it never met, which is generalization.
| Budget used | Solve rate | |||||
|---|---|---|---|---|---|---|
| Test | reset | learner | share saved | reset | learner | times as many |
| Seen test | 66.5% | 42.0% | 0.37 0.26 to 0.51 | 40.5% | 60.2% | ×1.49 ×1.27 to ×1.81 |
| Held-out test | 73.2% | 73.6% | −0.01 −0.05 to 0.03 | 32.1% | 32.5% | ×1.00 ×0.89 to ×1.14 |
Budget used is the share of the budget of 16 an instance used, averaged over the test, so lower is better, and an unsolved instance uses all of it; the solve rate is the share solved within it. Share saved is the share of its reset’s budget used that the learner no longer uses, and times as many is its solve rate over its reset’s. Small type is the 95 percent interval over task resamples. The leaderboard has every experiment’s numbers side by side.
The learner
- Student
- Qwen3.5-9B
- Teacher
- None
- Harness
- None
- Concurrent users
- 1
- Reset
- QF, the model alone, keeping nothing.