GE0
GLM-5.3 Flash in Prime Agent: given the expert guidance, wiped before every instance.
- Stream, share learned
- 0 its own reset
- Seen test, share saved
- −0.06 −0.24 to 0.07
- Held-out test, share saved
- −0.01 −0.15 to 0.12
The stream
Its runs go public with the paper, and its stream is drawn here once they do. Its numbers on the two frozen tests are below.
The two frozen tests
The learner is frozen and plays two tests, each a single round, and each is read against its reset on the same instances: the seen test, fresh instances of the tasks the stream serves, which is memorization, and the held-out test, 100 of ARC-AGI-1’s evaluation tasks it never met, which is generalization.
| Budget used | Solve rate | |||||
|---|---|---|---|---|---|---|
| Test | reset | learner | share saved | reset | learner | times as many |
| Seen test | 17.1% | 18.2% | −0.06 −0.24 to 0.07 | 99.0% | 98.0% | ×0.99 ×0.96 to ×1.01 |
| Held-out test | 17.9% | 18.1% | −0.01 −0.15 to 0.12 | 96.5% | 96.5% | ×1.00 ×0.97 to ×1.04 |
Budget used is the share of the budget of 16 an instance used, averaged over the test, so lower is better, and an unsolved instance uses all of it; the solve rate is the share solved within it. Share saved is the share of its reset’s budget used that the learner no longer uses, and times as many is its solve rate over its reset’s. Small type is the 95 percent interval over task resamples. It keeps nothing between instances, so on each test half its plays stand for its reset and the other half for the learner: how far chance moves a learner. The leaderboard has every experiment’s numbers side by side.
The learner
- Student
- GLM-5.3 Flash
- Teacher
- None
- Harness
- Prime Agent
- Concurrent users
- 1
- Reset
- Itself: it keeps nothing between instances, so its share learned is 0.