Committee Work
Two duel networks with different starts reached the same ceiling, and a committee of the two broke through it.
Two duel networks with different starts reached the same ceiling this week, and a committee of the two, then a network taught to play like it, broke through. The week began on the four-player game, where last week’s stand-in run was still training, and moved to the two-player duel (1v1) by its second day. Brian judged that the four-player committee and stand-in were still research results, not ready to serve, while the duel work kept giving clean measurements. Our best duel network, the soft-label student after 50 updates of self-play (below), wins 84.15% of 4,000 games against Rehex (1v1, where 50% is even), playing one forward. Our best single four-player network, the stand-in run at update 350, wins 49.40% one seat against three Rehex with nobody trading, where fair share is 25%. Brackets are 95% intervals.
The four-player run that stopped
The stand-in run trained our human-imitating network in one seat against three stand-ins, with nobody trading. Most of its gain came in the first 25 updates. After that it went flat on every table we measured, including two tables of held-out human models. A learning-rate cut and mixing a human-model teacher into the opponents’ seats added a little at the end. At update 350, one seat against three Rehex, it beat sp7@200 by +2.45 pp [+0.61, +4.29] over 4,000 games.
A side run tried to teach faster wins by paying a winner less for every round the game lasted, at 0.95 a round. Its registered test needed the median winning round to fall by one. It stayed at 18. The plain run’s wins got about as much quicker from strength alone. The discount was too weak to matter: a round cost about as much reward as a 2.5 pp drop in win share. Pricing the incentive against the main objective before launch would have shown that a setting this mild makes the whole run a no-op.
Committees
Early in the week the question was whether networks that play equally well also make the same mistakes. I measured how far apart five of our four-player networks’ move preferences were. They grouped by lineage. A committee of three equally strong networks from different lineages won 58.35% one seat against three stand-ins with nobody trading, +3.88 pp [+2.01, +5.74] over sp7@200. Adding a member from the same lineage, or a weaker one, gave no statistically significant gain. Brian raised whether we had tried this before and written it off as a search effect. The records showed the plain average had never been played. It had been ruled out by argument. Against Rehex, the committee’s edge over the best single network was +0.33 pp [−1.48, +2.13], not statistically significant.
The idea paid off more clearly in the duel.
A duel network from nothing
On 1 November we started a duel network from scratch, meaning from random weights, with no human games, no trading and a smaller input built for two seats. For 125 updates it won almost nothing. Then it climbed steeply:
Table
| training update | from random weights | f50 (human start) |
|---|---|---|
| 10 | 0.45 | 79.15 |
| 25 | 0.85 | |
| 50 | 0.43 | |
| 75 | 1.38 | |
| 100 | 1.73 | |
| 125 | 2.3 | |
| 150 | 5.25 | |
| 175 | 5.75 | |
| 200 | 12.22 | |
| 225 | 16.35 | |
| 250 | 19.68 | |
| 275 | 28.45 | |
| 300 | 31.75 | |
| 325 | 37.3 | |
| 350 | 41.1 | |
| 375 | 49.15 | |
| 400 | 58.37 | |
| 425 | 61.3 | |
| 450 | 63.3 | |
| 475 | 67.3 | |
| 500 | 69.27 | |
| 525 | 70.87 | |
| 550 | 72.3 | |
| 575 | 75.32 | |
| 600 | 75.5 | |
| 625 | 75.65 | |
| 650 | 76.62 | |
| 675 | 77.28 | |
| 700 | 78.3 | |
| 725 | 78.73 | |
| 750 | 78.95 | |
| 775 | 79.0 | |
| 800 | 79.0 | 79.15 |
One forward, nobody trading. 4,000 games a read, paired by board; f50 read on the same boards. 95% intervals for the comparisons are in the text.
At update 725 it scored 78.73%, against 79.15% for f50, our human-started duel network: −0.43 pp [−2.01, +1.16], not statistically significant. At update 775 the two played each other in 8,000 duels, and the scratch network won 50.18% [49.15, 51.20]. Two networks with different starts, inputs and histories had stopped at about the same height.
Two things from the climb carry to other projects:
- Probes before guesses. A linear probe found that the part of the network that scores bank trades knew its own seat’s exchange rate no better than chance. We added each seat’s production and exchange rates as inputs. The next 100 updates gained +3.08 pp [+1.57, +4.58]. That is a before-and-after comparison, so it cannot separate the new inputs from more training.
- Check what a knob controls. We had been growing games per update to match the measured gradient noise scale. Brian pressed on why a run was so slow. Each optimiser step still used 65,536 positions, so the growth didn’t reduce gradient noise; it only made the data fresher. We removed it and fixed 4,000 games an update.
The committee as teacher
The two networks played equally well, so Brian proposed a committee of them. It won 82.95% [81.80, 84.10] against Rehex, +3.80 pp [+2.39, +5.21] over f50 alone. Brian then proposed playing our own duel networks against each other to build our own training games. We used the committee as the teacher for 100,000 duels and trained one network to imitate it. A student trained only on the moves the committee played reached 81.53%, and 50 updates of self-play took it to 82.80%.
A committee’s value is in its soft labels: its whole spread over the moves says how close it judged the alternatives, not only which one it picked, and that is what two networks add over one. Trained on the spread, the student’s divergence from the teacher fell from 0.254 to 0.063. Fifty updates of self-play from that start beat the same from the played-move start by +1.35 pp [+0.11, +2.59] against Rehex. The learning rate also differed between the two runs, so that comparison is confounded.
Table
| estimate | low | high | n | |
|---|---|---|---|---|
| from random weights @725 | -0.43 | -2.01 | 1.16 | 4000 |
| distilled student | 2.38 | 0.94 | 3.81 | 4000 |
| student + 50 updates RL | 3.65 | 2.25 | 5.05 | 4000 |
| committee of the two | 3.8 | 2.39 | 5.21 | 4000 |
| soft-label start + 25 RL | 4.85 | 3.41 | 6.29 | 4000 |
4,000 games per arm, paired by board with f50 (79.15%); 95% intervals over board pairs.
By the end Rehex had stopped separating the strongest networks. The self-play network and committees with and without it all scored within 0.15 pp of each other against Rehex, though the self-play network beat the old committee head to head, 52.25% [50.91, 53.59]. Brian suggested d110 as a second benchmark. Against it, self-play from the soft start won 62.90%, +2.02 pp [+0.38, +3.67] over the played-move start.
One decision, two code paths
The same bug shape turned up three times this week.
- Live trading. In four-player games against people online, our seat struck robber deals with one approval rule and paid them through another, stricter one. Sometimes it refused to pay a deal it had made. The fix is one approval path for every stage.
- Search. A move is built from several sub-choices (piece, then location). When our duel search skipped a move, it played each sub-choice’s top option in turn instead of the network’s best whole move. That cost −3.00 pp [−4.62, −1.38] against Rehex. No live seat was affected.
- Training. A new duel network design, trained from scratch on the committee’s soft labels, was trained to choose setup roads one way and served choosing them another, so the road choice it used at the table was never trained. Playing it one move at a time recovered +3.20 pp [+1.96, +4.44] against Rehex.
Search, and why Rehex still wins
Search on the distilled network after 50 updates of self-play won 89.35% against 83.56% without it, +5.79 pp [+1.85, +9.72] over 216 board pairs, against Rehex at 1v1. We stopped it early: a searched game cost about 205 CPU-seconds, against 0.4 for one forward. Searching only the setup gained +7.20 pp [+5.15, +9.25] in duel self-play but +2.45 pp [+1.21, +3.69] against Rehex. Brian had predicted that gains would compress near the top against Rehex. Brian also raised how Rehex still wins at all: 688 of 4,000 games against the distilled network. Dice luck explains much of it, since two-thirds of those wins came in games where our network’s dice ran more than three cards behind. Even if the network won every game where its dice were level or better, it would win 88.3%, short of the 95% against Rehex (1v1) the project has aimed at since 22 October; the 99% bar in an earlier post was set against the old Rehex.
Against people online, f50 with search won 11 of 15 duels. In four-player games against people online, one seat against three (fair share 25%), sp7-scratch, our four-player network trained from random weights, won 29 of 90, and the committee seat 9 of 21.
Last post’s predictions
- The stand-in run ends within 2 pp of sp7@200: inconclusive. It ended +2.45 pp [+0.61, +4.29] ahead. The point estimate is just past the line, and the interval spans it. It did climb early and stall, as predicted.
- Its gain against Rehex stays smaller than against the stand-ins: held through update 300, then failed. At update 300 the gains over its start were +11.22 against the stand-ins and +9.32 against Rehex. At 350, after the teacher joined the opponents, they were +9.53 and +10.53 [+8.56, +12.49].
- A human model in the rollouts adds at most 2 pp over sp7 copies: untested.
Next, and what I expect
On the last day Brian moved the four-player game back to the front, to try the recipe that broke the duel ceiling: a committee teacher, a large set of its games with soft labels, then training.
- A four-player network distilled from the three-network committee comes within 1 pp of the committee, one seat against three stand-ins with nobody trading.
- Full search on the soft-label duel network adds at least 4 pp, against Rehex (1v1), over 2,000 games.
- No duel network passes 86% playing one forward against Rehex at 1v1 by the end of next week without a new teacher.