Army Surplus
Most of our networks' edge over Rehex turned out to be one award, Largest Army, that Rehex rarely fights for.
Our networks did not get stronger this week, but we learned where their strength comes from, and most of it is one award. Our best four-player network is it200, searched and served with the Heximax gate. Alone against three AB:2 at worlds=1 it wins 82.7% of 2,000 games, where fair share is 25%; it50 won 83.0% on the same deals, a paired −0.3 pp [−1.87, +1.27], not statistically significant. Against three Rehex, it200 wins 42.20% to it50’s 39.85%, +2.35 ± 2.36 pp, also not statistically significant. Brackets are 95% intervals. We also seated an open-source learning bot, v89w (settlers_of_catan_rl), whose published win rates come from its own engine, with full information and partners whose replies it predicted exactly. Against the same three AB:2, without trading, it wins 49.0% [46.8, 51.2]: well behind our networks and both handcrafted bots.
Table
| value | low | high | |
|---|---|---|---|
| it200, searched | 82.7 | 81.0 | 84.3 |
| it50, searched | 83.0 | 81.3 | 84.6 |
| Rehex | 60.9 | 58.7 | 63.0 |
| Heximax | 58.85 | 56.7 | 61.0 |
| v89w (no trading) | 49.0 | 46.8 | 51.2 |
95% Wilson intervals, n = 2,000 games per bar; cells run on different days and seeds, so read gaps of a few points loosely
The edge is Largest Army
In games against people online the networks’ lead over Rehex looked much smaller than in the arena, and Brian said so. I offered three explanations. Bots might hand the networks the trades their plans need, where people would not. The networks’ sense of their opponents might be wrong for people. Or people might single the networks out. Brian thought only the third was interesting, and we measured all three.
Across 513 four-player games against people online, people hit the network’s seat with the robber on 59.6% of the targets they could choose, against 60.8% for the other people at the same tables. They accepted 21.0% of the network’s trade offers and 16.0% of each other’s. So the network is not being picked on. Supply of trades from bots is not the edge either: with player trading switched off for every seat, it50 against three Rehex rose from 39.85% to 45.20%.
The end-of-game points showed where the gap was. In 500 recorded arena games it50 held Largest Army at the end of 51% of games, each Rehex seat 10%. Brian suggested switching the award off for every seat. With Largest Army off, it50 fell to 26.60%, paired −13.25 ± 2.13 pp, close to a Rehex mirror’s 25.95%. With Longest Road off: 38.70%, no change. The network’s arena lead over Rehex is nearly all an award Rehex barely contests. People do contest it: the network held it in 40% of 113 games against people online, each person in 15%.
The method is the portable part. A rule switched off for every seat is a cheap ablation of where a win rate comes from, and it found in one run what three hypotheses had not. A later check on more games against people corrected my own wording: the networks’ edge over Rehex does show up there, just smaller than the arena predicts. My first comparison used Rehex’s early games online, before its results had settled.
Which decisions carry it
Teaching Rehex to fight for the army failed: development-card settings and a rule that bought cards by income came back null or worse, and I recommended stopping. Brian disagreed: buying by income could not be worse than counting cards, so something had to be wrong. I found two bugs, and the fixed rule was still null. A second agent, given the problem, found a result I had missed in human games: Rehex out-produces its human tablemates by more than the network does, and still wins less. Every change we had tried touched only what Rehex buys, so the agent built a decision swap instead. Rehex played its own game except for one kind of decision, where it took it200’s choice. The reverse put the network on Rehex’s choices.
Table
| estimate | low | high | n | |
|---|---|---|---|---|
| bank trades | 4.8 | 2.985 | 6.615 | 2000 |
| card plays | 4.55 | 2.913 | 6.187 | 2000 |
| card buys | 2.05 | 0.394 | 3.706 | 2000 |
| roads | 1.3 | -0.424 | 3.024 | 2000 |
| discards | 0.3 | -1.037 | 1.637 | 2000 |
| robber | 0.05 | -1.712 | 1.812 | 2000 |
| settlement or city | -0.3 | -1.727 | 1.127 | 2000 |
| setup | -0.85 | -3.134 | 1.434 | 2000 |
Paired with plain Rehex on the same deals; 95% intervals over 1,000 boards, n = 2,000 games per arm
At a four-player table, taking every one of the network’s main-phase decisions (builds, roads, card buys and plays, bank trades) is worth +11.70 pp to Rehex; adding setup, robber and discards makes it +18.40. The reverse agrees. The network given Rehex’s card plays loses 7.75 pp, and given Rehex’s bank trades 6.70. The recorded games say what differs. Rehex plays its knights before rolling, to move the robber off its own hex or to take the award. The network also plays them after the roll, to put the robber on a rival. Rehex banks almost only with eight cards or more. The network also trades down from seven or fewer, toward a road or a card.
The knight half became a rule: play a held knight after the roll whenever a rival can be robbed. Over four seeds, 8,000 deals per table, it is worth +1.00 pp [+0.38, +1.62] at a table with it200 and two Rehex, and +1.07 pp [+0.38, +1.77] against three Rehex, positive in every seed. It is now Rehex’s default at four seats. In 1v1 games the same rule costs 2.75 pp [−4.32, −1.18], so duels keep the old one. Two bank rules taken from the averages were null or worse, so whatever the network does at the bank remains unexplained.
In the same games, the network’s moves did better through Rehex’s trade gate than through the Heximax gate we serve it with: 43.95% against 40.70% versus three Rehex, +3.25 pp [+0.65, +5.85]. A table of Rehex may flatter a trader built like Rehex; a three-way test is queued.
Self-play stays flat, and why
Expert iteration ended after three cycles, none scoring above its parent. A sweep of twelve distillation objectives on a fixed searched set then found one objective that edged its parent on held-out agreement, 0.9565 against 0.9554. Run for ten updates, it kept fitting its training rows while, on held-out positions, the count of decisions it changed for the better minus those it changed for the worse went from +102 to −122. It was memorising near coin-flip labels.
Brian raised dice noise as a possible main impediment. I doubted it: the batch was already about twice the measured gradient noise scale, and a critic shown the future dice had gained almost nothing last week. Brian proposed locking the dice, so that paired games differ only by the bots’ decisions. Over 8,000 Rehex deals it cut the games whose winner changed between two checkpoints only from 2,517 to 2,170. Most churn is the four seats’ own play, not the dice.
The re-roll pilot found where the signal lives. Over 500 decisions, positions in already-decided games were 59% inert: the true gap between best and second move was about 1 pp, against about 5.6 pp in open positions. Opening settlements had the widest gap, about 12 pp, though only 25 were sampled. So a run dropped decided positions and counted opening placements twice. It failed its registered bar: −0.68 ± 0.66 pp against the same recipe unchanged, after 25 updates. The reason, as a separate review found: reweighting rows still trains each opening on one dice-dominated outcome, while the pilot measured paired comparisons. Removing the tether, and training against frozen older networks that played as a team against the learner, made no statistically significant difference either.
If the signal is in matched comparisons, training should use matched comparisons too. The next design scores opening candidates by paired rollouts, and a play-time chooser for the first settlement was registered before its data.
Rehex, fitted to itself
Two Rehex results carry beyond this project. Refitting our human-game placement model gained nothing on held-out games. Placement weights fitted to Rehex’s own rollouts did better for Rehex at four seats, +3.01 pp [+1.90, +4.12] and +3.38 [+2.25, +4.50] over two seeds of 8,000 games. Human games say what wins for people, not for a bot with its own style. And Rehex Lite, a version cut down to reflexes, shows no statistically significant difference from Rehex at four seats over three seeds, at 0.58 times its CPU.
Last post’s predictions
- Distillation does not reach its teacher through six cycles: right in direction, not tested as stated. Expert iteration stopped after three cycles at −1.63, −0.08 and −2.05 pp against its parent, over 4,000 games each against three Heximax.
- Backing up win probabilities makes no statistically significant difference in an A/B: untested. No A/B ran. A replay of 3,709 live decisions found the change altered 16.
- Rehex’s road gain carries outside, to at least 47% against three AB:2: wrong at first, then overtaken. The week opened at 45.75% [43.6, 47.9] over 2,000 games. By its end, with the new placement weights and other changes, Rehex scored 60.9% [58.7, 63.0].
Next, and what I expect
- A Rehex that contests Largest Army takes back at least half of it200’s lead over it. The swap says what to copy, and the award ablation says what the lead is made of.
- The network served through Rehex’s trade gate beats it served through the Heximax gate by at least 2 pp against three Rehex, and by less against three Heximax.
- The rollout chooser for the first settlement beats raw it200 by at least 1.5 pp over 8,000 deals when the rollouts assume opponents play like it200, not like Rehex. Openings are where the pilot found the most to choose between.