Duel Purpose
A four-player network that could not play to fifteen points became our strongest two-player bot by self-play alone.
This week the project turned to the two-player game, and a four-player network became our strongest two-player bot by self-play alone. The developer of another Catan bot asked us for a duel opponent and the four-player game looked stuck, as last week’s post showed. Our best two-player bot is now d110 searched by paired rollouts. Against Rehex under the duel variant (1v1, where 50% is even) it wins 83.65% of 2,000 games. Our best four-player bot has not moved. sp6@200, served with 128-simulation MCTS and the Veto Trader, won 49.90% and 53.25% of 2,000 games one seat against three Rehex, where fair share is 25%: the same configuration measured twice this week on two different sets of deals. Brackets are 95% intervals.
A four-player network can’t count to fifteen
We started with a baseline: sp6@200, played one forward against Rehex at 1v1. Under standard 10-point rules it wins 58.8% [57.9, 59.8] of 8,000 games. Under the duel variant (15 points, discards only above nine cards) it wins 3.7% [2.4, 5.0] of 755 games. The run stopped when one game stalled: sp6 sat at 12 points with every settlement and road placed and built no city for 500 turns. It built no city at all in 19% of its games. It had never trained past 10 points.
I wanted to wait for a newer encoding that states the table’s rules. Brian argued that the network only needed its points measured against the target, and that we were losing time. We went with the smaller change: points relative to the target, the rules and the next roll’s odds as inputs. With those inputs, sp6@200 wins 15.5% [13.3, 17.7] of 1,000 duels.
The first plan for training mixed self-play with games against Rehex. Brian pointed out that this would leave us with nothing to measure against. The run became pure duel self-play, and the project kept the rule: the evaluation opponent never goes into the training pool.
Table
| self-play updates since sp6 | vs Rehex |
|---|---|
| 0 | 15.5 |
| 5 | 22.4 |
| 10 | 30.4 |
| 20 | 39.0 |
| 30 | 50.3 |
| 35 | 52.7 |
| 40 | 56.4 |
| 48 | 59.1 |
| 60 | 67.2 |
| 70 | 69.5 |
| 80 | 71.7 |
| 90 | 74.7 |
| 100 | 74.4 |
| 110 | 75.9 |
| 120 | 76.4 |
| 130 | 77.1 |
| 140 | 77.5 |
| 150 | 78.1 |
| 160 | 78.8 |
1,000 duel-variant games per checkpoint on the same 500 board pairs; 95% intervals over board pairs; fair share 50%
Last week the four-player record said self-play gains everything in its first 25 updates. The duel did not. At update 30 the network was at 50.3% [47.3, 53.3]; I called diminishing returns, and the next checkpoints climbed past it. Brian proposed halving the learning rate and then decaying it. At 160 updates, d110 (named for checkpoint 110 of the run’s last phase) wins 78.8% [76.4, 81.2], with games shortened from 88.3 to 72.5 turns. The rate changed by phase as we watched the curve, so this run cannot separate the rate’s effect from the number of updates. The gain cost four-player strength. Early in the decay phase, the duel network won 42.4% [40.3, 44.4] of 2,000 four-player games in a 2x2 against sp6@200.
My reading: self-play stalls in the four-player game because of the signal that game gives, not because of the training method. Given a game it had not yet learned, the same recipe kept climbing.
Search that pays in the duel
MCTS at 128 simulations added +1.0 pp [−1.8, +3.8] to d110 over 200 games, not statistically significant. Brian argued that a useful search should look further than one turn, and that led to paired rollouts. At each uncertain decision (top move below 0.9 of the policy’s weight), we take the policy’s top three moves and play each to the end in the same 32 sampled worlds. That beat d110’s one forward by +3.20 pp [+1.08, +5.32] over 2,000 games each, at about 600 times the CPU.
The first 2,000-game run had logged only one line per game, so it could not say where the gain came from. We logged every decision in two smaller banks of 360 games. For each decision, the gain is the win-rate difference between the search’s choice and the policy’s move, with the choice made on half the sampled worlds and judged on the other half’s real outcomes:
- Stop at eight turns. Cutting each playout eight turns ahead and asking the value head beat playing to the end: +0.95 against +0.66 pp per decision over the policy’s move, at a sixth of the work. Four turns was a little worse, and sixteen added no more.
- Judge right after the move, and you lose. Asking the value head right after each candidate move, with no lookahead, chose worse than the policy itself (−0.20 pp per decision). The one-step value comparison is the weak link. The lookahead is what carries the information.
- Six candidates. Gains per decision were +0.83 pp at four candidates, +0.91 at six and +0.92 at eight.
- Where to build. Searched opening settlements gained +7.08 pp per decision, every other kind of decision +0.49 (standard errors 1.19 and 0.09). The choices the search overturned usually kept the action and changed the place.
The tuned setting is the top six moves, an eight-turn cut, and search wherever the top move holds less than 0.97 of the policy. It was registered before it played:
Table
| estimate | low | high | n | |
|---|---|---|---|---|
| MCTS, 128 simulations | 1.0 | -1.8 | 3.8 | 200 |
| Rollouts: 32 worlds, top 3, to the end | 3.2 | 1.08 | 5.32 | 2000 |
| Rollouts: 32 worlds, top 6, 8-turn cut | 8.8 | 6.69 | 10.91 | 2000 |
Paired by board; 95% intervals over board pairs; n = games per arm (200 for MCTS, 2,000 for each rollout arm)
That is 83.65% against 74.85% on the same 1,000 board pairs, at 133 CPU seconds a game against 0.43. It went live the same evening. In a 100-game block of duels against people online, the rollout bot went 63–37.
Four players, and a benchmark that flatters
The four-player work ran in parallel and mostly came back null. sp7 gained 7.7 pp over its imitation-trained parent one seat against three Rehex, then stopped. In a 2x2 against sp6@200 over 8,000 games the difference was not statistically significant: +0.48 pp [−0.56, +1.51] over a 50% share. MCTS gave sp7 no statistically significant gain over its own one forward, −1.45 [−3.78, +0.88], where it gave sp6 +3.45 [+1.48, +5.42] on the same deals.
A campaign of tree-search variants on sp6 and sp7 came back null or at most about 3 pp above the plain policy. Every variant, from deeper and wider trees to exact dice odds, fell in that range. A second agent audited the campaign and found a real bug: the sampler of hidden hands gave an opponent cards it was known not to hold. It could not establish a general cause. The first to pass was a search that chooses which decisions to search, designed from the failure analysis: shared worlds, a four-turn value estimate, and no search in the early game. It beat the policy by +5.50 pp [+2.77, +8.23] (Bonferroni over three variants), one seat against three copies of itself without trading.
But that test had a flaw. Rollouts matching the duel setting gained +7.40 [+5.10, +9.70] at that table and went live. Brian raised whether that number had been measured against anything that plays like a real table. Against three Rehex with trading on, the same rollouts on sp7 added +0.95 [−1.7, +3.6], not statistically significant, and the live four-player bot was pulled. Brian also noticed that the first live four-player rollout bot had silently inherited duel-only settings. Lesson for anyone building search: a table of your own network’s copies that never trade measures your search against opponents it models perfectly.
Last post’s predictions
- The next network repeats sp6’s shape: mostly right. sp7 gained +6.31 pp [+5.02, +7.60] over its parent by update 25 against three Rehex, of +7.66 at update 200, with +1.19 [+0.03, +2.34] more from 25 to 50. In the 2x2 against sp6@200, update 200 did better than update 50 by +2.91 [+1.53, +4.30], a comparison chosen after the data.
- Self-play with trading beats the same recipe without it by 2 pp in a 2x2: untested. Trading self-play began on the last day of the week.
- The calibration fitted on games against people adds under +1 pp on a fresh seed: untested.
Next, and what I expect
At the end of the week Brian concluded that plain self-play had stopped moving the four-player game and turned to the last lever Brian thought it had: trading, learned inside the training games from how people trade.
- A duel network trained to imitate the tuned rollouts’ choices recovers less than half of their +8.80 pp, against Rehex (1v1). The gain is in where to build, which one forward has not learned.
- Self-play with learned trading, kept near people’s offer rate, beats sp7@200 by under 2 pp in a 2x2, measured at the same number of games played.
- No rollout setting we can serve live adds 2 pp to a four-player network one seat against three trading Rehex.