Veto Power
Letting the network veto a rule-based trader's offers, rather than choose its own trades, was the week's largest gain.
This week was about the four-player game, and mostly about trading: letting the network veto a rule-based trader’s offers, rather than choose its own trades, was the largest gain. Our best four-player bot is now sp5@200 (names of this kind mean self-play run 5 at update 200), served with a 128-simulation search and the Veto Trader. Alone against three AB:2 at worlds=1 it wins 80.1% of 1,000 games [77.6, 82.6], where fair share is 25%; it200 won 82.7% at the same setting last week, on other deals. Against three stock AB:2, which sees every hand, it wins 83.3% [81.0, 85.6]. AB:2 declines every player trade in our engine, so that table cannot show the trading gains below, and against it the networks did not get stronger. Brackets are 95% intervals.
From 9 to 12 October, with the site’s permission, we played six runs on colonist.io. There it200 with the Veto Trader won 27 of 50 ranked four-player games at tables averaging about 1646. A fit to those games alone puts its settled rating, where its rating would level off, at 1956 [1825, 2078].
Veto, don’t choose
Last week it200 won more through Rehex’s trade gate than through the Heximax gate we serve it with. When I called Rehex’s gate plus a value-head check “the right solution”, Brian objected that best-so-far is not a reason and asked why it worked. Three tests answered.
First, the value head gets a trade’s direction right but overstates its size. Let it choose the best of a menu of plausible trades, and it picks the trades whose value it overstates most. The same head picking from a short rules menu lost 2.89 to 4.24 pp to the head merely vetoing one rule-made ask, over 8,000 rotated games. That is the winner's curse, and it bites anywhere a learned estimate takes the maximum of many noisy candidates.
Second, a tournament of eight traders, all moving the same it200 policy without search:
Table
| value | low | high | |
|---|---|---|---|
| Value veto on Rehex Lite's trader | 27.57 | 26.22 | 28.99 |
| Value veto on Rehex's gate | 27.19 | 25.9 | 28.52 |
| Value veto on Heximax's trader | 25.93 | 24.56 | 27.19 |
| Rehex Lite's trader | 25.57 | 24.23 | 26.88 |
| Rehex's gate | 25.55 | 24.25 | 26.92 |
| Network's own gate | 25.12 | 23.86 | 26.44 |
| Heximax's trader | 23.57 | 22.25 | 24.78 |
| No trading | 19.5 | 18.31 | 20.67 |
4,200 games per trader out of 8,400; each table seats four different traders; 95% percentile bootstrap intervals over games
Removing the network’s veto from the Rehex-gate trader cost 2.35 pp [−3.85, −0.85]; removing any one of Rehex’s trade rules cost at most 0.85.
Third, fourteen batches of rotated tables built a trader around the veto, one rule at a time. Brian set the limits: no hand-written special cases, and three “bot tells” (share of turns with an offer, offers per such turn, unrelated counters) kept near human levels. It asks for the card its nearest purchase lacks, gives its most-held spare, prefers the partner with the fewest expected points and never trades with a seat within a point of winning.
Served the way it plays live, with it200 at 128 simulations one seat against three Rehex (the version before mid-week), the Veto Trader wins 45.60% to the Heximax trader’s 39.55% on the same 2,000 deals: +6.05 pp [+3.62, +8.48]. Against three Rehex Lite, trading is worth +14.95 pp [+12.61, +17.29] to that configuration, and search only +0.75 [−0.96, +2.46], not statistically significant.
Trading at people’s rate cost wins. People offer on about 30% of their turns, the Veto Trader moving sp5@200 without search on 41.6%; raising the veto’s floor brought that to 24.1% and cost 5.05 pp [−7.09, −3.01] against three Rehex. Ordering offers by a fitted model of whether people accept made no statistically significant difference in a 2x2 against the shipped trader, +0.08 [−0.36, +0.52], and Brian closed the question. Reading our own win chance through a calibration fitted on our games against people gained +2.10 pp [+0.44, +3.76] against three Rehex, but it was one of four comparisons made with no correction for multiplicity.
The right seat, the wrong information
On another seat’s turn, the gate asks the value head about a position where someone else is to move. The network had never learned its “whose turn” input, because in self-play the position is always read by the seat to move. I proposed the mover's frame. On a battery of give-away trades it saw what our own frame missed: it priced handing a rival the road card toward Longest Road at −0.16 pp [−0.29, −0.03], where our own frame gave zero. It shipped the same morning. A registered A/B then found it lost 1.25 pp [−2.06, −0.44] to our own frame over 8,000 games at a table of three sp5@200.
Brian argued the frame is right by construction, so the implementation must be at fault. Three tests said otherwise:
- Showing that frame our own hand still lost 1.04 pp [−1.85, −0.22], and so did averaging over frames.
- Declining offers at random at the same volume cost nothing (+0.16 [−0.50, +0.83]). So the mover’s frame was turning down the wrong offers, not just fewer of them.
- With every hand visible, a seat’s win chance differed between its own frame and the mover’s by 8.5 pp (standard deviation), more when it held development cards, which the mover’s frame cannot see by kind.
Our own frame went back as the default, and Brian made it a standing rule: the bot uses everything a live seat knows.
Self-play, and a benchmark with a ceiling
sp5 ran 204 untethered self-play updates from it200. One seat against three Rehex, every checkpoint scored below it200 or not significantly different from it (−1.70 to −0.01 pp, each about ±1.2), yet in a 2x2 against it200 sp5@200 led by +1.70 pp [+0.67, +2.73].
The new encoding came next: it gives the network each seat’s road distance to every building spot, and opponents’ hidden cards as expected counts. A network trained on it from scratch to imitate human games won 2.85 pp [+0.49, +5.21] more often than clio-r4, one seat against three Rehex. sp6 is self-play from that network:
Table
| self-play updates (0 = the imitation-trained parent) | sp6, one seat against three Rehex |
|---|---|
| 0 | 28.55 |
| 25 | 35.48 |
| 50 | 35.77 |
| 75 | 36.0 |
| 100 | 36.0 |
| 125 | 36.84 |
| 150 | 36.11 |
| 175 | 36.34 |
| 200 | 36.21 |
One forward pass, trading off, 8,000 paired deals per checkpoint; checkpoint minus parent about ±1.25 pp (95%, normal over board pairs)
From update 25 to 200 it moved +0.74 ± 1.23 pp. I recommended stopping. Brian ran it to 200 and asked for a 2x2 against sp5@200 with search and trading at every seat. sp6@200 won by +2.39 pp [+1.36, +3.41]. On the same deals, though, sp6@200 beat sp6@125 by +2.25 [+0.85, +3.65], and search added only +0.89 [−0.30, +2.07]. The last 75 updates carried the gain, and the one-against-three Rehex measurement did not see it: a fixed rules opponent is a benchmark with a ceiling, and a 2x2 against the network to beat caught what it missed. sp6@200 went live on the last evening.
Brian then raised whether all of self-play’s gain comes in the first 25 updates. On our large-batch runs the records say it has (sp5 never gained). My reading now: within about 25 updates the policy agrees with what its value head can rank, and that head explains about 0.25 of outcome variance however it is fitted. What has moved strength came from outside self-play: imitation before reinforcement, trading, the veto.
Rehex, renamed, and one duel
Rehex Lite replaced Rehex mid-week and took its name: from here on, Rehex means the former Rehex Lite. Its largest gain was a lower bar for asking toward purchases more than one card short: +4.11 pp [+3.15, +5.08] one seat against three Lite over 8,000 games, but offering on 57% of its turns. Brian capped that at 40%; asking also for one of two cards missing toward a city kept +1.89 [+1.19, +2.58] at 38.1%. In a 2x2 against Heximax, Lite’s side won 55.25% [53.1, 57.4] of 2,000 games. On colonist.io, Lite’s two four-player runs won 14 of 50 each; pooled, they settle at 1604 [1503, 1699].
The week’s one turn to the two-player game checked the renamed Rehex: the duel arena found no statistically significant difference from the old, +0.43 pp [−0.58, +1.43] over 8,000 games. Against people on colonist.io (1v1) it went 26–24, eight wins by the opponent leaving, against opponents averaging 1600.
A one-turn lookahead over Rehex’s duel options, scored by a win chance fitted to Rehex’s games, predicted outcomes well (AUC 0.853) and still did not beat Rehex. Predicting winners from Rehex’s own positions is not ranking moves Rehex would never make.
Last post’s predictions
- A Rehex that contests Largest Army takes back half of it200’s lead: untested.
- The network through Rehex’s gate beats the Heximax gate by at least 2 pp against three Rehex: not statistically significant. Rehex’s gate alone scored +1.55 pp [−0.21, +3.31] over 4,000 games; with the value veto on top, +3.85 [+2.11, +5.59], which became the Veto Trader. The Heximax half was not run.
- The first-settlement rollout chooser beats raw it200 by 1.5 pp with it200 as the opponents’ stand-in: wrong. It lost 1.85 ± 1.05 pp over 8,000 deals. With the true opponents in the rollouts it gained 2.36 ± 1.02.
Next, and what I expect
- The next network, which reads every seat through the same weights, repeats sp6’s shape: all its gain against three Rehex within 25 self-play updates, more in a 2x2 against its predecessor after that.
- Self-play with player trading inside the training games beats the same recipe without it by at least 2 pp in a 2x2, both served with the Veto Trader.
- Reading our own win chance through a calibration fitted on games against people does not hold up. Its +2.10 pp [+0.44, +3.76] against three Rehex was one of four uncorrected comparisons. On a fresh seed at a registered size I expect under +1 pp.