No Signal
Our best four-player network had no measurable learning signal left, in self-play or against Rehex.
All week the target was the four-player game. Last week’s post closed with self-play stalled and trading as the last lever. This week asked what the stall was made of. The answer: the network had nothing left to climb, against either opponent we could measure it on.
Our best four-player network is still sp7@200. One seat against three Rehex, with nobody at the table trading, it wins 46.95% of 4,000 games, where fair share is 25%. sp8 scored within 2 pp of sp7 at the same checkpoints, which is not statistically significant. Brackets are 95% intervals.
A benchmark that moved by itself
Brian raised whether the three Rehex had been trading among themselves in our measurements. They had, and “no trading” had meant three different things in our records:
- nobody trades;
- every seat trades;
- our seat makes no offers while the three Rehex trade freely. That last one ran 7,821 trades among the Rehex in 2,000 games.
That third setting costs sp7@200 about 7 pp (37.60% against 45.00%). Most of our earlier “no-trade” measurements were that third kind, including the self-play curves against three Rehex in the last two posts, sp6’s and sp7’s. A rule now forbids putting two numbers side by side unless they were measured on the same Rehex code. Changes to Rehex alone had moved one network’s score by 3 pp: sp7@25 scored 40.88% on one version and 37.83% on a later one.
The benchmark is now two tables at every checkpoint, 4,000 games each: nobody trades, and our seat silent while the Rehex trade. A table where everyone trades is measured only for questions about trading. The lesson for anyone measuring against a fixed bot: when the opponent’s own features change the outcome, fix them first, and stamp each measurement with the code it ran on.
Nothing left to climb
The week opened with a self-play run whose network had its own trade head. After 50 updates, with the head trading, it won 40.65% one seat against three Rehex. Its moves alone trailed sp7@25 by 2.65 pp [−4.42, −0.88]. Brian’s view since the start has been that leaning on trading is a trap: train without it and add trading afterwards as a bonus. I agreed the evidence now backs that view.
Then the question underneath: is there anything left for self-play to learn? We measured sp7@200’s learning signal directly, on about 880,000 self-play decisions and on 578,000 decisions from 12,000 games against three Rehex. Both estimates came out slightly below zero: no consistent direction measurable (the estimator reports no interval). The same measurement on a run still improving, sp7-scratch at update 75, found the consistent part made up 0.44 of a 65,536-decision batch’s squared gradient, so it does detect signal when there is some. sp7@200’s policy gradient has no measurable consistent direction against either opponent, so mixing in Rehex-like opponents offers it nothing to follow either. Value heads from different generations ranked positions almost identically (rank correlation 0.92 to 0.96), and averaging five of them did no better than the best one.
Starting from nothing
With the machine otherwise idle, Brian proposed running sp7’s recipe from random weights, with no human games at all. The result is sp7-scratch. At first it learned slowly. Brian pointed out that our earliest pure self-play had beaten Catanatron half the time. Measured again, that old checkpoint scored 4.96% against three Rehex, close to scratch’s 4.80%. Gains came in two phases:
- Raising the step size, lowering λ. Raising the learning rate together with a shorter credit horizon (λ from 0.95 to 0.9) gained +3.56 pp [+2.54, +4.59] in a 2x2 against the unchanged run. Neither change alone helped.
- Cutting the step later. Later, smaller steps broke a stall near 35%.
Table
| self-play update | sp7-scratch | sp7@200 | imitation parent |
|---|---|---|---|
| 25 | 3.02 | 46.95 | 37.77 |
| 50 | 4.8 | ||
| 75 | 7.05 | ||
| 100 | 10.67 | ||
| 125 | 16.78 | ||
| 150 | 22.18 | ||
| 175 | 26.73 | ||
| 200 | 27.5 | ||
| 225 | 29.27 | ||
| 250 | 30.43 | ||
| 275 | 34.2 | ||
| 300 | 33.5 | ||
| 325 | 35.33 | ||
| 350 | 34.93 | ||
| 375 | 36.25 | ||
| 400 | 38.25 | ||
| 425 | 38.5 | ||
| 450 | 39.75 | 46.95 | 37.77 |
4,000 games per read on the same deals; Wilson 95% intervals shaded. Flat lines: sp7@200 and sp7's imitation-trained parent on the same yardstick. Fair share 25%.
The run ended at update 450, its horizon after one extension from 400, winning 39.75% with nobody trading. The curve’s end looks like it was still climbing, but the last 50 updates added +1.50 pp [−0.14, +3.14], which is not statistically significant, and every gain after update 350 followed a cut in the step size. The 39.75% is +1.98 pp [+0.11, +3.84] over sp7’s imitation-trained parent and −7.20 [−9.07, −5.33] under sp7@200. A fitted ceiling of 31.5–33% was overtaken by the next measurement, and a flat one at update 200 by a jump at 275. One flat measurement is not a plateau.
Brian raised whether the two runs sit in separate basins. We blended the two networks’ weights after matching their hidden units:
- Between sp7 and scratch: a valley about 35 pp deep, with the 70% blend winning 0.75%.
- Between two sp7 runs from one start (the control): flat, 47.15% to 46.55%.
That is the shape of two basins. It could not be pinned down cleanly, because the two runs also began from different random seeds. Brian called that a seed-hygiene failure, and a later audit found about 280 distinct seeds across 2,051 evaluations. Evaluation and training now draw from a short fixed list, and overriding it takes an explicit bypass.
Training scratch against a pool of our other networks made it worse: −2.70 pp [−4.43, −0.97] after 25 updates. The registered measurement at 50 was −1.70 [−3.48, +0.08]. The run also raised the learning rate, so pool and step size are confounded. Search helped a little: with the served trader, one seat against three trading Rehex, a tree search over scratch added +1.80 pp [+0.32, +3.28] over 4,000 games. Against people online, one seat against three and mostly with search, it won 11 of 40. Its arena rate with search, 42.12% against three trading Rehex, would mean about 17. Whether a network with no human prior handles people worse is untested.
Who plays the other seats
This produced the week’s one large number. Brian proposed going after the moves where our plateaued networks disagree. We built a search that settles those disagreements by playouts, with three Rehex as the opponents inside the playouts, and, one seat against three Rehex with nobody trading, it beat sp8’s one forward by +7.55 pp [+5.03, +10.07] over 2,000 games. Brian argued that putting the real opponent inside the search was the whole effect. I conceded. The control, with sp7 copies in the playouts, came to +0.52 [−3.01, +4.05] over 961 games before it was stopped.
So the gain was opponent modelling. Next we trained a small human model and put it in the opponents’ seats inside sp7@200’s rollouts:
Table
| estimate | low | high | n | |
|---|---|---|---|---|
| sp7 copies in playouts | 2.05 | -0.52 | 4.62 | 2000 |
| human model in playouts | 3.6 | 1.06 | 6.14 | 2000 |
| human model vs sp7 copies | 1.55 | -0.98 | 4.08 | 2000 |
| Rehex itself in playouts | 7.55 | 5.03 | 10.07 | 2000 |
| same, sp7 in playouts | 0.52 | -3.01 | 4.05 | 961 |
Paired games on the same deals; 95% intervals. Top three rows: sp7@200 searched with paired rollouts, over its one forward unless stated. Bottom two: playouts at moves where our networks disagree, over the base network's one forward (sp8, then sp7 for the control, stopped at 961 games).
The human model beats one forward by +3.60 [+1.06, +6.14]. Its edge over sp7 copies, +1.55 [−0.98, +4.08], is not statistically significant. Giving the model an estimate of each opponent’s style made no statistically significant difference. The exact opponent inside the search is worth about 7.5 pp; realistic models of people are worth a fraction of it. Live, a version searching with a human model won 6 of 20 against people, one seat against three.
The stand-in
On the 28th Brian made the case that beating Rehex was never the aim. The aim is the best people, who will not play millions of training games. So we built a stand-in, a table of human models with a trader fitted to people’s, and on the last evening started training against it: our network trained to imitate elite human games, in one seat against three stand-ins. After 7 updates, with nobody trading, it gained +5.75 pp [+3.23, +8.27] against the stand-ins on 2,000 games. It also gained against a held-out larger human model (+3.55 [+0.99, +6.11]) and against Rehex, which it never trains against (+4.40 [+2.03, +6.77]). That took it from 37.55% to 41.95% against three Rehex, still about 5 pp under sp7@200’s 46.95%, so after 7 updates it had not raised the ceiling against Rehex. Whether it will is next week’s question.
Last post’s predictions
- A duel network imitating the tuned rollouts recovers less than half their +8.80 pp (1v1): untested.
- Self-play with learned trading beats sp7@200 by under 2 pp in a 2x2: not run as stated. One seat against three Rehex, the trade-head run’s moves trailed sp7@25 by 2.65 pp [−4.42, −0.88].
- No live-servable rollout setting adds 2 pp one seat against three trading Rehex: consistent with this week’s data. Search on scratch added +1.80 [+0.32, +3.28]. With nobody trading, human-model rollouts added +3.60 [+1.06, +6.14], but that table is not the one predicted.
Next, and what I expect
- The stand-in run ends within 2 pp of sp7@200 one seat against three Rehex with nobody trading. It climbs early, as every run from an imitation start has, and then stalls at the same height.
- Its gain against Rehex stays smaller than its gain against the stand-ins, measured at the same update.
- A human model in the rollouts never adds more than 2 pp over sp7 copies one seat against three trade-free Rehex, whatever its size.