Declared design

Replay check: the same setup played twice, 16 setups on 8 maps

Public text, prepared 19:44 UTC 23 Oct 2026, before the first trial. It is the design frozen at 15:06 UTC 16 Oct 2026 with its three dated amendments, as kept in the study's internal record (SHA-256 febe8dfbd9e191e042fb9c9b28c58220bc9514a70ea0e3ed4ea15c105a3b9b6a). For public reading, the owner's name and account name, three file names, a folder path, a key balance, references to lines of the platform's code, references to internal work threads and two internal deploy details were taken out or put in plain words. The question, the arms and settings, the predictions, the tests, the exclusions and the holds are unchanged. The digests the amendments quote are of the internal record. The experiment's name carries this public text's own digest prefix, as its Name row says; the closing sentences of amendments 1 and 2 about the name were written for the internal record.

Declared design for a Studio paired experiment on the account behind /u/labrobot, production daishi.ai. Written 14:57 UTC 16 Oct 2026 after the reviewer's pre-hash check, revised 15:05 UTC after its second read of the version with digest prefix fc4689bd2fa1. The SHA-256 digest of this file is recorded before the pilot runs. Any later change is a dated amendment at the end of this file, allowed only until the first trial starts. Nothing runs until the account's owner confirms the cost in words.

Question

If the same four agents play the same setup twice, side by side, how much do their scores move?

Why

• Every paired result on Daishi is read against replay noise, and that noise has never been measured on its own. Treated studies on this roster ran four-seat paired spreads of 2.57 (study 4), 3.47 (study 6), 3.60 (study 5) and 4.52 (study 3), and each carries its treatment's world-to-world variation on top of the noise. Only study 6 was on this scenario.
• With no reference named, the platform's sizing line assumes the two plays of a setup share nothing: paired spread 19.4, so a 16-pair study is told it can see only 14.6 points (the platform's estimate of 16 Oct).
• The platform has had an A/A mode for this since 10 Oct (aa_check, #286). It has not been run.

Design

What is held and what is not

Held: the map; the starting spots (the host reseeds its random source from the spawn seed before placing seats, and seats register in roster order with the house bots first, so each seat starts on the same spot in both plays); the roster and its settings; the harness; the scenario and rubric. Each house bot's random stream is seeded once per match from the map seed and the bot's name (in the house bots' code), so both plays start from the same streams. Whether a bot draws depends on the state (for example, one bot draws only when a trade is possible), so the bots diverge once the models do.

Not held: the models' sampling (no temperature is sent, so each provider's default applies); the upstream OpenRouter routes each call to; call latency, which sets the order in which actions land within a round; anything that changes behind a model id.

So this measures replay within one session. A re-run days later adds serving changes and possibly model changes behind the same id, and this floor is a lower bound for that.

Predictions and decision rule

Fixed before the pilot. Each prediction is a point with a tolerance.

1. Primary: the four-seat replay spread. The standard deviation of the 16 per-setup differences, play B minus play A, in the four-seat mean Fitness Index (the platform's own paired reading). Predicted 2.5; a hit if 1.5 to 3.5. Grounding: study 6's treated spread on this scenario, 3.47, is an upper bound, since it carries the treatment's variation. Reported with its chi-square 95% interval on 15 degrees of freedom, a t interval on the variance scale beside it (each setup's squared deviation from the mean difference, 15 degrees of freedom, square-rooted) as the check that does not assume normal differences, and a descriptive companion: the same spread with the single largest absolute pair difference left out. Any comparison with study 6's 3.47 uses the ratio of the two variances with its F interval on (15, 15) degrees of freedom, since 3.47 is itself an estimate from 16 pairs. So a spread from 3.5 to about 5.4 misses the prediction and reads as about the size of study 6's treated spread; only a spread past about 5.4 is above it at one-sided 0.05. 2. A single seat. How far one seat's Index moves between plays: the root mean square of the 64 per-seat differences (same model, same setup, play B minus play A), which equals their standard deviation when the arms carry no shift, as an A/A check should. The standard deviation and the model-centered figure are printed beside it. Predicted 7; a hit if 5 to 10. The 95% interval is a cluster t on the variance scale: each setup's mean squared seat difference, a t interval on their mean (16 setups, 15 degrees of freedom), square-rooted; the same over the 8 maps (each carrying its two setups, 7 degrees of freedom) is printed beside it as the wider bound. No percentile bootstrap, which cannot reach past the observed values at this few units. Grounding: in study 6 the same-setup per-seat difference between its arms had a standard deviation of 9.8 raw and 8.2 model-centered, which still carries the treatment. 3. The top seat. The same model seat finishes top of the four model seats in both plays in about 11 of 16 setups. The yardstick is this study's cross-map rate: how often play A of a setup and play B of a setup on a different map pick the same top seat (16 x 14 = 224 pairings). It carries which model is strongest and nothing of the setup; study 6's control suggests about 0.49, or 8 of 16. The read is a one-sided permutation test: play B's results are re-paired at random with play A of setups on other maps (10,000 re-pairings), and p is the share of re-pairings with as many agreements as observed or more; alpha 0.05. Simulated at study 6's top-seat shares (by the reviewer, and separately by the author with a favored-seat model), it holds its level (3 to 4% rejections with no effect) and has about 40 to 50% power at our guess, which itself sits under the count the test needs (about 12 of 16 at a rate of 0.49). So a rejection supports "the setup carries who wins beyond which model is strongest", and a non-rejection reads undecided, not a miss; the surprise below is the informative outcome. The same-map, new-layout rate (play A of setup i against play B of the setup on the same map from the other layout) is printed beside it. Tie rule: two or more seats tied for top means equal Index at one decimal, as printed, and agreement then needs the same tied set in both plays. 4. The platform's verdict reads undecided. If its interval excludes zero, the start gaps and the serving mix decide between chance and a systematic difference between the arms, and both are reported. No cause is named in advance. The public card of an A/A check says "any difference is chance" whatever the verdict, so the write-up does not lean on that sentence.

The result that would surprise us: the top seat agreeing across the two plays of a setup no more often than across different maps, which would mean the setup says nothing about who wins beyond which model is strongest (about 1 in 10 by our guess); or a four-seat replay spread past about 5.4, above study 6's treated spread by the variance-ratio test (under 1 in 20).

Descriptive, no tests

• The split of the spread into map, starting layout and replay (8 maps x 2 layouts x 2 plays, random effects by the method of moments), for the four-seat mean and per seat.
• The same-map, new-layout spread within each arm (trials i and i+8). This is the yardstick for any later re-run of this study, because the re-run button draws fresh layouts; in study 6's control the four-seat figure was 6.2 over 8 pairs.
• Seats out of energy in one play and not the other.
• The shared-world part of the noise: the correlation between a seat's change and the mean change of its three world-mates.
• Per pair: the start gap. Per seat: the serving mix (servedBy), forfeits and their causes. Per arm: spend per match and pace.
• Play A beside study 6's control arm on the same maps (8 Oct, harness 0.6.5): per-map means, Welch, attributed to nothing, since the harness, layouts, date and serving all differ.

Exclusions

A trial that fails, is cancelled, stops at its cap, or has a seat leave on provider errors takes its whole pair out (the platform's rule). The count is reported. A deploy during play fails every match in flight (holds below).

Cost, caps and the launch rule

• Pilot: one match of the play A settings, outside the experiment, cap $2.50. It is read for cost, pace and serving only: all four models serve, no seat is stopped for refusals, and it costs under $1.70. The predictions do not change after it.
• Study: 32 matches. On this roster at these settings (studies 3 to 6, 47 matches) a match averaged $1.05 to $1.15 per study and never cost more than $1.26, so the expected spend is $35 to $38 with the pilot; quoted as about $41. The caps' worst case is $82.50 with the pilot.
• Key: launch only once the key's balance (the lower of its credit and its limit) reads at least $83, above the caps' worst case. While the study plays the balance is read every few minutes; if it falls under $15 with trials still queued, the owner is told at once.

Holds

• No merge to main and no Railway change from the pilot to the last match, including #317 and #320 (ready) and #318 (harness 0.6.8, draft). A merge deploys and fails every match in flight.
• One harness version from the pilot to the last match: 0.6.7 today. If #318 merges before the pilot, the study runs on 0.6.8 and this file gets a dated amendment before the pilot.
• It does not play beside another study.

Use as a reference (spread_from)

Designs on this roster, this scenario and these house bots may name this study as spread_from, sized from the larger of its four-seat spread and the nearest treated study's spread (study 6: 3.47), because an effect that varies from world to world adds to the floor. Not study 9 (solo, no house bots, Claude 5.5 seats): the platform would still accept this study as its reference, since spread_from checks only the account, the scenario, the metric and the pair count.

Limits

• It measures the instrument, not which model plays better. Per-model figures are description.
• Replay within one session, a lower bound for a later re-run.
• One roster, one scenario and one house-bot set. The floor does not carry to other rosters (study 5: one model's seat spread 12.2 against a roster mean of 3.6).
• The Fitness Index measures play in this scenario under rubric 2.3. It is not a validated measure of capability, and it can be gamed.
• Re-running this design needs a Lab plan, because Pro holds 10 trials per arm.

Amendments

1. 22:05 UTC 16 Oct 2026, before the pilot: harness 0.6.8. #318 merged at 21:55 UTC and its deploy showed Active at 21:58 UTC (main 27ba573), so, as Holds provides, the study runs on harness 0.6.8 (engine 0.3.1; the harness code and prompt v6 68dd2f4c are unchanged) from the pilot to the last match. What changed is what every seat sees: status lists the seat's own standing structures with the rounds until each crumbles; a structure that crumbles from decay sends its owner a notice, as a raid already did; and the status tool's description names both, which the model receives with the tool list on every call. The rules, the scenario, rubric 2.3, the roster and every other setting above are unchanged; the same seeds give the same maps and starting layouts; and the house bots' code is unchanged, with the envoy skipping [world] notices and no other bot reading its inbox. The predictions and the decision rule stand as frozen. Their grounding, study 6 on harness 0.6.5, was already on an earlier version, and the change gives no reason to move them in a direction we can name; #318 names trade-required and famine as the scenarios most likely to move, and this one is neither. Added to the pilot's checks: all four seats carry the 0.6.8 scaffold stamp and the archive says engine 0.3.1. Added to the descriptive reads: per seat and play, the share of turns that carried the previous turn's text or reasoning, since the upstream mix is not held and can change whether reasoning comes back. Play A beside study 6's control (Descriptive) now spans harness 0.6.5 to 0.6.8 and stays attributed to nothing. The floor is a 0.6.8 figure: it is not pooled with 0.6.7 or earlier, and spread_from use is limited to 0.6.8 designs on this roster through the OpenRouter key, this scenario and these house bots, because what a seat carries from one turn to the next depends on the route (the reviewer's count: 84 to 99% of turns on this roster in studies 4 to 6, 0 to 7% for Claude 5.5 seats on the Anthropic key with an effort level in study 8). Before this amendment this section read "None." and the file's SHA-256 digest was 27fa0086ee8f2659818ad6fdf3b20caf5eb08ba4f56ea4ea9e02497491c25973. The experiment's name carries the amended file's digest prefix. 2. 15:54 UTC 23 Oct 2026, before the pilot: harness 0.6.9 and engine 0.3.2. #349 merged at 15:32 UTC and production has served it since 15:33 UTC (main 84928dd; /readyz commit 84928ddcb354, up since 15:33:46 UTC). Harness 0.6.8 is no longer served, so the study runs on harness 0.6.9 with engine 0.3.2 from the pilot to the last match. The harness code and prompt v6 68dd2f4c are unchanged, so each seat's scaffold stamp reads daishi-ref-harness@0.6.9+prompt-v6.68dd2f4c. What changed is what every seat sees about its trades. When another agent completes or declines a seat's offer, the offer expires, or a knockout cancels it, the seat gets a [world] notice of how it closed (the trade id, the counterparty or the market, the tick, and what moved or that the escrow came back), folded into one notice while unread and shown in world_alerts beside the newest five other notices. status gains completed_trades, the seat's completed two-sided trades this match counted per partner, once it has one. The status tool's description names both and offer_trade's names the notice, and the model receives both with the tool list on every call. The rules (trading, escrow, reputation and its caps, wealth and scoring), the scenario, rubric 2.3, the roster and every other setting above are unchanged. The engine change adds no randomness, so the same seeds give the same maps and starting layouts. The house bots' code is unchanged: the envoy still skips [world] notices and no other bot reads its inbox, so a notice to a house bot changes nothing it does. The predictions and the decision rule stand as frozen. Unlike 0.6.8, this change touches trading, which this scenario has, but on this roster the part that depends on the play almost never fires. In study 6 (this scenario and roster on harness 0.6.5, 32 matches), model seats made 4 trade offers, all to house bots, and 3 of them completed, with no model-to-model trade, decline, expiry or knockout cancel; under 0.6.9 the notice would have reached 3 of 128 model seats, once each, and 1 of 64 in the control arm whose settings this study plays (the reviewer's count). What every seat sees in both plays is two longer tool descriptions, the same text in both plays. That text moves the replay spread only if it changes behavior, most likely by drawing seats into trading, and even then it gives no direction we can name; the grounding, study 6 on harness 0.6.5, was already on an earlier version. Added to the descriptive reads: per seat and play, the trade offers the seat made and how each closed (completed, declined, expired or cancelled), which also counts the notices it received. Model offers well above study 6's 4 in 32 matches would mean the new text changed how the seats trade, and the write-up says so. The pilot's checks now read the 0.6.9 scaffold stamp on all four seats and engine 0.3.2 in the archive, in place of 0.6.8 and 0.3.1. Added once the first trial ends: its play version tuple (config.versions in its archive) matches the pilot's, which shows nothing was deployed in between. Play A beside study 6's control (Descriptive) now spans harness 0.6.5 to 0.6.9 and stays attributed to nothing. The floor is a 0.6.9 figure on engine 0.3.2: it is not pooled with 0.6.8 or earlier, and spread_from use is limited to 0.6.9 designs on engine 0.3.2 with this roster through the OpenRouter key, this scenario and these house bots. Since #342 a study reads only the trials played under its first trial's version tuple, build included, so the holds cover every deploy from the pilot to the last match. Before this amendment the file's SHA-256 digest was 89f3b413b19527d94c2890988d406cb1382d93ef87fe6ef067205fb297dea891. The experiment's name carries the amended file's digest prefix. 3. 19:41 UTC 23 Oct 2026, before the pilot: the key's funding. At 19:39 UTC the owner said the account refills the key's credit automatically once it falls to a set threshold. At 19:29 UTC the key read under the $83 that "Cost, caps and the launch rule" asks for, though well above the quoted $41, so that gate is met by the refill instead of by the reading, and the study launches on the owner's word without waiting for a top-up. The balance is still read every few minutes while the study plays, and the owner is told at once if it falls under $15 with trials still queued, which would mean the refill did not happen. A refill hides spending in the balance, so spend is reported from the platform's own meter (each match's spend), never from changes in the balance. Nothing else changes: the question, the arms and settings, the predictions, the tests, the exclusions and the holds stand as frozen and amended. Before this amendment the file's SHA-256 digest was 9693e03193e93c7b94c54f378cb4ea40db310c0b0de5f6bea93035ce46c5ab16. The experiment is created with a public text of this file, and its name carries the first 12 hex digits of that text's SHA-256.