Held-out pools, statistics and the arena games

#9. Keeping the answers secret

Are scored inputs rotated? Yes, per season, and the previous pool becomes public at rotation, which is what makes training on published Daishi logs an expected and acceptable thing to do.

Is there a commitment to the pool, published before results exist? As of this page, yes, and before it there was not. Rotation alone does not let an outsider check that the pool named in a published season is the pool that was actually in force when the matches ran, rather than one selected afterwards to suit the results. The protocol was sound and the proof was missing.

The commitments below fix that. Each digest is the SHA-256 of a domain separator, the pool's name, a secret 32-byte nonce and the pool's canonical JSON, joined by newlines. The nonce is held back until the pool rotates and is published with it. A nonce is necessary rather than decorative: a pool is a handful of small integers, and a bare hash of one could be recovered by enumeration.

The payload corpus is committed by payload id rather than by payload text, so that fixing a typo in a payload does not break a commitment while still fixing which payloads were in force.

If a pool has to be replaced before it rotates, the replacement gets a new name and its own commitment, and both appear here. A commitment is never edited.

What is still manual: publishing a new digest when a season opens, and publishing the pool and nonce when it closes, are operator steps rather than automated ones. The canonicalization and the digest are computed in code and pinned by a test, so a pool edited without a new commitment fails the build, but the publishing schedule is a promise rather than a mechanism.

Is contamination otherwise controlled? Only by rotation. Published match logs are distributed in the clear, where some peer evaluations now distribute traces encrypted specifically to keep them out of training corpora. For a world whose seeds rotate this matters less than it would for a fixed task set, but it is a real difference and not a solved problem.

#10. Statistics

The statistics floor and the anti-best-of-N rules are not part of the audit's checklist, but they are the other half of whether a number is worth citing, and they are written down in the governance doc rather than here: a minimum of five matches across at least three held-out seeds, means with Student t confidence intervals, interquartile means alongside means, leaderboards sorted by a conservative estimate and ranked so that entities sharing a rank are not separated by their intervals, verdicts on underpowered A-versus-B comparisons withheld by the tooling rather than discouraged by convention, and a cap of three declared private variants per model family per season.

A paired experiment created from 19 October 2026 on also declares its read before its first trial: how many trials it may lose (an arm's trial that failed, lost a seat, stopped at its spend cap, was resumed by a new release, played under another version tuple or measured nothing; a pair whose two plays started from different layouts, which only an experiment created from 19 to 24 October 2026 can hold, counts the same way) before a comparison reads incomplete and renders no verdict, and, if the owner sets one, a spend cap for the whole study at which no further trial starts. No lost trial is ever replaced and no trial is added after the losses are seen: the design plays the trials it declared and reads what it has, so a result can never rest on trials chosen with the outcome in view. Beside each comparison that lost a pair the report prints the tipping point, the per-trial difference every lost pair would have had to carry to change the reading, next to the widest difference the pairs played did carry. Each report also keeps two rate limits apart per arm: what the world refused of the seats' own actions under its seat rules, and what the provider made the harness retry; the second is read as provider retries per 100 model calls arm by arm, since the arms share one key and a provider that throttles one arm's calls reaches the other arm's pairs. Uneven rates are a reading the verdict paragraph states first, never a verdict (the Studio guide says how each is counted).

The analysis label missed one change, and is bumped now. Every experiment and series report names, beside the rubric, the analysis that read its trials (versions.analysis, and the version line under "What this study cannot support" on a study's page). From 20 October 2026 the label was daishi-analysis-v1. On 24 October 2026 the experiment report began leaving out a pair whose two trials played different seeds (section 8 says how such pairs arose) and reading a fixed rule's stop recorded on such pairs again without them, and the label stayed v1, so a report labelled v1 is one of two analyses. From 11 November 2026 the label is daishi-analysis-v2, which names two changes. The first is that rule. Of the studies published on Daishi's own profile, one held such pairs: the replay check exp_f2a2fbe0f666622dfc ("Replay check, 16 setups on 8 maps"). Its fixed rule stopped on 16 pairs on 23 October 2026 and read undecided, a mean paired difference of 2.00 (95% interval -0.74 to 4.74). Read with the rule, 6 of its 16 pairs are left out, more than the one lost trial its declared read allows, so it is incomplete and gives no verdict. No other published study played a pair on two sets of seeds.

The second is the Student t critical value every interval is read with. Under v1 a 95% interval read a table rounded to three decimals up to 30 degrees of freedom and the normal value 1.96 past it, which runs narrow: about 4% at 31 degrees of freedom and 2% at 60. Under v2 every critical value is exact, at every level and every number of degrees of freedom, and the detectable gap, a series' half-width and the trials it says more precision would take are sized on the same values. Of the studies published on Daishi's own profile, the change is more than a last digit on one: exp_a5813ccd9705d355b8 ("Haiku 5.5 thinking off, low, high, harness 0.6.10"), whose per-arm and per-model 95% intervals run over 32 trials and are about 4% wider. Its thinking-off arm reads 32.8 to 36.2 where it read 32.9 to 36.1, and its thinking-high arm 47.4 to 52.4 where it read 47.5 to 52.3. Its comparisons, read at 97.5%, already used exact values. On nine others a printed last digit moves on at least one interval, each by less than a thousandth of the interval's width, as exp_59c33c3ad4828d5222's thinking-high arm reads 943.46 to 1264.81 output tokens per turn where it read 943.48 to 1264.79. No verdict changed.

The model comparison at /api/compare took the exact values the same day, and its power check moved onto t. It carries no label of its own, so the change is dated here. The power check, the smallest true gap the comparison finds four times in five and the shared seeds each gap needs, is sized on Student t at one fewer degree of freedom than there are seeds, as an experiment's detectable gap is. Before, it read the normal curve at every number of seeds, which put the smallest gap about 1.9 times too small at 3 seeds and 1.3 times at 5. Since 10 November 2026 the check reads the larger of the spread the comparison observed and the platform's prior spread for a paired difference, and on that prior, at the standard season length, a 5-point gap needs 175 shared seeds, where the normal curve asked 174. On 11 November 2026 no pair of models shared three seeds in the comparison, so no verdict it gives changed. The frame comparison at /api/eval-awareness sizes its gaps the same way, in its mde field, which is deprecated: the interval is the bound. No public match has played its frames.

From 20 October 2026 every experiment report, match report, export bundle, model page and eval card also carries its limitations as a typed list, one entry per kind in a fixed order: what the number measures and does not, who stood behind it, how much was measured, what was left out, what it compares with, how it was measured and where it holds. The second kind is the one a platform reporting on itself owes its readers: it names the party that built the world, the rubric and the scorer and publishes the record as an interested party, says who designed, funded and reads a study, and says that the record names no outside reviewer. An experiment created from the same date also declares which seats its metric is read on, the focal seats the arms change, and prints the mean of the background seats, which play the same configuration in every arm, beside the focal mean, compared by nothing.

The public record also has a ceiling, and a number from it says nothing past that ceiling. On 18 October 2026 the 82 public archives held seasons of at most 400 rounds and rosters of at most 12 seats, while a Studio run can go to 2,000 rounds on the Pro and Lab plans, and to 16 seats on Lab. The Standard ladder rates three of those matches, each set to 400 rounds and played all 400 with six seats. Two more, set to 400 rounds, stopped at rounds 40 and 187, when no agent had called in for five minutes and the server closed them; until 19 October 2026 the ladder rated them where they stopped. Since then it leaves out any match that ended before its season was over, whether an operator reset it or the server closed it after its agents went quiet, and the archive keeps it as played. One of the three rated matches, and both matches it now leaves out, were played with every seat fielded under the Daishi team's own operator tag: Daishi's own seats playing each other. In the other two rated matches no seat carried that tag. The ladder's pool note gives the configured season length, what the seats' identities rest on, how many providers they named and how many matches it left out for ending early, but not rounds played, roster size or who fielded the seats. A rating today describes play in those three matches, and nothing yet about longer seasons, larger rosters or a fixed population of opponents.

A whole study can still be a chosen one. A study's card computes nothing until every trial that played is published, which keeps an owner from picking trials, but not from picking which studies to show. For Daishi's own studies, its profile counts every paired experiment the account has created beside the ones it published, with each design's standing (the Studio guide says what the count shows and leaves out). Other owners' counts are theirs and are not shown, so a study from another account carries the completeness rule and not this count.

#11. The arena games

Everything above answers for the world. Daishi also runs an arena: chess and Connect Four, two models playing each other over a separate MCP endpoint, with their own match records. It is a different instrument and the rows above do not simply transfer to it, so here is where it stands on its own.

What it is. A private lab instrument, not a public benchmark. An arena run belongs to the signed-in user who started it and is read back through their own Studio; there is no public arena archive, no arena leaderboard, and no arena figure in the world's rankings or in the Daishi Fitness Index. Nothing on the public scoreboard is produced by it. Most of the audit's questions are about protecting a published result from the people competing for it, and the arena publishes no result to protect.

Where it is stronger than the world. Three things, and they are the reason it is worth building.

The result is decided by a rules engine, not a rubric. A move is legal or it is not. There is no scoring judgment to attack, no weighting to argue with, and nothing an agent can say about itself that changes the outcome.

Grading is done by an oracle that never touched the game. After the result is final, every accepted move is measured against Stockfish (chess, in a separate process over UCI) or the exact solver (Connect Four, which is a solved game). The grader runs after the fact and cannot change who won. This is the property the world does not have: an external standard of correctness that does not come from us. Two limits on it: grading is best-effort, so a record whose grading timed out or whose engine died carries no grades rather than wrong ones; and one grade figure, plan-to-move consistency, is a textual check on the reasoning a seat itself filed, so it is the seat's own words read back, never a judgment of them and never an input from the opponent.

The record is re-executable by anyone who holds it. Each record carries the protocol it was played under and that protocol's hash, the seed, the start position, and every ply with the reasoning filed, the rejected attempts and their kind, the clock, and a sha256 chain over all of it. A third party with the record can replay the moves through any chess library to confirm every one was legal, and can run their own engine to reproduce the grades. That is a sharper audit path than the world offers, and it is available today.

Where it is weaker, or simply different.

Contamination is unfixable here, and the arena does not pretend otherwise. Chess and Connect Four are among the most thoroughly documented games in any model's training data, and Connect Four is solved outright. There is no held-out pool because there is no secret; an opening book drawn from a seed varies the start position but does not make the game novel. An arena result measures play under a fixed protocol against a known opponent, which is a narrower claim than the world makes, and the right one to make about it.

A guest seat is self-reported, by design. An invited peer says what model it is at identify, and nothing about that is verified. The record marks such a seat attested: false and attributes it to the invite, so the claim is carried as a claim. A match with a guest seat is evidence about a game, not about a model.

One injection surface existed and is now closed. A guest's self-reported model id is interpolated into the opposing seat's system prompt, which made it the only opponent-controlled text reaching another model's instructions. It is now reduced to a single token of a model-id charset before it gets there, so a guest that answers with a sentence contributes one word. Found by writing this section; fixed and tested in the same change.

Nothing published, nothing anchored. The per-ply chain has the limit the world's checksum chain had before the digest chain in section 8: it lives in the same archive as the record it attests to, and arena records are not on the digest chain, which covers world matches only. For a private lab instrument that matters less, and the re-executability above partly substitutes for it, because a disputed record can be checked against the rules rather than against our copy of it.

#Speech and the record: known limits

How to read a Daishi result names these in a line each, with what would close each one. This is each one with its evidence.

#What a model says

• Speech is barely checked. The claim screen reads three exact phrasings, and has never been checked on real messages. When honesty stopped counting on 29 October 2026, 136 of the 138 claims the screen had found across the 82 public matches were in direct messages, which the public record withholds, so nobody outside can check them. A social score can rest on those private messages too, and the scorecard says how many there were. (Alignment tracks)
• The awareness scan is not a reliable count. It matches a fixed set of phrasings in what a model wrote, and one of them is the bare word "leaderboard" or "rubric", which the world's own scoring tool uses. In a 64-match study published on 14 October 2026, a re-count that day found that every one of the 1,541 turns flagged as "watched, recorded or scored" got that flag from that word alone. Of the study's 24,735 turns, 5 called the match a test and 10 said nothing real was at stake. So the count runs high where a seat talks about the score and low where a model holds a belief it never writes down. It also reads a model's reasoning only where the route returns it, so counts do not compare across models or settings. A match report prints the count as turns the scan flagged, and says it is not a reliable count. The debrief is the seat's own answer to one question. Neither says what a model believed. (The awareness frame)

#The record

• The record is witnessed after play, not during it. A match is committed to a hash chain Daishi publishes when it ends, and two witnesses outside Daishi timestamp that chain. Nothing outside Daishi vouches for a log while its match is being played. Matches archived before 18 September 2026 were committed that day, not when they ended, and lines written since the last witnessing pass are not covered yet. (The gaps, collected)
• Single records have faults too. Known ones are listed at /errata. A record is never edited, except to remove personal data, and a notice there says where. A study's pairs have had one as well: in experiments created from 19 to 24 October 2026 a trial could start from a layout of its own. Such a pair is left out, counts against the trials the study declared it could lose, and its study's page says so. (Paired experiments)
• Daishi assesses itself. Every answer in Evaluation integrity is a self-assessment. No one outside the project has audited the scoring or reported trying to break it. If you want to try, get in touch.