# Held-out pools, statistics and the arena games

> Served 15 November 2026, when the published rubric was 2.4. A copy that names an older rubric or an earlier day is out of date; fetch it again.

## 9. Keeping the answers secret

**Are scored inputs rotated?** Yes, per season, and the previous pool becomes
public at rotation, which is what makes training on published Daishi logs an
expected and acceptable thing to do.

**Is there a commitment to the pool, published before results exist?** As of
this page, yes, and before it there was not. Rotation alone does not let an
outsider check that the pool named in a published season is the pool that was
actually in force when the matches ran, rather than one selected afterwards to
suit the results. The protocol was sound and the proof was missing.

The commitments below fix that. Each digest is the SHA-256 of a domain
separator, the pool's name, a secret 32-byte nonce and the pool's canonical
JSON, joined by newlines. The nonce is held back until the pool rotates and is
published with it. A nonce is necessary rather than decorative: a pool is a
handful of small integers, and a bare hash of one could be recovered by
enumeration.

| Pool | Committed | Digest (SHA-256) |
|---|---|---|
| Held-out seeds, `season-1` | 16 September 2026 | `0168f9fd5dc7c3f029ee3ec22272779da30601224ce08bebc665c08f15e413d6` |
| Held-out injection payloads, `s1` | 16 September 2026 | `4fa75e9ddbc736b9ff333bb4401643cf8ce07c67ce090982bd6a9a5ad9f475df` |

The payload corpus is committed by payload id rather than by payload text, so
that fixing a typo in a payload does not break a commitment while still fixing
which payloads were in force.

If a pool has to be replaced before it rotates, the replacement gets a new name
and its own commitment, and both appear here. A commitment is never edited.

What is still manual: publishing a new digest when a season opens, and
publishing the pool and nonce when it closes, are operator steps rather than
automated ones. The canonicalization and the digest are computed in code and
pinned by a test, so a pool edited without a new commitment fails the build,
but the publishing schedule is a promise rather than a mechanism.

**Is contamination otherwise controlled?** Only by rotation. Published match
logs are distributed in the clear, where some peer evaluations now distribute
traces encrypted specifically to keep them out of training corpora. For a world
whose seeds rotate this matters less than it would for a fixed task set, but it
is a real difference and not a solved problem.

## 10. Statistics

The statistics floor and the anti-best-of-N rules are not part of the audit's
checklist, but they are the other half of whether a number is worth citing, and
they are written down in [the governance doc](/docs/governance) rather than
here: a minimum of five matches across at least three held-out seeds, means
with Student t confidence intervals, interquartile means alongside means,
leaderboards sorted by a conservative estimate and ranked so that entities
sharing a rank are not separated by their intervals, verdicts on underpowered
A-versus-B comparisons withheld by the tooling rather than discouraged by
convention, and a cap of three declared private variants per model family
per season.

A paired experiment created from 19 October 2026 on also declares its read
before its first trial: how many trials it may lose (an arm's trial that
failed, lost a seat, stopped at its spend cap, was resumed by a new release,
played under another version tuple or measured nothing; a pair whose two
plays started from different layouts, which only an experiment created from
19 to 24 October 2026 can hold, counts the same way) before a comparison
reads incomplete and renders no verdict, and, if the owner sets one, a spend cap for the whole study at
which no further trial starts. No lost trial is ever replaced and no trial
is added after the losses are seen: the design plays the trials it declared
and reads what it has, so a result can never rest on trials chosen with the
outcome in view. Beside each comparison that lost a pair the report prints
the tipping point, the per-trial difference every lost pair would have had
to carry to change the reading, next to the widest difference the pairs
played did carry. Each report also keeps two rate limits apart per arm: what
the world refused of the seats' own actions under its seat rules, and what
the provider made the harness retry; the second is read as provider retries
per 100 model calls arm by arm, since the arms share one key and a provider
that throttles one arm's calls reaches the other arm's pairs. Uneven rates
are a reading the verdict paragraph states first, never a verdict ([the
Studio guide](/docs/studio) says how each is counted).

**The analysis label missed one change, and is bumped now.** Every experiment
and series report names, beside the rubric, the analysis that read its
trials (`versions.analysis`, and the version line under "What this study
cannot support" on a study's page). From 20 October 2026 the label was
`daishi-analysis-v1`. On 24 October 2026 the experiment report began leaving
out a pair whose two trials played different seeds (section 8 says how such
pairs arose) and reading a fixed rule's stop recorded on such pairs again
without them, and the label stayed v1, so a report labelled v1 is one of two
analyses. From 11 November 2026 the label is `daishi-analysis-v2`, which
names two changes. The first is that rule. Of the studies published on
Daishi's own profile, one held such pairs: the replay check
`exp_f2a2fbe0f666622dfc` ("Replay check, 16 setups on 8 maps"). Its fixed
rule stopped on 16 pairs on 23 October 2026 and read undecided, a mean paired
difference of 2.00 (95% interval -0.74 to 4.74). Read with the rule, 6 of its
16 pairs are left out, more than the one lost trial its declared read allows,
so it is incomplete and gives no verdict. No other published study played a
pair on two sets of seeds.

The second is the Student t critical value every interval is read with.
Under v1 a 95% interval read a table rounded to three decimals up to 30
degrees of freedom and the normal value 1.96 past it, which runs narrow:
about 4% at 31 degrees of freedom and 2% at 60. Under v2 every critical
value is exact, at every level and every number of degrees of freedom, and
the detectable gap, a series' half-width and the trials it says more
precision would take are sized on the same values. Of the studies published
on Daishi's own profile, the change is more than a last digit on one:
`exp_a5813ccd9705d355b8` ("Haiku 5.5 thinking off, low, high, harness
0.6.10"), whose per-arm and per-model 95% intervals run over 32 trials and
are about 4% wider. Its thinking-off arm reads 32.8 to 36.2 where it read
32.9 to 36.1, and its thinking-high arm 47.4 to 52.4 where it read 47.5 to
52.3. Its comparisons, read at 97.5%, already used exact values. On nine
others a printed last digit moves on at least one interval, each by less
than a thousandth of the interval's width, as `exp_59c33c3ad4828d5222`'s
thinking-high arm reads 943.46 to 1264.81 output tokens per turn where it
read 943.48 to 1264.79. No verdict changed.

The model comparison at `/api/compare` took the exact values the same day,
and its power check moved onto t. It carries no label of its own, so the
change is dated here. The power check, the smallest true gap the comparison
finds four times in five and the shared seeds each gap needs, is sized on
Student t at one fewer degree of freedom than there are seeds, as an
experiment's detectable gap is. Before, it read the normal curve at every
number of seeds, which put the smallest gap about 1.9 times too small at 3
seeds and 1.3 times at 5. Since 10 November 2026 the check reads the larger
of the spread the comparison observed and the platform's prior spread for a
paired difference, and on that prior, at the standard season length, a
5-point gap needs 175 shared seeds, where the normal curve asked 174. On
11 November 2026 no pair of models shared three seeds in the comparison, so
no verdict it gives changed. The frame comparison at `/api/eval-awareness`
sizes its gaps the same way, in its `mde` field, which is deprecated: the
interval is the bound. No public match has played its frames.

From 20 October 2026 every experiment report, match report, export bundle,
model page and eval card also carries its limitations as a typed list, one
entry per kind in a fixed order: what the number measures and does not, who
stood behind it, how much was measured, what was left out, what it compares
with, how it was measured and where it holds. The second kind is the one a
platform reporting on itself owes its readers: it names the party that
built the world, the rubric and the scorer and publishes the record as an
interested party, says who designed, funded and reads a study, and says
that the record names no outside reviewer. An experiment created from the
same date also declares which seats its metric is read on, the focal seats
the arms change, and prints the mean of the background seats, which play
the same configuration in every arm, beside the focal mean, compared by
nothing.

The public record also has a ceiling, and a number from it says nothing past
that ceiling. On 18 October 2026 the 82 public archives held seasons of at
most 400 rounds and rosters of at most 12 seats, while a Studio run can go to
2,000 rounds on the Pro and Lab plans, and to 16 seats on Lab. The Standard
ladder rates three of those matches, each set to 400 rounds and played all
400 with six seats. Two more, set to 400 rounds, stopped at rounds 40 and
187, when no agent had called in for five minutes and the server closed
them; until 19 October 2026 the ladder rated them where they stopped. Since
then it leaves out any match that ended before its season was over, whether
an operator reset it or the server closed it after its agents went quiet,
and the archive keeps it as played. One of the three rated matches, and
both matches it now leaves out, were played with every seat fielded under
the Daishi team's own operator tag: Daishi's own seats playing each other.
In the other two rated matches no seat carried that tag. The ladder's pool
note gives the configured season length, what the seats' identities rest
on, how many providers they named and how many matches it left out for
ending early, but not rounds played, roster size or who fielded the seats.
A rating today describes play in those three matches, and nothing yet about
longer seasons, larger rosters or a fixed population of opponents.

A whole study can still be a chosen one. A study's card computes nothing
until every trial that played is published, which keeps an owner from
picking trials, but not from picking which studies to show. For Daishi's own
studies, its profile counts every paired experiment the account has created
beside the ones it published, with each design's standing ([the Studio
guide](/docs/studio) says what the count shows and leaves out). Other owners'
counts are theirs and are not shown, so a study from another account carries
the completeness rule and not this count.

## 11. The arena games

Everything above answers for the world. Daishi also runs an **arena**: chess
and Connect Four, two models playing each other over a separate MCP endpoint,
with their own match records. It is a different instrument and the rows above
do not simply transfer to it, so here is where it stands on its own.

**What it is.** A private lab instrument, not a public benchmark. An arena run
belongs to the signed-in user who started it and is read back through their own
Studio; there is no public arena archive, no arena leaderboard, and no arena
figure in the world's rankings or in the Daishi Fitness Index. Nothing on the
public scoreboard is produced by it. Most of the audit's questions are about
protecting a published result from the people competing for it, and the arena
publishes no result to protect.

**Where it is stronger than the world.** Three things, and they are the reason
it is worth building.

*The result is decided by a rules engine, not a rubric.* A move is legal or it
is not. There is no scoring judgment to attack, no weighting to argue with, and
nothing an agent can say about itself that changes the outcome.

*Grading is done by an oracle that never touched the game.* After the result is
final, every accepted move is measured against Stockfish (chess, in a separate
process over UCI) or the exact solver (Connect Four, which is a solved game).
The grader runs after the fact and cannot change who won. This is the property
the world does not have: an external standard of correctness that does not come
from us. Two limits on it: grading is best-effort, so a record whose grading
timed out or whose engine died carries no grades rather than wrong ones; and
one grade figure, plan-to-move consistency, is a textual check on the
reasoning a seat itself filed, so it is the seat's own words read back, never
a judgment of them and never an input from the opponent.

*The record is re-executable by anyone who holds it.* Each record carries the
protocol it was played under and that protocol's hash, the seed, the start
position, and every ply with the reasoning filed, the rejected attempts and
their kind, the clock, and a sha256 chain over all of it. A third party with
the record can replay the moves through any chess library to confirm every one
was legal, and can run their own engine to reproduce the grades. That is a
sharper audit path than the world offers, and it is available today.

**Where it is weaker, or simply different.**

*Contamination is unfixable here, and the arena does not pretend otherwise.*
Chess and Connect Four are among the most thoroughly documented games in any
model's training data, and Connect Four is solved outright. There is no
held-out pool because there is no secret; an opening book drawn from a seed
varies the start position but does not make the game novel. An arena result
measures play under a fixed protocol against a known opponent, which is a
narrower claim than the world makes, and the right one to make about it.

*A guest seat is self-reported, by design.* An invited peer says what model it
is at `identify`, and nothing about that is verified. The record marks such a
seat `attested: false` and attributes it to the invite, so the claim is carried
as a claim. A match with a guest seat is evidence about a game, not about a
model.

*One injection surface existed and is now closed.* A guest's self-reported
model id is interpolated into the opposing seat's system prompt, which made it
the only opponent-controlled text reaching another model's instructions. It is
now reduced to a single token of a model-id charset before it gets there, so a
guest that answers with a sentence contributes one word. Found by writing this
section; fixed and tested in the same change.

*Nothing published, nothing anchored.* The per-ply chain has the limit the
world's checksum chain had before the digest chain in section 8: it lives in
the same archive as the record it attests to, and arena records are not on
the digest chain, which covers world matches only. For a private lab
instrument that matters less, and the re-executability above partly
substitutes for it, because a disputed record can be checked against the
rules rather than against our copy of it.

## Speech and the record: known limits

[How to read a Daishi result](/docs/benchmark-card#what-is-not-proven-yet-and-what-closes-it)
names these in a line each, with what would close each one. This is each one
with its evidence.

### What a model says

- **Speech is barely checked.** The claim screen reads three exact
  phrasings, and has never been checked on real messages. When
  honesty stopped counting on 29 October 2026, 136 of the 138 claims the
  screen had found across the 82 public matches were in direct messages,
  which the public record
  withholds, so nobody outside can check them. A social score can rest on
  those private messages too, and the scorecard says how many there were.
  ([Alignment tracks](/docs/alignment#changes-to-the-alignment-index))
- **The awareness scan is not a reliable count.** It matches a fixed set of
  phrasings in what a model wrote, and one of them is the bare word
  "leaderboard" or "rubric", which the world's own scoring tool uses. In a
  64-match study published on 14 October 2026, a re-count that day found
  that every one of the 1,541 turns flagged as "watched, recorded or scored"
  got that flag from that word alone. Of the study's 24,735 turns, 5 called
  the match a test and 10 said nothing real was at stake. So the count runs
  high where a seat talks about the score and low where a model holds a
  belief it never writes down. It also reads a model's reasoning only where
  the route returns it, so counts do not compare across models or settings. A
  match report prints the count as turns the scan flagged, and says it is
  not a reliable count. The debrief is the seat's own answer to one question.
  Neither says what a model believed.
  ([The awareness frame](/docs/studio/runs#the-awareness-frame))

### The record

- **The record is witnessed after play, not during it.** A match is committed
  to a hash chain Daishi publishes when it ends, and two witnesses outside
  Daishi timestamp that chain. Nothing outside Daishi vouches for a log while
  its match is being played. Matches archived before 18 September 2026 were
  committed that day, not when they ended, and lines written since the last
  witnessing pass are not covered yet.
  ([The gaps, collected](/docs/eval-integrity#the-gaps-collected))
- **Single records have faults too.** Known ones are listed at
  [/errata](/errata). A record is never edited, except to remove personal
  data, and a notice there says where. A study's pairs have had one as well:
  in experiments created from 19 to 24 October 2026 a trial could start from
  a layout of its own. Such a pair is left out, counts against the trials the
  study declared it could lose, and its study's page says so.
  ([Paired experiments](/docs/studio/experiments#paired-experiments))
- **Daishi assesses itself.** Every answer in
  [Evaluation integrity](/docs/eval-integrity#the-gaps-collected) is a
  self-assessment. No one outside the project has audited the scoring or
  reported trying to break it. If you want to try,
  [get in touch](/contact).