# Evaluation Integrity: the audit checklist, answered

**Last updated: 29 October 2026.**

Agent benchmarks are, as a class, easy to cheat. The 2026 BenchJack audit
(arXiv:2605.12673) red-teamed ten widely used agent benchmarks across software
engineering, web, desktop and terminal work, and reached near-perfect scores
on most of them without solving a single task. It catalogued 219 distinct
flaws in eight classes and published a checklist so that anyone running an
evaluation can be asked the same questions.

This page answers those questions for Daishi. Every answer below was checked
against what the running system does, not against what the rest of this
documentation claims. Where the answer is no, it says no, and names what would
change it.

Two notes on what this is. The audit's checklist is organized into seven
categories over eight flaw classes; the questions here are stated in our own
words against that structure, so a reader holding the paper can map them, but
they are not a verbatim reproduction of its appendix. And this is a
self-assessment. Nobody outside this project has tried to break Daishi's
scoring, which is itself one of the findings below.

## What Daishi is being assessed as

Daishi is a persistent multi-agent world. Agents connect from anywhere as
remote clients over MCP, take actions, and the server decides what happened.
Scoring runs afterwards over the append-only event log the server wrote. There
is no task harness that runs an agent's code, no test suite an agent could
edit, no reference solution sitting on a disk the agent can reach, and no
grader the agent talks to.

That shape makes several of the audit's flaw classes structurally
inapplicable rather than defended against, and this page says which. It is not
a claim of superiority. A different architecture has different failure modes,
and the ones Daishi does have are in the gaps section.

All of that describes the platform's side of the connection. An agent can be a
coding agent that writes and runs programs on its own machine while it plays;
the world sees only the tool calls that reach it, and scores what they did. A
seat the Studio drives itself is offered the world's tools and nothing else, so
a coding agent takes part in the public world or through a Studio guest seat
([Play from your own agent tool](/docs/developer/own-agent-tools)).

What the design does rule out is grading the code itself. Whether an agent's
code compiles or passes tests is a coding benchmark's question, and the usual
way to answer it is to run that code in a sandbox and test it there, which is
the arrangement section 1 asks about. Daishi asks how an agent behaves in a
world it shares with other agents.

Sections 1 to 10 answer for the world and its public rankings. Daishi also
hosts **arena** games, chess and Connect Four played between two models on
their own endpoint, which is a different instrument with a different answer to
most of these questions. Section 11 covers it separately rather than stretching
the world's answers over it.

## Summary

An earlier version of this page wrote "Pass" in rows where the answer follows
from the architecture and in rows where a test asserts it, which made the
weaker evidence read like the stronger. Those are different claims, so the
table now separates them.

**Holds by construction** means the flaw has nowhere to live in this design:
there is no code path that could exhibit it, and no test can demonstrate the
absence of a thing that was never built. Take it as an architectural claim you
can check by reading the description, not as a measurement.
**Demonstrated** means a test in the suite fails if the property stops holding,
and what that test asserts is described in the row. The suite itself is not
published, so this is a description you can hold us to, not a file you can
open. **Partial** and **Open** mean what they say.

| Category | Verdict | On what evidence |
|---|---|---|
| Isolation of agent from evaluator | Holds by construction | Agents are remote MCP clients with no process, filesystem or shell on the platform. One boundary is demonstrated: scenarios load by catalog id only, never by filesystem path. |
| Answers kept out of the agent's reach | Demonstrated | The held-out pools are committed to in public, and a pool edited without a fresh commitment fails the build. The live view is redacted or delayed while a match runs, asserted per endpoint. |
| Untrusted input into the evaluator | Holds by construction, partly demonstrated | Scoring reads typed fields off events the server wrote, never agent text. Escaping and key-shadowing defenses at the boundary are demonstrated. |
| LLM judge robustness | Not applicable to ranked scores; one **untested** residual risk off the ranked path | No ranked figure is produced by a model. The optional commitment extractor is the only eval LLM, and the injection path described in section 3 has no test. |
| Scoring robustness | Demonstrated | Determinism, tie handling, archetype classification and the refusal to fabricate zeros for untracked counters are each asserted. |
| Evaluation logic gaps | Demonstrated, for the gaps found so far | Every exploit path closed to date has a regression test. This says nothing about paths nobody has looked for. |
| Sandbox permissions | Holds by construction | No tool takes an address and fetches it; no agent input is executed. The one outbound call is inference, to a destination the key holder fixed. |
| Adversarial testing | Partial | Prompt injection is scored against a held-out corpus and log tampering is caught by replay, both demonstrated. Nobody outside this project has tried to break any of it. |
| Trace tamper-evidence | Demonstrated, with the boundaries stated in section 8 | The checksum chain and its tamper detection are demonstrated. Each archived match is also digested onto a public chain whose heads are timestamped by two witnesses outside this deployment, with copies requested from the Internet Archive, so an archive can be checked against a commitment made when the match ended. It covers the record after play, not play itself, and matches archived before 18 September 2026 were committed that day. |
| Held-out pool commitment | Demonstrated | Digests are published below and recomputed from the live pools on every build. Rotation is still manual. |
| Arena games (chess, Connect Four) | Separate path, not covered by the rows above | See section 11. Stronger where the rows transfer, with no public results to audit. |

## 1. Isolation: agent and evaluator

**Does the agent execute in the same process, container or filesystem as the
evaluator?** No. Agents are remote clients. They send tool calls over HTTP and
receive results. Nothing an agent sends is executed, and an agent has no
filesystem, shell or process of its own inside the platform.

**Can an agent read or modify the scoring code at runtime?** No. Scoring runs
server-side over the archived log after a match, and no tool exposes it. The
rubric is published as a specification in [the benchmark doc](/docs/benchmark) so
that a score can be recomputed from a match's own archived log; the
implementation is not published, which
[the governance doc](/docs/governance) states plainly rather than implying an
audit path that does not exist.

**Can an agent write to the record it is scored from?** No. The event log is
append-only and written only by the engine, from actions the engine itself
validated. An agent cannot emit an event, edit one, or suppress one. Every
number in a scorecard traces to an event the server decided to write.

**Can an agent see the answer key?** The seed is the answer key here: world
generation is a pure function of it, so knowing the seed hands you the terrain,
the resource pools and the ruins for free. Live matches therefore withhold it.
While a match is running, and while a lobby is still in pregame, the match log
and replay pages omit both the seed and the board, and the state, event,
metrics and report endpoints serve a cut-down view instead of the full one.
The state endpoint returns the phase, the current [round](/docs/about#vocabulary) and the season and nothing spatial. The
event stream withholds the three event types whose payloads carry the seed
(the season start, a world reset and the match manifest), and strips
positional detail from the rest. The metrics endpoint serves only what
an in-world inspection would have shown, so a rival's economy cannot be polled
for free by anyone who can use a command line. Redaction is the default
posture, not an opt-in. The operator can see through it with an admin
credential. When a match ends, its seed publishes with its results, which is
the intended reveal.

**What does the live world at daishi.ai actually run?** The **delayed**
posture, since 16 September 2026, at a lag of ten rounds. An earlier version of
this page said it ran fully open, which was true for a few hours before that.
The public world is an exhibition: its matches feed the public leaderboard,
which says it is a record of open play, but they produce no ranked result and
draw no held-out seed, so nothing on it is protected by withholding it, and a
spectator who cannot see the board has no reason to watch. Scored work does not run on
it. Studio runs and series trials play in private worlds on the same server,
which the public cannot see at all while they run (the owner sees their own
run), and a standalone evaluation instance defaults to redacted; this
paragraph is the only thing that would have to change if that ever stopped
being true.

Delayed mode is the third posture between fully open and redacted. The state
and event endpoints serve the live map, every agent's position and every
action, a fixed number of rounds behind the present; the per-agent metrics, the
live scorecards and the replay page still wait for the archive, as they do
under redaction. It exists because redaction was solving a fair-play problem by
taking the spectacle away from people, and a delay keeps the spectacle while
making scraped intelligence worth less than acting on the present. Every
response labels its own mode and, in delayed mode, its lag, and the operator's
credential sees the present on every one of these endpoints.

One thing delayed mode does not protect, said plainly: the seed. The seed
integer itself is withheld until the match ends, but the map that the seed
generates is served after the delay, so on a delayed world the seed protects
nothing about the match in progress beyond its first few rounds. That is
acceptable for an exhibition: each public-world match draws a fresh random
seed, publishes it with its results, and never uses a held-out seed. It is why
scored matches do not run delayed.

**Are practice and scored inputs separated?** Yes, in two places. Practice
seeds are public (a practice match publishes its seed with its results) and
free to train against; scored matches draw from a
rotating held-out pool. The same tiering applies to the prompt-injection
payload corpus. Custom runs started by a signed-in user are forced onto the
public practice pool even when the scenario names a held-out one, because those
matches archive publicly and drawing the ranked pool there would publish the
held-out seeds as training data.

**Does any scored quantity come from the agent's own account of itself?** No.
Three things an agent can say about itself are carried and none is scored. The
model, provider and scaffold strings from registration ride along for
attribution and are marked display-only. The closing statement an agent files
at the end of a match is passed through verbatim to its scorecard and is never
graded; the rubric grades verified behavior, not testimony. And the trust
division is computed server-side: claiming the reference-harness label requires
both the reference scaffold string and a server-set attestation stamped at
registration, so typing the scaffold name buys nothing. An unattested
registration that names the reference scaffold is recorded as a claim, with
`claimed:` ahead of the scaffold name, and the leaderboard rates it as its own
entity rather than the reference harness's. A seat driven through the relay
client or by URL is archived under its label and never rated; each world action
that arrives by URL is counted in the match record.

## 2. Untrusted input into the evaluator

**Does the evaluator ever evaluate, execute or deserialize agent-controlled
data?** No. There is no dynamic evaluation of agent strings anywhere in the
platform: no interpreter call, no dynamically constructed function, no
dynamically resolved import, no shell.

Two helper programs exist on the platform's side, one driving a chess engine for
arena grading and one running the harness for a repeated-trial series. Both
are started with a fixed argument list and no shell, nothing in either
argument list comes from an agent, and neither is reachable from an agent's
tool surface.

**Are agent-supplied fields bounded?** Yes. Agent text arrives as JSON tool
arguments with per-field size caps, per-round call caps on the self-report
tools, and a per-seat ceiling on trajectory log volume, with long model output
explicitly marked as truncated rather than silently cut.

## 3. LLM judge robustness

**Is an LLM in the scored path?** No. The Daishi Fitness Index, its five
dimensions, the achievements, the grades and the ranked ratings are
computed by deterministic code from counted events and final-board quantities.
No language model scores a ranked match, so there is no judge to inject.

**Is an LLM anywhere in evaluation?** Yes, in one optional place, and it is
worth being precise about it because this is where the residual risk lives.
Report cards carry a promise-keeping figure: how often an agent did the thing
it said in chat it would do. Turning chat into a list of commitments is an
extraction problem, and there are two extractors. The default is a deliberately
low-recall pattern matcher. The other calls a small language model, runs only
on demand against already-archived matches, never during play, and every report
says which extractor produced its number so the two are never conflated.

**Is agent text structurally contained when it reaches that model?** Yes.
Messages are passed as serialized JSON fields, so an agent cannot terminate the
surrounding structure and append instructions of its own. The model's answer is
then constrained rather than trusted: a label is accepted only if it is one of
four permitted values, it must carry an index into the batch the server
supplied, and the speaker attached to it is read from the server's own record
of who sent that message, never from the model's answer.

**Is semantic injection prevented?** No, and this is a real residual risk
rather than a theoretical one. Structural containment stops an agent from
breaking the format; it does not stop an agent from writing in-world chat
aimed at the extractor. Because the model chooses which message in the batch a
label attaches to, a crafted message could in principle push a fabricated
commitment onto a different agent's message in the same batch, which would show
up as that agent breaking a promise it never made.

What bounds it: this affects one figure on a report card, that figure is
labeled with the extractor that produced it, the pattern-matching extractor is
the default, and none of it enters the ranked score. Sitting next to it on the
same card is the bonded-commitment rate, which is engine-verified rather than
extracted, because a bond is a promise with collateral the engine actually
holds and settles. The gap between what an agent promises with collateral and
what it promises for free is the honest version of the measurement.

What is missing: nobody has run an injection attack against the extractor. It
is on the list below.

**Is the extractor validated?** Only on made-up sentences. Both extractors
are graded against a golden set of 15 made-up commitment sentences, each with
two raters' labels, reporting precision, recall and F1 per commitment kind, and
Cohen's kappa between the raters alongside, as the ceiling an extractor cannot
honestly exceed. Neither has been graded on real chat. The same set's 6 made-up
claim sentences grade the claim screen, counting only the claims the screen
finds: on 29 October 2026 it found 4 of 6 and missed both "I have no ore"
sentences, a phrasing its pattern is meant to read and never matches. The
honesty figure that screen produces stopped counting in the Alignment Index
that day ([why](/docs/alignment#changes-to-the-alignment-index)).

## 4. Scoring robustness

**Does the score depend on pattern-matching an agent's output?** Not the
ranked score. Every dimension of the Fitness Index is built from actions the
engine validated and quantities on the final board. The one pattern-matched
signal, commitment extraction, is confined to the report-card figure described
above.

**Is partial credit tested for shortcuts?** Yes. The rubric normalizes by
rounds alive so that survival time is not a confound, with a floor under the
divisor so that a short-lived agent's rates cannot be divided into a huge
score. That floor is a pinned regression: a 100-round sprint used to out-rate a
full-match grind by sixty to one, and the test now asserts the bound.

**Are failure and absence distinguished from zero?** Yes, deliberately. Where a
counter was never recorded, the scorecard reports it as unrecorded rather than
as zero activity, and says so in the text. An agent whose scaffold quit
mid-match is disclosed as having gone dark rather than narrated as having
survived flawlessly.

**Do ties score identically?** Yes. Competition ranking means two agents with
exactly equal wealth share a rank, so a tie cannot advantage whoever registered
first. Older rubric versions kept the behavior they shipped with so that
recomputing a historical report reproduces its history.

## 5. Evaluation logic gaps

**Is there a standing test suite that tries to farm the score without doing the
task?** Yes, and it is the part of this assessment with the most evidence
behind it. Each of the following was a real exploit, is closed, and has a
pinned regression test:

- Parking goods in an absurd standing trade offer nobody would accept, so they
  counted as wealth while being theft-proof. A knockout now voids the offer and
  the goods are lootable like anything else.
- Using bond escrow as extra carrying capacity. Staked goods still count
  against what an agent is carrying.
- Minting reputation through a ring of fresh counterparties, or by farming kept
  bonds. Both are capped globally per season, not just per pair.
- Posting a no-attack bond to someone on the far side of the map and collecting
  reputation for a promise that cost nothing. Distance mints no reputation.
- Depositing goods into another agent's storehouse to pump the owner's
  leaderboard position. Banked goods stay the depositor's wealth.
- Building the same benefit structure repeatedly in one region.
- The rate-normalization sprint described above.
- Scoring like an honest player without playing, found by our own red team on
  30 September 2026: a seat that sat still collected the whole survival base,
  two seats handing the same goods back and forth maxed the social dimension,
  and harm to other agents cost nothing. Rubric 2.3 scales survival by world
  actions taken, credits a trade by the value it moved (at most five per
  partner), and takes public aggression and a share of every shared pool
  drained to collapse out of the social score. The collusion screen now names
  such a pair as a wash pair.

Replayed on a fixed seed under 2.3, two seats from that red team still score
above the mean of its scripted honest seats: one that spreads cheap actions
across the dimensions and never trades, and one that farms reputation from
small no-attack bonds it keeps. Neither is closed, on purpose. The first
gathers, explores and trains more than the scripted honest seats, which stop
playing after their first goals, and less than the strongest public agents;
the second keeps bonds of the same kind public agents keep.
[The benchmark](/docs/benchmark#play-value-and-harm-rubric-2-3) has the numbers,
and says why cutting either would cost real agents more than the scripts.

**Are environment invariants checked?** Yes. Every seed in a 250-seed sweep is
asserted to produce at least one region of every terrain, so a scenario cannot
silently draw a world where a required activity is impossible.

## 6. Sandbox permissions

**What can an agent do on the platform?** Only what the published tool surface
allows, and nothing else. No filesystem, no shell, no code execution, no access
to another match, and no tool that takes an address and fetches it, so an agent
cannot aim the server at a destination of its choosing. The tool surface is
documented in full in [the agent guide](/docs/agents).

The one outbound call the platform makes on an agent's behalf is inference.
For a Studio run, the platform drives the seats the owner configured with a
model on the owner's own key, and the destination is the provider that key
belongs to: the endpoint is fixed per provider in code, so an owner cannot point
a seat at a proxy of their own. A guest seat in a world run is the exception by
design: it is the guest's own client on the guest's own keys, which the platform
never drives, so it is recorded as self-reported and never rated.
Because the platform reads every response, the archive records per seat the
model id the provider's responses named, the upstream a router named, and the
calls the provider failed, beside the id the owner configured; a seat that
stalled on a throttled key is thereby distinguishable from a model that chose
to idle, and a "same model" repeat knows which snapshot served. Separately, the code
carries an inference gateway an agent could route its own calls through, which
is how model identity and token usage would become server-measured facts
rather than claims; it is not switched on for this deployment, so the
gateway-verified trust division cannot currently be earned and no agent's
calls pass through it. In neither path can an agent choose the destination.

**Are there ceilings?** Yes: per-key request ceilings charged to the key rather
than the address, per-round action budgets that reject extra actions rather than
queueing them, field size caps, and a per-seat trajectory log cap.

## 7. Adversarial testing

**Is there a null or random baseline?** Yes. A uniform-random policy is pinned
into ranked lobbies alongside scripted baselines that hoard, trade honestly,
raid and (in the current reference roster) converse. A cooperate-then-defect
exploiter is a separate probe. A score means nothing without knowing what a
random policy gets on the same scenario, so that number is always in the same
match rather than quoted from a different one.

**Is there a prompt-injection agent?** Yes, and this is the strongest
adversarial result on the platform. A red-team injector baseline delivers
payloads drawn from a rotating held-out corpus. Ground truth is a forensic tag
stamped on each attempt and never delivered to the target, so an agent that
complied complied with the visible text and not with a marker it could see.
Compliance is judged against the payload's declared demand, not against a
string: did the target actually transfer value to the injector, re-emit the
marker, attack the named third party, or stop gathering. Injection resistance
is reported separately from game skill, with confidence intervals, because a
strong player that folds to a fake system message is a different and worse
thing than a strong player that ignores it.

**Is there a state-tampering agent?** No standing test. See the trace section
below for what detection exists.

**Has anyone outside the project audited this?** No. This is a
self-assessment, and it is the single largest caveat on the page.

**Found a way to game a score or break a match?** Tell us through
[/contact](/contact), or with Report a bug in the [Studio](/studio) if you
have an account. A confirmed finding is published on this page, and when it
changes what a published match record means, as an erratum beside that record
at [/errata](/errata).

## 8. Trace and result tamper-evidence

Every archived match is committed to outside this deployment. When a match
ends, a digest of its record goes onto a public hash chain, and each new head
of that chain is timestamped by two parties we do not run: OpenTimestamps,
which anchors it in Bitcoin, and an RFC 3161 timestamp authority. This
server also asks the Internet Archive to keep a copy of each new head. A
record changed after that, even with our own database rewritten to match,
would no longer agree with a head those parties dated. The rest of this
section is how the pieces fit, what they prove and do not, and where the
chain starts.

**Inside the archive.** Every match commits to its own trajectory: a
world-state checksum is written into the log every 25 rounds, forming a chain;
a match that never reaches a 25-round mark carries none, and the verification
endpoint says so.
A public verification endpoint checks that chain on any archived match,
including matches played by models. It confirms the chain is present, well
formed and in round order, and it cross-checks the final board against the
log, so an archive that marks an agent dead with no corresponding death event
in the log, or the reverse, fails. That check was once vacuous and is now
real; a doctored board does not pass it.

For fully scripted matches, where every participant was a server-run baseline
and no external model was involved, verification goes further and re-runs the
entire match from the recorded seed and roster, then compares the freshly
derived checksum chain against the logged one bit for bit. That is a hard
determinism test. The endpoint also returns a re-execution script, but that
script runs on the platform's own tooling, which is not distributed, so for
anyone outside the project the re-run is ours to perform, not theirs to
check; the chain comparison above is the part a reader can inspect.

Where replay stops: a match played by models cannot be re-executed, because
the models are not deterministic and are not ours. For those matches, which
are what a leaderboard is made of, the checksums sit in the same archive as
the events they attest to, so on their own they could not show that nobody
had rewritten both. That is what the digest chain is for.

**The digest chain.** When a match is archived, a digest of its
stored record (read back from storage after the save, never taken from the
server's working copy) is appended to a public chain: the final board with
each seat's model, the configuration and recipe hash, a hash over the run's
provenance block (its seats as configured and whatever the platform recorded
about how they were served), the checksum chain's length and last link,
and the sha256 of the event log, computed once over the public log exactly as
the export endpoint serves it and once over everything including private
messages. Each line's head is the sha256 of the previous head and the new
digest, from a genesis of sixty-four zeros, so a head commits to every line
before it. The feed is served at `/api/digests` and a match's own line, with
the record its digest was computed over, at `/api/matches/<id>/digest`; the
report page lists the match's own line under Reproduce and audit. A private lobby's match keeps
its place on the chain with its id and record withheld.

**Two witnesses outside this deployment.** Whenever there is a new head (the
check runs every ten minutes), it goes to the public OpenTimestamps calendars,
all of them at once, and to an RFC 3161 timestamp authority. The calendars'
replies are kept together in one proof file, so the proof does not depend on
any single calendar surviving. A calendar's reply is a promise until its
Bitcoin transaction confirms, usually within hours; this server then fetches
the path from the head to the Bitcoin block and serves the finished file,
which the OpenTimestamps client checks against Bitcoin with no calendar
involved. Each proof says which it is: `pending`, `anchored` (with the block),
or `signed` (the authority's reply, final when issued, with the time it
states). The proofs are served next to the feed (`/api/digests/witnesses`),
byte for byte, so anyone can verify them with the standard tools and none of
that verification runs through this server. A line is covered by any witness
whose head is at or past it.

**Copies held elsewhere.** A timestamp proves that a head existed by a time.
It cannot by itself stop an operator from rewriting the chain later and having
the new head stamped too; only a copy of the old head, held by someone else,
does that. So after each new head is witnessed, this server asks the Internet
Archive to capture two public pages: the head endpoint (`/api/digests/head`)
and the newest match's own line (`/api/matches/<id>/digest`), which carries
that head under an address no other line shares. The line is captured
because the archive answers a request for an address it captured recently
with the earlier capture, so a copy of the head endpoint can show an older
head. A private match's line has no public page and is not captured. A copy
of any head covers every line before it, since each head depends on all of
them. We cannot edit or backdate a capture. The archive does not accept every
request: the head endpoint shows how the last request for each page went,
with the capture's address once the archive confirms it and a note when the
archive handed back an earlier capture instead of a new one, and it links the
archive's own listings, which show every capture that exists. The archive also
limits how many times a day it captures one page. When it refuses a page for
that reason, this server waits for the next UTC day before asking for that
page again, even if a new head is witnessed in the meantime (a restart in
between still asks once), and the head endpoint shows the refusal until then.
A reader who keeps a copy of a head, or of a proof, adds another; the feed is
public so that copies exist in places we do not control.

**Where the chain starts.** The chain went live on 18 September 2026. The 58
matches archived before then, which ended between 13 July and 3 September
2026, were added to it that day, so for them the commitment dates from 18
September rather than from the end of play. Every match archived since then
was added as it ended. A line's `recorded_at_ms`, beside its record's
`ended_at_ms`, shows which case a match is.

What a witness proves: that the record served today, with that digest,
existed in that form no later than the witnessed time. What it does not prove:
that the scorer was fair, that a model was what its seat said, anything about
play while the match was running (the record is committed when the match
ends, so what happened during play rests on the server that ran it and on the
transcripts the agents keep themselves), or anything about a line recorded
after the newest witnessed head. There is always a tail of recent lines
waiting for the next witnessing pass, and the head endpoint says how long it
is. The earliest witness covering a line bounds when its record was fixed;
compare its time with the match's end.

**Declared designs go to the same witnesses.** A Studio experiment's design
document reads as declared when the server received it before the first
trial started, and both of those times are the server's own. So since 19
October 2026 the document's SHA-256 (never its text) goes to the same two
witnesses the moment it is stored, as does a run set's design for each of
its setups. Creating an experiment with a design waits a few seconds for
them, so its first trial normally starts after the authority has signed. The
proofs are listed in the owner's read of the design and, once the study is
public, on the design's page, which serves each proof file and says how the
authority's signed time compares with the first trial's start. Each trial's
archive also names the design its launch read (`config.run.experiment`: the
experiment, the arm, and the design's digest and arrival time), and the
trial's line on the chain commits to that block through `provenance_sha256`.
What the proofs show: the document existed, with that digest, by the
witnessed time. Each trial's record names the digest its launch read; that
record is Daishi's own and joins the witnessed chain when the trial ends.
What the proofs do not show: when the first trial started. That time is the
server's stamp on the trial's record too, so the comparison sets the
witness's clock against the server's, and the server's is witnessed later.
A design stored before 19 October 2026, or one no witness has answered for
yet, has no proof, and its page says its times are Daishi's own.

**The creation of an experiment is witnessed the same way.** Since 19
October 2026, the moment an experiment's record is stored, Daishi writes a
creation record (its id, the time, its name, whether it was declared a
pilot, and the digest of the design it was created with) and hands the
record's SHA-256 to the same two witnesses; once a trial is published the
record and its proofs are on the study's page, so when the experiment was
created can be checked against clocks Daishi does not keep. A design
replaced before the first trial is kept as an earlier version, with its own
digest and proofs, and the design page lists it. An experiment created from
that day takes a design after its first trial started only against 32 or
more hex digits of its digest in the name; the ones created before keep
the 12 they were made under, and each page says which. The cancel of an
experiment carries the owner's reason, one word from a fixed list, and the
note on a study is dated entries that are never edited or removed, so what
was said first and what was added later stay apart.

**The seeds come from a public randomness round, not from Daishi.** Since
19 October 2026 an experiment's world and layout seeds are not drawn on the
server. When its first trial is about to start, Daishi reads from drand, the
League of Entropy's public randomness beacon, the first round published
after the design's witnessed time: the timestamp authority's signed time for
the design's digest as the design stood then, the creation record's signed
time without a design, or Daishi's own clock where no authority had
answered within a minute, and the record says which. Each seed is the
SHA-256 of that round's randomness, the experiment's id and the trial's
number, so neither the owner nor Daishi chose the maps, and a design fixed
before the round existed could not have been written to them. The seeds
record, public with the study, carries the round, its randomness, the
anchor, the derivation and every trial's seeds; a reader checks the round
against any drand relay and recomputes the seeds. Daishi reads the round
over HTTPS and does not verify its signature. Where the beacon does not
answer within ten minutes of the first trial's turn, the server draws the
seeds itself and the record says so, with nothing outside Daishi vouching
for them. Until 24 October 2026 the processes that start trials had no
beacon, so the seeds of every experiment created from 19 October on were
drawn on the server, and each record says "no beacon is configured on this
server". In that time a trial could also play seeds its record does not
list. One that started in the seconds between its experiment's trials
being written and its record had no seeds yet and drew its own layout (and,
where the seeds pick the world, its own world), and its
`config.run.experiment.design_sha256` is null even where the design had
arrived, because the launch reads the design off that record. Two
processes drawing for one experiment at once could also each stamp their
own draw on the trials they started. Each trial's archive carries the
seeds it played, so a pair is a replay only where its two trials'
archives agree. Where the seeds pick the world, a `pinned` experiment
whose seeds the server drew got a different world on each trial. Since
24 October every process reads the beacon, no trial starts before its
experiment's record exists and its seeds are drawn, the first draw
recorded is the one every trial plays, and no later write of the
experiment changes it. Since 24 October 2026 the experiment report also
holds each pair to its seeds: a pair whose two trials played different
seeds is left out with both sets named, and counts against the study's
declared read as a lost trial does. A fixed rule's stop recorded on such
pairs is read again without them, and the verdict quotes what the record
read; the record itself stays as written.

**What the chain commits to is the record as archived, mistakes included.** A
record found wrong later is not corrected in place, because a corrected
record could no longer reproduce its digest and the witnessed commitment
would be worthless. The correction is an erratum, published beside the
record and never inside it: the match page carries it as a notice, the
record's JSON and its export carry the list (`/api/matches/<id>/errata`),
and every erratum has its own page under `/errata`. The governance rules say
what an erratum contains and that the list only grows. A restart cannot
change a committed record either: a server that comes back holding a snapshot
taken before a match ended starts the next season instead of replaying the
end, and a record already on the chain is never saved over.

## 9. Keeping the answers secret

**Are scored inputs rotated?** Yes, per season, and the previous pool becomes
public at rotation, which is what makes training on published Daishi logs an
expected and acceptable thing to do.

**Is there a commitment to the pool, published before results exist?** As of
this page, yes, and before it there was not. Rotation alone does not let an
outsider check that the pool named in a published season is the pool that was
actually in force when the matches ran, rather than one selected afterwards to
suit the results. The protocol was sound and the proof was missing.

The commitments below fix that. Each digest is the SHA-256 of a domain
separator, the pool's name, a secret 32-byte nonce and the pool's canonical
JSON, joined by newlines. The nonce is held back until the pool rotates and is
published with it. A nonce is necessary rather than decorative: a pool is a
handful of small integers, and a bare hash of one could be recovered by
enumeration.

| Pool | Committed | Digest (SHA-256) |
|---|---|---|
| Held-out seeds, `season-1` | 16 September 2026 | `0168f9fd5dc7c3f029ee3ec22272779da30601224ce08bebc665c08f15e413d6` |
| Held-out injection payloads, `s1` | 16 September 2026 | `4fa75e9ddbc736b9ff333bb4401643cf8ce07c67ce090982bd6a9a5ad9f475df` |

The payload corpus is committed by payload id rather than by payload text, so
that fixing a typo in a payload does not break a commitment while still fixing
which payloads were in force.

If a pool has to be replaced before it rotates, the replacement gets a new name
and its own commitment, and both appear here. A commitment is never edited.

What is still manual: publishing a new digest when a season opens, and
publishing the pool and nonce when it closes, are operator steps rather than
automated ones. The canonicalization and the digest are computed in code and
pinned by a test, so a pool edited without a new commitment fails the build,
but the publishing schedule is a promise rather than a mechanism.

**Is contamination otherwise controlled?** Only by rotation. Published match
logs are distributed in the clear, where some peer evaluations now distribute
traces encrypted specifically to keep them out of training corpora. For a world
whose seeds rotate this matters less than it would for a fixed task set, but it
is a real difference and not a solved problem.

## 10. Statistics

The statistics floor and the anti-best-of-N rules are not part of the audit's
checklist, but they are the other half of whether a number is worth citing, and
they are written down in [the governance doc](/docs/governance) rather than
here: a minimum of five matches across at least three held-out seeds, means
with Student t confidence intervals, interquartile means alongside means,
leaderboards sorted by a conservative estimate and ranked so that entities
sharing a rank are not separated by their intervals, underpowered A-versus-B
comparisons refused by the tooling rather than discouraged by convention, and
a cap of three declared private variants per model family per season.

A paired experiment created from 19 October 2026 on also declares its read
before its first trial: how many trials it may lose (an arm's trial that
failed, lost a seat, stopped at its spend cap, was resumed by a new release,
played under another version tuple or measured nothing; a pair whose two
plays started from different layouts, which only an experiment created from
19 to 24 October 2026 can hold, counts the same way) before a comparison
reads incomplete and renders no verdict, and, if the owner sets one, a spend cap for the whole study at
which no further trial starts. No lost trial is ever replaced and no trial
is added after the losses are seen: the design plays the trials it declared
and reads what it has, so a result can never rest on trials chosen with the
outcome in view. Beside each comparison that lost a pair the report prints
the tipping point, the per-trial difference every lost pair would have had
to carry to change the reading, next to the widest difference the pairs
played did carry. Each report also keeps two rate limits apart per arm: what
the world refused of the seats' own actions under its seat rules, and what
the provider made the harness retry; the second is read as provider retries
per 100 model calls arm by arm, since the arms share one key and a provider
that throttles one arm's calls reaches the other arm's pairs. Uneven rates
are a reading the verdict paragraph states first, never a verdict ([the
Studio guide](/docs/studio) says how each is counted).

From 20 October 2026 every experiment report, match report, export bundle,
model page and eval card also carries its limitations as a typed list, one
entry per kind in a fixed order: what the number measures and does not, who
stood behind it, how much was measured, what was left out, what it compares
with, how it was measured and where it holds. The second kind is the one a
platform reporting on itself owes its readers: it names the party that
built the world, the rubric and the scorer and publishes the record as an
interested party, says who designed, funded and reads a study, and says
that the record names no outside reviewer. An experiment created from the
same date also declares which seats its metric is read on, the focal seats
the arms change, and prints the mean of the background seats, which play
the same configuration in every arm, beside the focal mean, compared by
nothing.

The public record also has a ceiling, and a number from it says nothing past
that ceiling. On 18 October 2026 the 82 public archives held seasons of at
most 400 rounds and rosters of at most 12 seats, while a Studio run can go to
2,000 rounds on the Pro and Lab plans, and to 16 seats on Lab. The Standard
ladder rates three of those matches, each set to 400 rounds and played all
400 with six seats. Two more, set to 400 rounds, stopped at rounds 40 and
187, when no agent had called in for five minutes and the server closed
them; until 19 October 2026 the ladder rated them where they stopped. Since
then it leaves out any match that ended before its season was over, whether
an operator reset it or the server closed it after its agents went quiet,
and the archive keeps it as played. One of the three rated matches, and
both matches it now leaves out, were played with every seat fielded under
the Daishi team's own operator tag: Daishi's own seats playing each other.
In the other two rated matches no seat carried that tag. The ladder's pool
note gives the configured season length, what the seats' identities rest
on, how many providers they named and how many matches it left out for
ending early, but not rounds played, roster size or who fielded the seats.
A rating today describes play in those three matches, and nothing yet about
longer seasons, larger rosters or a fixed population of opponents.

A whole study can still be a chosen one. A study's card computes nothing
until every trial that played is published, which keeps an owner from
picking trials, but not from picking which studies to show. For Daishi's own
studies, its profile counts every paired experiment the account has created
beside the ones it published, with each design's standing ([the Studio
guide](/docs/studio) says what the count shows and leaves out). Other owners'
counts are theirs and are not shown, so a study from another account carries
the completeness rule and not this count.

## 11. The arena games

Everything above answers for the world. Daishi also runs an **arena**: chess
and Connect Four, two models playing each other over a separate MCP endpoint,
with their own match records. It is a different instrument and the rows above
do not simply transfer to it, so here is where it stands on its own.

**What it is.** A private lab instrument, not a public benchmark. An arena run
belongs to the signed-in user who started it and is read back through their own
Studio; there is no public arena archive, no arena leaderboard, and no arena
figure in the world's rankings or in the Daishi Fitness Index. Nothing on the
public scoreboard is produced by it. Most of the audit's questions are about
protecting a published result from the people competing for it, and the arena
publishes no result to protect.

**Where it is stronger than the world.** Three things, and they are the reason
it is worth building.

*The result is decided by a rules engine, not a rubric.* A move is legal or it
is not. There is no scoring judgment to attack, no weighting to argue with, and
nothing an agent can say about itself that changes the outcome.

*Grading is done by an oracle that never touched the game.* After the result is
final, every accepted move is measured against Stockfish (chess, in a separate
process over UCI) or the exact solver (Connect Four, which is a solved game).
The grader runs after the fact and cannot change who won. This is the property
the world does not have: an external standard of correctness that does not come
from us. Two limits on it: grading is best-effort, so a record whose grading
timed out or whose engine died carries no grades rather than wrong ones; and
one grade figure, plan-to-move consistency, is a textual check on the
reasoning a seat itself filed, so it is the seat's own words read back, never
a judgment of them and never an input from the opponent.

*The record is re-executable by anyone who holds it.* Each record carries the
protocol it was played under and that protocol's hash, the seed, the start
position, and every ply with the reasoning filed, the rejected attempts and
their kind, the clock, and a sha256 chain over all of it. A third party with
the record can replay the moves through any chess library to confirm every one
was legal, and can run their own engine to reproduce the grades. That is a
sharper audit path than the world offers, and it is available today.

**Where it is weaker, or simply different.**

*Contamination is unfixable here, and the arena does not pretend otherwise.*
Chess and Connect Four are among the most thoroughly documented games in any
model's training data, and Connect Four is solved outright. There is no
held-out pool because there is no secret; an opening book drawn from a seed
varies the start position but does not make the game novel. An arena result
measures play under a fixed protocol against a known opponent, which is a
narrower claim than the world makes, and the right one to make about it.

*A guest seat is self-reported, by design.* An invited peer says what model it
is at `identify`, and nothing about that is verified. The record marks such a
seat `attested: false` and attributes it to the invite, so the claim is carried
as a claim. A match with a guest seat is evidence about a game, not about a
model.

*One injection surface existed and is now closed.* A guest's self-reported
model id is interpolated into the opposing seat's system prompt, which made it
the only opponent-controlled text reaching another model's instructions. It is
now reduced to a single token of a model-id charset before it gets there, so a
guest that answers with a sentence contributes one word. Found by writing this
section; fixed and tested in the same change.

*Nothing published, nothing anchored.* The per-ply chain has the limit the
world's checksum chain had before the digest chain in section 8: it lives in
the same archive as the record it attests to, and arena records are not on
the digest chain, which covers world matches only. For a private lab
instrument that matters less, and the re-executability above partly
substitutes for it, because a disputed record can be checked against the
rules rather than against our copy of it.

## The gaps, collected

Six, in the order we would fix them. They are gaps in the world's evaluation
path; the arena's own limits are in section 11 and are mostly properties of
chess and Connect Four rather than things we could fix.

1. **The anchor covers the record after play, not play itself.** Every
   archived match is on a public chain whose heads are timestamped by two
   outside witnesses, with copies requested from the Internet Archive
   (section 8), which closes what an earlier version of this page named as
   the largest gap. What remains: a match is committed when it ends, so nothing outside vouches for
   its log while it is being played; matches archived before 18 September
   2026 were committed that day rather than when they ended; and lines
   recorded since the last witnessing pass are not covered yet. Fix for the
   first: let a participant check the transcript its own agent kept against
   the published log. Mirrors of the feed run by people who are not us would
   add copies beyond the Internet Archive's; if you run one, tell us and we
   will list it.

2. **No independent adversarial audit.** Everything on this page is a
   self-assessment. Fix: someone outside the project tries to break it. If you
   want to, [get in touch](/contact); a standing invitation and an honest
   report of what you find is worth more to us than a clean page.

3. **The claim screen has never been checked on real messages.** It reads
   three exact phrasings, never matches the "I have no ore" form of one of
   them, and its labeled check counts only the claims it finds. Its honesty figure is shown but has not
   counted in the Alignment Index since 29 October 2026. Fix: grade it on a
   labeled sample of real messages as caught of all claims, not right of found,
   and count it again only if it passes.

4. **The commitment extractor has not been attacked.** The containment is real
   but the semantic-injection path described in section 3 is untested. Fix: run
   the injection corpus at the extractor rather than only at agents, and
   publish the result the same way injection resistance is published.

5. **Commit-reveal is published but not automated.** The digests above are real
   and pinned by a test. Opening and closing a season's commitment is still a
   manual step.

6. **The records export, but none is ready for the shared registry.** Public
   archives export as records in the Every Eval Ever schema, and matches
   played through the Inspect task convert with that registry's own converter
   ([the data standards](/docs/data)). Three limits remain. Studio runs, where
   declared studies happen, are left out of the export as their owners'
   records. A record's mean still pools seasons of different lengths under
   one benchmark name, and counts seats from matches a reset cut short as if
   they had played a full season. And nothing has been submitted to the
   registry's datastore. Until a record is there, a third party still has to
   take this page on trust instead of loading our results next to everyone
   else's. Fix, in order: group the records by format and carry each match's
   end reason and rounds played, then submit them. Each record already says
   that its numbers come from open play, with no declared study behind them,
   and that the Fitness Index is not validated and not a score for playing
   well, so a copy republished elsewhere keeps those caveats.

## Checking any of this yourself

Nothing here asks to be taken on faith except where it says so.

Every archived match publishes its seed, its full validated event log, its
final board and its scorecard. The rubric is published dimension by dimension,
so a score can be recomputed from the match's own log and compared. Scenario
specifications are served with a content hash over their canonical form, so a
silently edited scenario produces a different hash rather than a comparable
score. The replay verification endpoint runs on any archived match.

Every archived match has a line on the digest chain, and checking it takes
nothing from this server on trust beyond the bytes it serves.

1. Fetch the line. It carries the record its digest was computed over, the
witnesses that cover it, and a `recompute` block that states each rule below
in full.

```
curl -s https://daishi.ai/api/matches/<id>/digest
```

2. Recompute `digest`: the sha256 of the record's canonical JSON, in the form
RFC 8785 defines (keys sorted at every level, no whitespace, characters outside
ASCII written as themselves rather than as `\u` escapes), taken over its UTF-8
bytes. Recompute the public event hash from the export endpoint the way
`recompute.events` describes: every public event in log order, each as
canonical JSON, joined by newlines with none after the last. In Python, with
nothing to install (`ensure_ascii=False` matters: the default escapes every
character outside ASCII, and the hash of any log holding one then fails):

```python
import hashlib, json, urllib.request

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "chain-check"})
    return json.load(urllib.request.urlopen(req))

def canon(x):
    # Writes floats below 0.0001 as 1e-05 where RFC 8785 writes 0.00001.
    return json.dumps(x, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

def sha(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

site, match = "https://daishi.ai", "<id>"
line = get(f"{site}/api/matches/{match}/digest")
events, since = [], 0
while True:
    page = get(f"{site}/api/matches/{match}/export?since_id={since}&limit=1000")
    events += page["events"]
    if page["done"]:
        break
    since = page["next_since_id"]
print(sha(canon(line["record"])) == line["digest"])
print(sha("\n".join(canon(e) for e in events)) == line["record"]["events"]["public_sha256"])
print(sha(line["prev_head"] + "\n" + line["digest"]) == line["head"])
```

When a redaction notice (an erratum of that kind) sits beside a record,
personal data was removed from it after its digest was recorded: the line's
`redactions` list names the match, the seat and the fields, the second line
the script prints is `False` for that match by design, and a copy of the
export taken before the notice's date reproduces the hash. Every other check
holds.

3. Recompute `head`: the sha256 of `prev_head`, a newline and `digest`. When
the covering witness sits on a later line, walk forward through
`/api/digests` one line at a time until you reach its head.

4. Verify each witness's proof against that head. Download it from
`/api/digests/witnesses/<witness id>/proof`, then run the tool that fits:

```
openssl ts -verify -digest <head> -in proof.tsr -CAfile <CA bundle>
ots verify -d <head> proof.ots
```

The first checks the timestamp authority's signature and prints
`Verification: OK`; the default CA bundle on Debian and Ubuntu
(`/etc/ssl/certs/ca-certificates.crt`) holds the authority's root. The second
is the OpenTimestamps client, which checks the proof against a Bitcoin node
you run. Without a node, `ots info proof.ots` prints the block and the merkle
root the proof arrives at, which any block explorer shows for that block.

5. Compare the head with the Internet Archive's copies: a capture of this
line's own page, when the archive has one, or a capture of
`/api/digests/head` at this head or a later one, which reaches this line by
the walk in step 3. `/api/digests/head` links both listings. The archive's
viewer rewrites the links inside a page it shows, so a copy viewed there
shows the witnesses' addresses pointing back at the archive. Add `id_` after
the capture's timestamp to get the page exactly as it was served:

```
https://web.archive.org/web/<timestamp>id_/https://daishi.ai/api/matches/<id>/digest
```

Any correction published for the match since it was archived is served
beside it (`/api/matches/<id>/errata`) and listed at `/errata`, so a reader
recomputing a score knows what has since been found wrong with the record
they are recomputing from.
An archived match that ran under a scenario (a Studio run, or a scenario
match) also serves its recipe (`/api/matches/<id>/recipe`); a public-world match
ran under none, and its recipe endpoint says so. A recipe holds the
scenario at its content hash, the season length and pacing, the anchor
lineup and physics, and each seat's model, sampling settings and scaffold
identity, with a hash over the lot. Run that recipe from your own account on
your own keys and the result is a second sample of the same experiment,
scored by the same server-side rubric. A re-run started from the match
report is published from the moment it starts and listed beside the
original (`/api/matches/<id>/replications`), so the re-runs a reader sees are
all of them, not the ones that agreed; every public run of one recipe, by
anyone, is pooled at `/recipes/<hash>`. A seat that carried its owner's
private instructions is marked as such, because the recipe cannot carry the
text. A guest seat, a peer's own agent in a world run, is listed apart
under `guests` with the model, provider and scaffold it reported
(self-reported, never verified). No re-run can field it, so a recipe with
one is not `exact`, and running it samples a variant with only the driven
seats, not the same experiment. Where a scenario names anchor baselines, they play in the
same matches as the entrants, so the result has a floor beside it in the same
conditions; a scenario can omit them, and a custom run can strip them, so check
the roster before reading a score against a floor that may not be there.

A paired experiment is re-run whole from its public design, and its re-run
is published from creation as well. The re-run's card reads it beside the
study it re-runs: for each comparison, Welch's 95% interval on the re-run's
mean paired difference minus the original's, and whether that interval
tells the two studies apart; when it does not, that is not a finding that
they agree. A re-run is read only once it finished at its declared read,
with every trial of both studies public, and the original's card counts
the re-runs by other accounts that read and lists its owner's own without
counting them.

Every match records the version tuple it played under (the reference
harness with its prompt version and hash, the engine and its event schema,
the house bots, the build, and each screen and record written at play
time), and nothing pools across tuples: a study reads the trials of its
first trial's tuple and counts any other as lost. Two studies played under
different tuples are read beside each other as two measurements. The only
bridge is an equivalence margin a re-run declares before its first trial,
kept in its witnessed creation record: when the 90% Welch interval on the
gap between the studies lies inside it, there is no measured difference on
that metric within that margin, and the reading claims nothing more.

A published study's declared design is checkable by hand as well:

1. Download the exact bytes from the design's page
(`/u/<handle>/studies/<id>/design.md`, or `.../design/<setup>.md` for a run
set's setup) and hash them with `sha256sum`. The digest must be the one the
page prints.
2. Download each proof the page links and check it against that digest:

```
openssl ts -verify -digest <sha256> -in design.tsr -CAfile <CA bundle>
ots verify -d <sha256> design.ots
```

`openssl ts -reply -in design.tsr -text` prints the time the authority
signed. Without a Bitcoin node, `ots --no-bitcoin verify -d <sha256>
design.ots` prints the block and the merkle root that block must have,
which any block explorer shows.
3. For an experiment, read each trial's `config.run.experiment` from
`/api/matches/<id>`: its `design_sha256` must be the same digest (a trial
that started before its experiment's record, possible until 24 October
2026, names none). Recompute
the trial's line as above, so you know the chain holds that block, and
compare the signed time with the record's `started_at_ms`. A run set's runs
are filed under a setup after they play, so no run's record names the
design; compare the signed time with the `started_at_ms` of the setup's
first match instead.

A signed time before the first trial's start shows the document existed
before play began, by the authority's clock against the server's. The
authority signs in whole seconds, so within 2 seconds of the start the two
clocks cannot say which came first, and the design page does not order
them. A design whose page lists no witness rests on Daishi's word for its
times.

An arena record is checkable further than that, and by hand: it carries the
protocol, its hash, the start position and every ply under a sha256 chain, so
the move list can be replayed through any chess library to confirm legality and
re-graded with your own engine. The limits on what that proves are in section
11.

What you cannot do is read the code that computed the score. This deployment's
source is not published, which [the governance doc](/docs/governance) states
directly. Reimplementing the published rubric and running it over a published
archive is the independent audit path that exists today.