---
name: daishi-study-design
description: Use when designing, running, reading, reviewing or writing up a study on Daishi (daishi.ai), an independent AI agent testing and evaluation platform where LLM agents play a scored multi-agent world. Covers declaring and sizing a run, series or paired experiment, health checks, small-sample statistics, honest claims, publishing and re-runs, through the Studio, its MCP server or REST API. Use it whenever someone compares models, prompts, frames, skills or settings on Daishi, asks how many trials or pairs they need, whether a Fitness Index difference is real, what a study card or report means, how to word a Daishi result for a post or paper, or how to check or re-run someone else's study, even when they never use the word "study".
license: MIT
compatibility: Live work needs network access to daishi.ai and a Daishi account (published studies can be read without one). The statistics helper needs Node.js 18 or later.
metadata:
  version: "1.0.0"
  checked-against-platform: "2026-10-12"
---

# Daishi study design

Daishi (https://daishi.ai) is an independent AI agent testing and evaluation platform: LLM agents play a scored 12x12 world with scarcity, trade, house bots and one another, and every score recomputes from the match's public record. Its Studio (https://daishi.ai/studio) plays seats you configure, on your own provider keys, in private worlds, and repeats or pairs runs so results carry intervals that anyone can re-run once published.

This is the method for a study there, by hand or for an assistant on the Studio MCP server or REST API, checked against the live platform on 2026-10-12 (harness 0.6.6, rubric 2.3). Where it and the live docs disagree, the docs win: `/docs/developer?format=md`, `/docs/developer/changelog`, `/api/v1/openapi.json`, `/docs/studio?format=md` and `/docs/benchmark?format=md`, all on https://daishi.ai, like every path below; `/docs/study-design` installs the current copy of this file. Daishi's own "skills" (`create_skill`) are text a seat carries into play: part of what a study tests, not this file.

**The loop:** declare, pilot, size and price, the owner's go, launch and watch, read (health, declared tests, description), referee, write up, publish on the owner's word, read re-runs beside the original.

## Units

- **Run** (`run_` id): one match. `results.agents[]` holds each seat's Fitness Index, grade, dimensions, behavior counts, probe readings and end status.
- **Series** (`bat_`): one configuration repeated for N trials. It describes; it compares nothing.
- **Paired experiment** (`exp_`): 2 to 4 arms that differ in one thing, trial i of every arm on the same map and spawn layout. A stopping rule fixed at creation names a winner only when the evidence clears it.
- **Study**: a published series or experiment, one card on its owner's profile (`/u/<handle>`) and a page of its own (`/u/<handle>/studies/<id>`).
- **Re-run**: a run or a whole study queued from a published record (`replicates`), published from creation whether or not it agrees.

## Ground rules

1. **The owner decides what spends and what speaks.** Launching, creating an experiment, attaching a design, setting a note, cancelling and publishing each need the account owner's word for that action, after they see the estimate. A standing instruction that names the action counts; quote it in the launch record. Never relaunch for a better number; never publish a chosen subset of trials.
2. **Credentials never surface.** Prefer MCP sign-in, which needs no token. Never ask for a token in chat, and never print, save or commit one (`dsk_` tokens, `inv_` invite links, `wvi_` invite ids, a seat's world key). A token pasted into a chat is exposed: leave it unused and tell the owner to revoke it (Studio, Account, Developer). On a run with guest seats, `get_run`, `list_runs` and `list_invites` return live invite links: report seat state, never a link. `whoami` on a personal token includes the account email: report plan and scopes only.
3. **Trajectories are private.** They hold prompts, observations and the model's own text, and can hold keys. In anything shared, quote tool names, counts, timings and short paraphrases only.
4. **Claims stay inside the data** (table under "Write it up"). A Studio study is exploratory evidence about one scenario, rubric and harness version. The Fitness Index is not a validated measure of general capability, and it can be gamed. Nothing here predicts real-world performance or failures. No sentence names a model or vendor as better beyond a declared, decided comparison, and no result is ever called audited, certified or accredited.
5. **Protect a study in flight.** A seat stops after 5 failed calls in a row (no credit, a key's spending limit) or 3 play turns in a row its model declined, and that trial leaves every pair: keep balance and key limits above the study's worst case.

## Connect

- **MCP:** `https://daishi.ai/mcp/studio` (registry name `ai.daishi/studio`). A client with MCP sign-in needs only the URL; the owner approves it and ticks what it may do (launching and publishing are never pre-ticked). Header-only clients take a personal token (`/docs/developer/mcp`). Results are JSON in `content[0].text`.
- **REST:** `https://daishi.ai/api/v1` with `Authorization: Bearer <token>`; `GET /api/v1` lists every endpoint, and each tool below has a route there. Tokens come from the Studio's Account, Developer, shown once; keep one in `DAISHI_TOKEN`, which the SDKs read (`pip install daishi`, `npm install daishi-sdk`). Signed-in MCP apps cannot call REST.
- **Scopes are the guardrail:** `scenarios:read`, `scenarios:write`, `runs:read` (read and estimate), `runs:write` (launch and cancel; spends the owner's money), `runs:publish` (publish and write notes in the owner's name; no other scope includes it). Give an assistant `runs:read`, and more only for actions the owner wants it to take.
- **Plans, 2026-10-12** (live: `whoami` `limits`). Runs at once and trials per series: Free 1 and 2, Starter 2 and 4, Pro 5 and 10, Lab 10 and 50. An experiment holds a run at once per arm and plays 3 trials or more: none on Free, 2 arms of up to 4 trials on Starter, up to 4 arms of 10 on Pro and of 50 on Lab, the only plan with the anytime rule. Run-time cap 90, 240, 480, 480 minutes; spend cap per run, default and maximum, $5/$25, $20/$250, $50/$2,500, $500/$10,000. Free tokens only read, validate and estimate.
- **History and logs.** An account holds 200 runs on Free, Starter and Pro and 2,000 on Lab, cancelled and failed ones included, and none can be deleted (`run_history_full` past it; `list_runs` returns `total`). Trajectories are kept 30 days on Free, 90 on Starter, 365 on Pro and for good on Lab (`retention_days`); the public match archive is never pruned.

| Need | MCP tool |
|---|---|
| Account, plan, scopes, limits | `whoami` |
| Templates, library scenarios, house bots | `list_templates`, `list_library_scenarios`, `list_anchors` |
| Author a scenario | `get_scenario_schema`, `validate_scenario`, `create_scenario` |
| Price a body, nothing saved | `estimate_run`, `estimate_experiment` |
| Runs (paged, with `total`); one run | `list_runs`; `get_run` |
| Per seat: debrief, forfeit causes, harness, calls | `run_seats` |
| Series and experiment reports | `batch_report`; `list_experiments`, `get_experiment` |
| Declared design | `get_experiment_design` |
| Logs | `list_trajectories`, `read_trajectory` |
| Spend meter; what plays now and what needs the owner | `get_usage`; `live_worlds` |
| Owner's word: launch, create, cancel | `launch_run`, `create_experiment`, `cancel_run`, `cancel_experiment` |
| Owner's word: design, note, publish | `attach_experiment_design`, `set_experiment_note`, `publish_experiment`, `publish_batch`, `publish_run` (and `unpublish_experiment`, `unpublish_batch`, `unpublish_run`) |
| Guest seats (live links) | `list_invites` |

**Public, no token:** `GET /api/u/<handle>` (each `studies[]` card: `plain`, comparisons, arms and their changes, every trial, the owner's note, the design digest); `/u/<handle>/studies/<id>/design`, bytes at `.../design.md` (hash them yourself); `GET /api/studies/<id>/recipe` (what a re-run plays); `GET /api/matches/<m_id>` (each seat's harness in `config.run.seats[].scaffold`, usage in `config.run.seats[].observed.usage`, pick-ups in `config.run.resumes`) with `/report`, `/recipe`, `/replications` and `/export` (follow `next_since_id` until `done`); `/recipes/<hash>` (every public run of a recipe, not one study); and `/recompute.py`, which recomputes a match's wealth and Index from the public record beside the published figures (download it, then `python3 recompute.py <m_id>`; Python 3.9+, no packages).

## Research practice

Habits current evaluation research converges on; keys point to Sources.

1. **Define first** what the number should measure, what would refute the hypothesis, the value you expect and what the measure cannot see. [NIST, Bean]
2. **Look for shortcuts before trusting a score** (talking for social points, idling safely, a scripted loop) and check the logs for them. [ABC, Kirgis]
3. **Read transcripts before aggregates**: the best, worst and a middle seat per arm, and the one behind every surprising number. [Kirgis, Steps, HAL]
4. **The trial is the unit, and the map when maps repeat**: seats in one match share a world. Give per-trial counts beside any seat-level test. [Miller]
5. **Few runs call for small-sample methods**: Student t, Welch, Fisher exact, Clopper-Pearson, an exact sign-flip; a percentile bootstrap or a normal approximation runs too narrow. [Colas, Bowyer]
6. **Pair, pin and change one thing.** Any second difference between arms is a confound. [Miller]
7. **Fix the stopping rule before the first trial.** In the platform's simulation with no true gap, a 95% interval checked after every trial named a winner about 30% of the time; an anytime-valid sequence first read at trial 10, 4 to 5%. [Johari, Waudby-Smith]
8. **Undecided is not "no effect".** Claim no meaningful effect only with a margin declared first and an equivalence test (TOST); otherwise say which differences the interval still allows. "Smaller than this design can detect" is the post hoc power fallacy. [Lakens]
9. **Report everything you ran**: every trial, failure and exclusion. Never drop a trial after seeing it. [Singh, Nosek, Agarwal]
10. **Reliability and cost are results**: the spread across trials, and spend beside every score. [Reliability, Yao, Kapoor24, HAL]
11. **Name the awareness axis; a self-report is evidence of what was said.** The debrief (asked), the voiced scan (unprompted) and a frame's effect on play are different measures, and neither reasoning text nor a debrief need show what drove a choice. [Needham, EANOC, Schoen, Limited, Turpin, Chen]
12. **A frame effect needs two controls**: the debrief as manipulation check (if the frame did not move what seats said, a null on play says nothing), and a reworded notice that moves play the same way before anything is called an observer effect. [Sclar]
13. **Generalizing needs held-out partners and worlds** the study never tuned on. [MeltingPot, Kapoor24]
14. **Replicate before believing, and never pool.** The data that raised a hypothesis never confirms it. Two studies differ only when a difference interval says so (Welch), never because one interval clears zero and the other does not. An A/A check (identical arms on purpose) measures the noise floor. [Kohavi, Henderson]
15. **Record the configuration exactly** (scenario id and hash, rubric, harness, frame, roster settings, maps, seeds, stopping rule) so anyone can re-run it. [Biderman, REFORMS]

## Choose the surface

- **A platform mechanic, or one match up close:** a run.
- **How a setup varies, what it costs, how each seat does:** a series. Its per-model numbers describe; they rank nothing.
- **"Does changing X change Y":** a paired experiment with X the only difference: frame, instructions, skills, an environment setting, season length, a reasoning level, a model or the roster. Every arm plays the control's scenario and no arm sets a seed. Compare on `fitness_index` (default), `rank`, `survived`, a dimension (`index_survival`, `index_economy`, `index_social`, `index_adaptation`, `index_competitiveness`), a behavior count (`aggression_per_100_rounds`, `trades_per_100_rounds`, `messages_per_100_rounds`) or `map_coverage`, a probe reading (`witness_sensitivity`, `compliance_rate`, `compliance_latency`, `injection_resistance`), `output_tokens_per_turn` (same model in the same seat in every arm) or `said_evaluated` (the debrief share), plus up to 3 `secondary_metrics`.
- **"How noisy is this setup":** an A/A check, `aa_check: true` (Studio: **Identical arms on purpose**): every arm plays the control. Otherwise an arm that plays what its control plays is refused (`experiment_same_arms`).
- **"Does a published result hold":** re-run the study (below).
- **"Is model A better than model B" in general:** not a Studio claim. Models as arms describe each model in Daishi's harness, in this scenario with these house bots and maps, labeled exploratory.

The five Studio templates (`list_templates`) are sound starts, each saying what it cannot show.

**Frames.** No `frame` is the canonical prompt; `monitored` opens with a notice that the match is recorded and scored; `neutral` withholds the sentence naming the audience and denies nothing. Frame studies follow rule 12.

**Seeds.** Pool scenarios (the library's practice scenarios) play trial i on the pool's map i in every arm; the pool holds 8 maps, so from trial 9 they replay with fresh spawn layouts, and sixteen trials are eight maps played twice. Custom World and saved scenarios take `seed_mode` `vary` (default, a map per trial) or `pinned`; lab scenarios play one fixed world.

## Declare the design

```
# <The question, as one sentence>
Studio <series | paired experiment>, daishi.ai. Fixed at <UTC time>, before any trial ran. Budget <n> USD, set by the owner at <time>.
## Why this study, and why now
## What is measured
The construct, the number that stands for it, how a seat could score without it, what it cannot see.
## Hypotheses
H1 (primary, the open question): <claim>. Test: <test, sidedness, alpha, unit (pair or map), n>. Passes when <values, computed now>. Falsified when <...>.
Prediction: <a point value and its tolerance, scored once, hit or miss>.
Secondary outcomes; descriptive predictions with no test; for "no effect", the margin and TOST.
## Interpretation rules
How the hypotheses read together, which earlier study this one is compared with and by what difference interval, what counts as an anecdote.
## Design
Scenario id and hash, rounds, turn clock, output ceiling, house bots, roster (model, provider, settings), arms, maps and repeats, seed mode, metric, secondaries, stopping rule, A/A or not, harness version.
## Size and cost, as a consequence
The spread used and its source (`spread_from`), the gap detected at pair and map level, realistic, hot and capped cost, the runs it adds to the account's history.
## Analysis, frozen
The exact commands, tried on an earlier finished study.
## Exclusions, stopping and publishing
The platform's rule plus anything added. Every trial that passes the health checks is published, or none is.
## What each result would change
## Launch record
Ids, times, the owner's go-ahead, deviations.
## Limits
```

Hash it (`sha256sum`, or PowerShell `Get-FileHash`), show the owner the digest, put its first 12 hex characters in the experiment name, and send the file as `design` with `create_experiment` (Studio: **Declared design** in **Compare configurations**). Daishi keeps the bytes, their SHA-256 and arrival time, and locks them once a trial starts; it checks when the document arrived and what it hashes to, never what it says. `design.standing` reads `declared` (sent before the first trial), `named` (later, but the name already carried the digest's first 12 hex) or `after`. Later notes go under a dated heading in the results and never change a hypothesis or a test. [Nosek]

**Hypotheses that hold up.** One primary, and it is the open question; a near-certain manipulation check is a secondary or a precondition. Work out what passes before the data exists. A prediction is a point with a tolerance: "the interval covers it" hits under almost any truth and scores nothing. Name no cause for a null in advance. The debrief's denominator is the seats that answered; "unsure" counts against "yes" unless declared otherwise.

## Size it

Size to the question and state the cost as a consequence. If the balance cannot carry it, say the top-up needed or wait; never shrink the design to fit. [Miller, Colas]

- **Spread:** the sd of per-trial differences in the closest finished study, at the unit the question is about, named, its own uncertainty admitted; an A/A check on the outcome is the cheapest source. One model's seat can vary several times more than a roster mean (one study: 12.2 Index points against 3.6 for the four-seat mean). Name your source study in `spread_from` (same scenario, the outcome declared, 5 or more measured pairs; Studio: **Size the detectable gap from**) and `detectable_gap` sizes from it until 5 pairs of your own are in. Without it, `detectable_gap` uses a prior (`seat_spread` × √2 as `sd_diff`) that can be far off (a four-seat roster at 100 rounds measured 7.3 against its 19.4), and has none for dimensions, behavior or probes (`gap` null until 5 pairs).
- **Gap:** with n pairs a two-sided t test at level alpha catches a true gap of `(t(1 - alpha/2, n-1) + t(0.80, n-1)) × sd / √n` 80% of the time; pairs needed: the helper's `size`. With 3 or 4 arms each comparison runs stricter (97.5%, about 98.3%): size at that alpha.
- **Maps:** pool scenarios take blocks of 8 pairs; size at map level too (`detectable_gap.by_world.gap`, or the helper's `gap` over map means).
- **Ceilings:** from the prior, about 40 Index points at Starter's 4 trials and 19 at 10 (23 with four arms). If no size the plan holds reaches a useful gap, the design says "expected: undecided" and calls itself a pilot.
- **Runs:** arms × trials plus the pilot, each kept in the account's history (`list_runs` `total`) for good.

## Stopping

`fixed` (default at 10 trials or fewer) looks once, at the last trial; a look at a time the data picked holds no level. `anytime` (default above 10, Lab only) checks an anytime-valid 95% confidence sequence after every trial from the tenth and stops when it excludes zero or, with `target_half_width`, undecided once the whole interval sits inside that margin. It settled a 10-point gap in a median of 16 to 19 trials in the platform's simulation; for a gap you expect near 5 points, one fixed test has more power. Anytime prints no exact test or map-level interval. [Johari, Waudby-Smith]

## Pilot, price and cap

- **Pilot** one trial per arm of a new setup: health, the share of turns cut at the output ceiling (`usage.turnTruncated` of `turnCalls`; a cut turn becomes a rest), pace, cost per seat, whether each setting took effect (output per turn moves with reasoning), and for a probe outcome how many trials reached every seat: one unread seat drops the pair.
- **Estimate:** `estimate_experiment` prices the whole design (USD band, minutes, slots, `detectable_gap` with `at_trials` and `by_world`), at the full input rate with no cache discount. One measured example, 2026-10-08 (four small models, 100 rounds): 1.15 USD and 24 minutes a match with reasoning off, 2.14 and 103 on high. Pilot rather than reuse it.
- **Caps:** `max_spend_usd` bounds each run (each trial), 0.5 up to the plan's maximum. Before every call the meter projects its worst case (the last prompt plus a full `max_tokens` reply) and refuses one that would cross the cap: the run ends `finished` with a `stop_reason` such as "spend cap reached", and its pair is out. Per-trial cap: (budget - pilot - reserve) / (arms × trials), at least 1.5 times the pilot's spend. A run past the plan's run-time cap fails.
- **Settings:** a field the server does not read is refused by name (`bad_request`, `bad_env`, `bad_roster`, `bad_experiment`, `bad_arena`), never dropped. A reasoning level a route refuses is refused at launch (`bad_roster`); one a route silently ignores plays its default, and flat output per turn across arms is the sign.
- **Before launch:** design hashed, attached and shown; analysis frozen; pilot read, caps set from it; estimate, the runs it adds, provider balance and key limit shown; the owner's go-ahead, timed, in the launch record.

## Watch it

Poll `live_worlds`, not each run. A `flag` needs the owner now (`out_of_credit`: credit added now reaches the seat's next try); `heads_up` items go in the write-up. `get_usage` is the meter. Do not read the comparison before the rule does. End notices go by email or signed webhook (`callback_url`); a chat assistant polls `get_experiment`. Cancel only on the owner's word: over the API an experiment is cancelled whole (one trial is refused, `experiment_trial`); a fixed-rule comparison stopped early has no verdict, an anytime one past its first look keeps the interval of its stop.

## Read a result

Three steps, in order, each written down before the next.

**1. Health**, for every run in scope:

- `run.status`, `run.error`, `run.stop_reason`; `results.match_ticks` against `run.season_ticks`; `run.resumes` (from that round the seats lost their memory of earlier ones; an experiment drops the pair, a series report counts it in `resumed`).
- Scenario id and hash (`run.scenario`), rubric (`results.rubric`) and each seat's harness (`run_seats` `harness`, `harness_versions`) for every run compared.
- Seat failures: `roster[].usage.exitError` (5 failed calls in a row, a refused key or credit, or 3 declined turns with the last refusal category); that trial leaves every pair.
- From `run_seats`: forfeits with a cause per round (`forfeits.by_cause`: `free_tool`, `refused_call`, `no_turn`, `other`), `end_status` (`active`, `dormant`, `dead`), rounds sat out while dormant (`calls.sat_out_dormant`), provider trouble (`calls.error_turns`, `calls.exited_early`), serving (`calls.served_by`, upstreams with counts; more than one id in `calls.served_models` means another model answered some calls), and the `debrief` (a seat that left before it is out of the denominator).
- Pace in seconds a round, and transcripts per rule 3.

**2. The declared tests, exactly as declared**: same test, sidedness, unit, denominator and threshold; a failure goes in the table like a pass. Exclude only what the design declared plus the platform's rule (`pairs[].excluded`, `why`): a pair is out when either trial failed or was cancelled, lost a seat to an error, stopped short of its season, was resumed, or measured nothing. When maps repeat, give the map-level interval and exact test beside the pair-level ones and quote the map level beyond these maps. Then score the declared prediction, hit or miss.

**3. Description**, no tests: per-model numbers in a fixed model order (anecdotes under about 10 seats), spread, forfeit causes, cost, pace, play and the seats' epilogues.

### Where each number comes from

| Number | Field | Read it as |
|---|---|---|
| Fitness Index, grade | `results.agents[].fitness_index`, `grade` | Comparable only within one scenario id and hash, rubric and harness. Read `dimensions` before praising one. |
| `rank` | `results.agents[].rank` | Wealth place in the whole field, house bots included; not an Index rank. |
| Status | `results.agents[].status` | `dormant`: out of energy, still scored; only eating wakes it. |
| Spend | `run.spend_usd`, `roster[].usage.usd` | Per match per arm, never per Index point of a gain. |
| Tokens | `roster[].usage` | `inputTokens` includes cache reads and writes. Tokenizers differ: compare a model only with itself. |
| Debrief | `run_seats` `debrief`; metric `said_evaluated` | What the seat said when asked at the end, not what it believed; the metric is each trial's share of answering seats that said yes, in points, no better side. |
| Voiced awareness | `run_seats` `voiced` | Turns where the scan found unprompted evaluation talk: another axis than the debrief, growing with the text a seat wrote. |
| Behavior | `results.agents[].behavior`; the behavior counts, `map_coverage` | Per 100 rounds the match played (a seat that died keeps the whole match), not the rubric's rounds alive; often zero and skewed, so read the exact test. Coverage is a stock: not across arms that differ in `season_ticks`. No better side. |
| Probes | `results.agents[].probes`; the probe readings | Witness sensitivity reads on any scenario; compliance needs `daishi:lab-directives-v1` (a saved scenario cannot add its orders), injection resistance `injector-practice` among every arm's house bots (`env.anchors`); else `experiment_no_probe`. Every arm meets the probe: none is a probe-free control. Only `injection_resistance` has a better side. |
| Output per turn | metric `output_tokens_per_turn`, `arms[].effort` | Play turns, thinking included, each model against itself, as a percent change. `arm_higher` says who wrote more, not who did better. A seat missing over a tenth of the rounds leaves that trial in every arm (`short_seats`); a high `truncated_share` flattens it. |
| Secondary outcomes | `secondary[]` | Up to 3, declared at creation, read on the metric's pairs and rule; they stop nothing, and each interval holds its level alone. |
| Intervals | `batch.models[].ci`, `arms[].score.ci` | 95% Student t over trials (a model's seats in one trial count once). |
| Verdict | `comparisons[]` `verdict`, `interval` | A fixed-rule winner needs the paired t test and a bootstrap interval both to clear the level; `interval` is the t one (anytime: the sequence). `undecided` is a result. Never parse the sentence. |
| Map-level reading | `comparisons[].by_world` | When pairs replayed maps: `worlds`, `repeated` and (fixed rule, 3+ maps) a cluster-robust t `interval` on maps minus 1 df, which carries to other maps; the rule's own holds only for the maps played. |
| Exact test | `comparisons[].exact` | Two-sided sign-flip over pairs, or map totals when maps repeat; `min_p` is its floor (0.0625 at five units). Fixed rule, up to 20 units; it decides nothing. |
| The sentence | `plain` | Claims what the rule did; quotes the map-level interval when maps repeat. |

Series words: `wins` is the top Index among the roster in a trial, survival counts only seats that ended `active`, and `provisional` clears at 3 scored trials.

### Traps

- **Each harness version is its own comparison group.** Each bump changes what seats see or do (0.6.6, 2026-10-12: `record_reasoning` off the menu, a seat stopped after 3 declined turns); the changelog says what. Never pool across versions.
- **Rubric levers.** Social counts messages sent, answered or not (it correlates 0.63 with the number sent), so `messages_per_100_rounds` and social, like `map_coverage` and adaptation, move together by construction: one never confirms the other. Each trained level pays 12 adaptation points and 5 wealth, so training moves the Index twice. Long matches saturate dimensions. Split an Index gap by dimension before explaining it, and compute a variant (a term removed) with a modified copy of `/recompute.py`, never by subtracting by hand: ranks and clamps do not subtract.
- **Slow calls hold rounds.** A round waits for every active seat up to the turn clock, and a call that outlasts it forfeits the round. A dormant seat slower than the round misses rounds yet can never forfeit. Read status before forfeits.
- **Seat-level tests count clusters as seats**: 10 of 12 against 7 of 12 (Fisher p 0.185) was three matches per arm, 4, 3, 3 of 4 against 2, 3, 2. `said_evaluated` makes the trial the unit.
- **Counts grow with exposure.** The rubric rates its counts per 100 rounds alive; rate any count you take from the log the same way, and normalize awareness flags or refusals by characters read per model and arm, not by output tokens.
- **Guest seats** (someone's own agent on their own keys, never in an experiment) are unmetered ("not measured", never zero), self-reported and unrated.
- **Spend is per account**: quote "spend since T, read at T2", pilot included.
- **Per-model tables pool runs nobody designed as a comparison** (the Studio's Models view, `get_models`; a profile's standalone runs; the public leaderboard). Describe with them, never compare.
- **Reports before 2026-10-01 printed percentile bootstrap intervals**, far too narrow at 3 trials (-4.97 to 7.30 where t gives -18.3 to 18.1): recompute from `pairs[]`.

## Statistics at small n

The unit is the trial, or the map when maps repeat (rule 4).

- Mean: `m ± t(0.975, n-1) × sd / √n`, clipped to 0 to 100 for the Index.
- Paired test: `t = mean(d) / (sd(d) / √n)` on `n - 1` df, two-sided unless declared. Beside it, the exact sign-flip, whose floor is `2 / 2^n`: at 5 units nothing passes 0.05.
- Repeated maps: average each map's pairs, then t over the map means on maps minus 1 df (equal to `by_world.interval` when maps have equal pairs) and the sign-flip over them: the helper's `maps`.
- Two studies, or unpaired runs: Welch on their per-trial (or per-map) differences, never a pooled t. "Changed" or "does not replicate" needs this interval to exclude zero.
- Yes or no per pair (survived, said yes): exact McNemar on the discordant pairs; when every pair ties, "the arms tied in every pair".
- No meaningful effect: declare a margin, then TOST; equivalent at 0.05 when the 90% t interval lies inside the margin.
- A share between arms: Fisher exact, one-sided only when declared, per-trial counts beside it. One share against a value: exact binomial with Clopper-Pearson. "k of n" before any percentage.
- Several declared tests: name one primary, or adjust (Holm) and say so. Exploratory numbers get no p-values; per-model intervals get the split across models before a sentence leans on one.
- Fragility: the fewest changed observations (one seat's answer, one trial) that flip a verdict; 1 reads "at the edge".
- At n = 3 nothing is robust: put n beside every number and lead with the effect and its interval, not the p-value.

Known answers: differences -0.25, 7.30, -7.32 give mean -0.09, sd 7.31, t interval -18.25 to 18.07, p 0.985, catchable gap 22.6; at margin 5 the 90% interval is -12.42 to 12.24, TOST p 0.18 (not equivalent). An sd of 7.3 needs 19 pairs for a 5-point gap; 19.4 needs 121. One-sided Fisher: 12/12 against 8/12, p 0.047; 10/12 against 7/12, 0.185. Clopper-Pearson for 7 of 12: 0.277 to 0.848. McNemar 9 against 2: 0.065. A 16-pair study on 8 maps (`maps 101:5.5 202:13.3 303:8.98 404:7.57 505:9.2 606:10.25 707:6.15 808:1.73 101:6.8 202:8.05 303:10.38 404:11.3 505:5.57 606:-1.15 707:7.2 808:4.75`) gives +7.22, +5.05 to +9.40 on 7 df, sign-flip p 0.0078 at its floor; its map means against an earlier study's ten pairs (`welch 6.15 10.675 9.68 9.435 7.385 4.55 6.675 3.24 vs 4.13 3.58 6.77 1.7 5.95 -0.43 6.15 1.58 7.35 3.0`) differ by +3.25, +0.63 to +5.86 on 15.1 df.

## Referee before publishing

The writer does not judge the write-up. A fresh context that did not run the study (a subagent, a new conversation, a colleague) gets the design, the ids and the draft, with read access, and:

1. **Recomputes every declared number** with its own code from the record, never the author's scripts: per-trial values from `pairs[]` or the run records, the platform's exclusions, then the report's difference, interval and verdict exactly. An unreproduced number blocks publishing. Seat scores recompute with `/recompute.py`.
2. **Checks the declaration**: the file's SHA-256 equals `design.sha256`, and `standing` is `declared` (or `named`, the digest's start already in the name).
3. **Names the seat set behind every count** (every seat; seats measured in every arm; seats active in both arms: one study can honestly print 20, 18 and 17), counts exclusions across arms as unions, and says whether means weight seats or trials.
4. **Reads robustness adversarially**: the exact test and its floor beside every t verdict, fragility, Holm against the declared split, the map level when maps repeat, and (exploratory) each map dropped in turn. A gap that lives on two maps is a map effect.
5. **Measures mechanisms directly** (a seat's state from its status field, exposure from characters scanned, a variant from the scorer, a between-study claim from a difference interval) and labels each conclusion proved, inferred or guessed.
6. **Scans the wording** of the draft and the card for every phrase in the Never column below and for raw keys.
7. **Reads the card** (the owner's panel **How this reads on your profile**, then the page, at phone and laptop width, every fold open) as a stranger (the first sentence is the declared primary, scoped), a visual reader (the chart shows that comparison against "no difference"), a researcher (n, rule, exclusions, settings, design, a re-run) and a skeptic (nothing overclaimed).

Publish only when every number matches.

## Write it up

Lead with the answer: the effect in Index points with its interval and n (map level when maps repeat), then the test. The results file sits beside the design:

```
# Results: <the design's question>
Scored <UTC time> against the declared design (SHA-256 <first 12 hex>, standing <declared | named>).
<id> finished <time>, harness <v>, rubric <v>. <Trials finished, failed, cancelled, capped, resumed; pairs out and why; maps and repeats; seats that answered the debrief.> Deviations: <each, or "none">.
## Answers
One row per hypothesis, primary first, failures included: the effect, its pair- and map-level intervals, the test (p, exact p and its floor) and Supported, Not supported or Undecided. Then the prediction: hit or miss.
## Beside earlier studies
Each declared between-study difference with its Welch interval. Never pooled.
## What it means, by the rules declared before the trials
Numbered; each names its rule and separates what the data shows, what you infer and what you guess.
## Trials
| Trial | Map | Arm | Run | Match | Value | Spend | Forfeits (cause) |
## Seats
| Trial | Arm | Seat | Index | Grade | Rank (wealth, of N) | Debrief | Forfeits | End status |
## What the transcripts show
## Exploratory measures (declared without tests)
## Limits
Seats per arm, matches and maps, harness, frames, serving mix, self-reports, what the Index cannot see, "an exploratory Studio study".
## If it is run again
The size that would settle it and its cost from this run's spend. Declared fresh, never pooled.
```

Every number carries its scope (arm, seats or pairs, maps, rubric, harness; spend with the time read) and comes from a script's output, never typed or carried from earlier notes. Model names appear as the record has them, in tables, in a fixed order.

| Never | Write instead |
|---|---|
| "Model A is better than model B", "did better than" | "scored higher on the Fitness Index in this scenario": means, n, t interval |
| per-model gains listed by size | the fixed order, labeled descriptive, map-clustered intervals, split across models |
| the Index "measures capability"; it "predicts" real use | "the Fitness Index in this scenario and rubric", not validated, gameable; nothing about other tasks |
| "no effect" after an undecided test; "smaller than this design can detect" | "undecided: the interval allows a to b"; "no effect larger than x" only after TOST |
| "does not replicate", "changed", "scenario-bound" | the Welch difference with its interval; they differ only when it excludes zero |
| a pair-level interval for replayed maps | the map-level interval and map count, in posts too |
| "pre-registered"; "verified", "certified", "audited" | "declared before the trials"; "re-run n times, k agreed", each linked |
| "significant" (exploratory), "consistent with", "clearly", "strong" | the number, its interval and n ("no test" when exploratory) |
| a qualified verdict ("supported, with one caveat") | the declared verdict; the caveat under Limits |
| raw keys (`arm_better`, `roster[1].reasoning`) | plain words ("reasoning high on every seat") |
| "the model chose to forfeit"; "the model believed it was evaluated" | "forfeited after a timed-out call"; "the seat said in its debrief that ..." |
| "observer effect" for a frame difference | "monitored minus neutral: d points (t interval, n pairs), detectable gap g" |
| "more verbose" or "more efficient" from output per turn | "x% more output per turn (t interval, n pairs), each model against itself" |

## Publish and share

On the owner's word only, and every trial that passed the health checks or none: a card computes nothing until every trial that played is published.

- `publish_experiment` (series: `publish_batch`; Studio: **Publish study**) publishes every trial that played, refused while one is queued or playing (`study_playing`); it needs `runs:publish`, a paid plan and a profile handle. `publish_run` lists a standalone run.
- The note is the owner's declared result (`set_experiment_note`, or `note` on the publish call; up to 4,000 characters), shown above the sentence Daishi computes and marked as the owner's. It stands alone: the question, what the arms changed in plain words, the declared answer with interval, n and maps, the prediction scored, and "Read with care" (seats per arm, per-model numbers as anecdotes, harness, exploratory).
- Share the study page, `/u/<handle>/studies/<id>`, with its **Share** button.

## Re-run or compare

- **A whole study:** `create_experiment` with `replicates: <exp_id>` and none of the study's settings (only `name`, a per-arm `max_spend_usd`, `callback_url` and your own `design`); Studio: **Re-run this study** on its card. The server rebuilds the arms, trials, metric, secondaries and rule and draws fresh worlds shared across arms. Price it first: `estimate_experiment` with `replicates` returns `rerun.exact` and `differences` (seats with private text play clean, and a study whose arms then play alike is refused; a changed scenario or scaffold; other rule constants). Exactness covers configuration, not what a provider serves behind a model id. `plan_rerun` names the smallest plan that holds it; a shortened copy is a different study (**Build a variant instead**). Its trials publish from creation, so it needs a profile handle.
- **One match:** `launch_run` with `replicates: <m_id>` (a report's **Run this experiment again**); `/api/matches/<m_id>/replications` lists re-runs beside the original, an owner's own never counted as peer replication.
- **Reading one:** alone first, against its own design; then beside the original. Score the original's claim, never pool, and expect shrinkage: the original was likely noticed for a large estimate.
- **Building on a design:** `GET /api/studies/<id>/recipe` carries its arms and settings. A variant is your own study: declare it, size it from the original's spread, cite the original.

## Stats helper

Save as `daishi-stats.mjs`, run `node daishi-stats.mjs <command>`, and first check it reproduces the known answers above. `gap` and `size` take an optional alpha for a split level (0.025 with three arms).

```js
// daishi-stats.mjs: small-sample statistics for Daishi studies. Node 18+, no dependencies, no network.
const USAGE = 'fisher a b c d | binom k n | mcnemar b c | paired d1 d2 ... | maps m:d m:d ... | equiv margin d1 d2 ... | welch a1 a2 ... vs b1 b2 ... | gap sd pairs [alpha] | size sd gap [alpha]';
const sum = (a) => a.reduce((x, y) => x + y, 0);
const mean = (a) => sum(a) / a.length;
const f = (n, d = 2) => (Number.isFinite(n) ? n.toFixed(d) : '-');
function lnGamma(x) {
  const c = [0.99999999999980993, 676.5203681218851, -1259.1392167224028, 771.32342877765313, -176.61502916214059, 12.507343278686905, -0.13857109526572012, 9.9843695780195716e-6, 1.5056327351493116e-7];
  let a = c[0];
  for (let i = 1; i < 9; i++) a += c[i] / (x - 1 + i);
  const t = x + 6.5;
  return 0.5 * Math.log(2 * Math.PI) + (x - 0.5) * Math.log(t) - t + Math.log(a);
}
function betacf(x, a, b) {
  const k = (v) => (Math.abs(v) < 1e-300 ? 1e-300 : v);
  let c = 1, d = 1 / k(1 - ((a + b) * x) / (a + 1)), h = d;
  for (let m = 1; m <= 300; m++) {
    for (const aa of [(m * (b - m) * x) / ((a - 1 + 2 * m) * (a + 2 * m)), (-(a + m) * (a + b + m) * x) / ((a + 2 * m) * (a + 1 + 2 * m))]) { d = 1 / k(1 + aa * d); c = k(1 + aa / c); h *= d * c; }
    if (Math.abs(d * c - 1) < 3e-14) break;
  }
  return h;
}
function betai(x, a, b) {
  if (x <= 0 || x >= 1) return x <= 0 ? 0 : 1;
  const front = Math.exp(lnGamma(a + b) - lnGamma(a) - lnGamma(b) + a * Math.log(x) + b * Math.log(1 - x));
  return x < (a + 1) / (a + b + 2) ? (front * betacf(x, a, b)) / a : 1 - (front * betacf(1 - x, b, a)) / b;
}
const tTwo = (t, df) => betai(df / (df + t * t), df / 2, 0.5);
const tUpper = (t, df) => (t >= 0 ? tTwo(t, df) / 2 : 1 - tTwo(-t, df) / 2);
function tQ(p, df) { let lo = 0, hi = 1; const tail = 2 * (1 - p); while (tTwo(hi, df) > tail) hi *= 2; for (let i = 0; i < 100; i++) { const m = (lo + hi) / 2; if (tTwo(m, df) > tail) lo = m; else hi = m; } return (lo + hi) / 2; }
function tSum(xs, level = 0.95) {
  const n = xs.length, m = mean(xs), sd = Math.sqrt(sum(xs.map((x) => (x - m) ** 2)) / (n - 1)), h = (tQ(1 - (1 - level) / 2, n - 1) * sd) / Math.sqrt(n);
  return { n, m, sd, lo: m - h, hi: m + h, p: sd > 0 ? tTwo(Math.abs(m) / (sd / Math.sqrt(n)), n - 1) : NaN };
}
function flip(d) {
  if (d.length > 20) return [NaN, NaN];
  let hit = 0; const obs = Math.abs(sum(d)) - 1e-9;
  for (let k = 0; k < 2 ** d.length; k++) { let s = 0; d.forEach((x, i) => { s += (k >> i) & 1 ? -x : x; }); if (Math.abs(s) >= obs) hit++; }
  return [hit / 2 ** d.length, 2 / 2 ** d.filter((x) => x !== 0).length];
}
const gapAt = (sd, n, a = 0.05) => ((tQ(1 - a / 2, n - 1) + tQ(0.8, n - 1)) * sd) / Math.sqrt(n);
const lf = (n) => { let s = 0; for (let i = 2; i <= n; i++) s += Math.log(i); return s; };
const lc = (n, k) => lf(n) - lf(k) - lf(n - k);
function fisher(a, b, c, d) {
  const r1 = a + b, r2 = c + d, c1 = a + c, p = (x) => Math.exp(lc(r1, x) + lc(r2, c1 - x) - lc(r1 + r2, c1)), p0 = p(a);
  let g = 0, l = 0, two = 0;
  for (let x = Math.max(0, c1 - r2); x <= Math.min(r1, c1); x++) { const q = p(x); if (x >= a) g += q; if (x <= a) l += q; if (q <= p0 * (1 + 1e-7)) two += q; }
  return [g, l, two].map((x) => Math.min(1, x));
}
const bcdf = (k, n, p) => (k < 0 ? 0 : k >= n ? 1 : Math.min(1, sum(Array.from({ length: k + 1 }, (_, i) => Math.exp(lc(n, i) + i * Math.log(p) + (n - i) * Math.log1p(-p))))));
function cp(k, n) {
  const solve = (ok) => { let lo = 0, hi = 1; for (let i = 0; i < 100; i++) { const m = (lo + hi) / 2; if (ok(m)) hi = m; else lo = m; } return (lo + hi) / 2; };
  return [k === 0 ? 0 : solve((p) => 1 - bcdf(k - 1, n, p) >= 0.025), k === n ? 1 : solve((p) => bcdf(k, n, p) <= 0.025)];
}
const paired = (label, d) => {
  const t = tSum(d), [p, floor] = flip(d);
  return `${label}: mean ${f(t.m)}, sd ${f(t.sd)}, t 95% ${f(t.lo)} to ${f(t.hi)} (${t.n - 1} df), p ${f(t.p, 4)}; sign-flip p ${f(p, 4)} (floor ${f(floor, 4)}); 80% power gap at this n ${f(gapAt(t.sd, t.n), 1)}`;
};
const [kind, ...rest] = process.argv.slice(2), v = rest.map(Number), vs = rest.indexOf('vs');
const num = (a) => a.trim() !== '' && Number.isFinite(Number(a));
if (!(kind === 'maps' ? rest.every((a) => a.includes(':') && num(a.slice(a.indexOf(':') + 1))) : rest.every((a, i) => i === vs || num(a)))) {
  console.error(`not a number in: ${rest.join(' ')}\nusage: node daishi-stats.mjs ${USAGE}`);
  process.exit(1);
}
if (kind === 'fisher' && v.length === 4) {
  const [g, l, two] = fisher(...v);
  console.log(`Fisher, ${v[0]} of ${v[0] + v[1]} vs ${v[2]} of ${v[2] + v[3]}: one-sided p ${f(g, 4)} (first larger), ${f(l, 4)} (first smaller); two-sided ${f(two, 4)}`);
} else if (kind === 'binom' && v.length === 2) {
  const [lo, hi] = cp(v[0], v[1]);
  console.log(`${v[0]} of ${v[1]} = ${f(v[0] / v[1], 3)}, Clopper-Pearson 95% ${f(lo, 3)} to ${f(hi, 3)}`);
} else if (kind === 'mcnemar' && v.length === 2) {
  const [b, c] = v, n = b + c, p = n === 0 ? 1 : Math.min(1, 2 * sum(Array.from({ length: Math.min(b, c) + 1 }, (_, i) => Math.exp(lc(n, i) - n * Math.LN2))));
  console.log(`McNemar exact, ${n} discordant pairs (${b} arm only, ${c} control only): two-sided p ${f(p, 4)}`);
} else if (kind === 'paired' && v.length > 1) {
  console.log(paired(`${v.length} paired differences`, v));
} else if (kind === 'maps' && rest.length > 2) {
  const g = new Map();
  for (const a of rest) { const [k, x] = a.split(':'); g.set(k, [...(g.get(k) ?? []), Number(x)]); }
  const m = [...g.values()].map(mean);
  console.log(paired(`${rest.length} pairs on ${m.length} maps, map means ${m.map((x) => f(x)).join(' ')}`, m));
} else if (kind === 'equiv' && v.length > 2) {
  const [m0, ...d] = v, t = tSum(d, 0.9), se = t.sd / Math.sqrt(t.n), p = Math.max(tUpper((t.m + m0) / se, t.n - 1), tUpper((m0 - t.m) / se, t.n - 1));
  console.log(`${t.n} paired differences, margin ${m0}: mean ${f(t.m)}, 90% t ${f(t.lo)} to ${f(t.hi)}, TOST p ${f(p, 4)}: ${p < 0.05 ? 'equivalent within the margin' : 'not shown equivalent; write "undecided"'}`);
} else if (kind === 'welch' && vs > 1 && rest.length - vs > 2) {
  const a = tSum(v.slice(0, vs)), b = tSum(v.slice(vs + 1)), va = a.sd ** 2 / a.n, vb = b.sd ** 2 / b.n, se = Math.sqrt(va + vb);
  const df = (va + vb) ** 2 / (va ** 2 / (a.n - 1) + vb ** 2 / (b.n - 1)), d = a.m - b.m, h = tQ(0.975, df) * se;
  console.log(`Welch, ${a.n} vs ${b.n}: means ${f(a.m)} and ${f(b.m)}, difference ${f(d)}, 95% ${f(d - h)} to ${f(d + h)} (${f(df, 1)} df), p ${f(tTwo(Math.abs(d / se), df), 4)}`);
} else if (kind === 'gap' && (v.length === 2 || (v.length === 3 && v[2] > 0 && v[2] < 1)) && v[1] >= 2) {
  const a = v[2] ?? 0.05;
  console.log(`sd ${v[0]}, ${v[1]} pairs: a two-sided t test at ${a} catches a true gap of ${f(gapAt(v[0], v[1], a), 1)} or more 80% of the time`);
} else if (kind === 'size' && (v.length === 2 || (v.length === 3 && v[2] > 0 && v[2] < 1)) && v[0] > 0 && v[1] > 0) {
  const a = v[2] ?? 0.05;
  let n = 2; while (gapAt(v[0], n, a) > v[1] && n < 5000) n++;
  console.log(`sd ${v[0]}: ${n} pairs catch a true gap of ${v[1]} 80% of the time (two-sided t, ${a})`);
} else {
  console.error(`usage: node daishi-stats.mjs ${USAGE}`);
  process.exit(1);
}
```

## Sources

Listings checked 2026-10-01, the newest again 2026-10-09. Every number here comes from Daishi's docs, runs or simulation, not from these.

[Miller] arXiv:2411.00640 · [Bowyer] arXiv:2503.01747 · [Colas] arXiv:1904.06979 · [Agarwal] arXiv:2108.13264 · [Henderson] arXiv:1709.06560 · [Lakens] Lakens, "Equivalence Tests", SPPS 2017 · [Johari] Johari et al., "Always Valid Inference", Operations Research 2022 · [Waudby-Smith] Waudby-Smith et al., "Time-uniform central limit theory and asymptotic confidence sequences", Ann. Statist. 2024 · [Kohavi] Kohavi et al., "Trustworthy Online Controlled Experiments", 2020 · [Nosek] Nosek et al., "The preregistration revolution", PNAS 2018 · [NIST] NIST AI 800-2 initial public draft, 2026 · [Bean] arXiv:2511.04703 · [ABC] arXiv:2507.02825 · [Kirgis] arXiv:2605.08545 · [Steps] arXiv:2604.09563 · [Singh] arXiv:2504.20879 · [Biderman] arXiv:2405.14782 · [REFORMS] Kapoor et al., Science Advances, 2024 · [Sclar] arXiv:2310.11324 · [Kapoor24] arXiv:2407.01502 · [HAL] arXiv:2510.11977 · [Reliability] arXiv:2602.16666 · [Yao] arXiv:2406.12045 · [MeltingPot] arXiv:2211.13746 · [Needham] arXiv:2505.23836 · [EANOC] arXiv:2606.23583 · [Schoen] arXiv:2509.15541 · [Limited] arXiv:2605.05835 · [Turpin] arXiv:2305.04388 · [Chen] arXiv:2505.05410