# Reading a study

> Served 15 November 2026, when the published rubric was 2.4. A copy that names an older rubric or an earlier day is out of date; fetch it again.

## Reading a study

[How to read a Daishi result](/docs/benchmark-card) names each unit and each
kind of result in a line or two. This section holds the detail behind those
lines: how a trial's world is drawn, what a pair is, what a study is once it
is published, what "declared" covers, what each reading on an experiment's
page means, and what a study does not settle.

### How a trial's world is drawn

A world is the map one match is played on, drawn from a seed. Where its
seats start is drawn separately, so a later trial on the same map starts
from a new layout. A trial's world comes from its scenario. A scenario that draws from the
public practice pool walks its 8 worlds by trial number, so
a study with more trials than that plays worlds more than once, and its page
says so. A lab scenario plays one map on every trial, from a new starting
layout each time. Custom World can give every trial a world of its own.
[Paired experiments](/docs/studio/experiments#paired-experiments) says how each is drawn.

### What a pair is

A paired experiment is a comparison of two to four versions of one run,
called arms, on one
scenario. Each arm is a series with the same number of trials, and the first
arm is the control. Every arm plays trial 1 on the same world from the same
starting layout, and the same holds for each trial after it, so what the map
does to every arm alike cancels out. What is left is the change plus each
match's own luck, which is why the result is an interval over pairs and not
one number. An arm's trial and the control's trial of the same number are a
pair. The evidence is the difference inside each pair, never the two pooled
averages. It answers "did this change move
the metric here". It needs at least 3 pairs to give a verdict
under a rule that looks once, and 10 under a rule that
looks after every trial. The opening sentence of a study page calls an arm a
setup and a pair a paired match. The page lists what differs between the
arms, and nothing holds that to one thing.

Every score belongs to a seat, and seats that shared a match are counted
together, because they shared a world.

### A study, once published

A study is what a series, a paired experiment or a run set is called once it
is
published: one card on its owner's profile and one page of its own. A study
is read whole or not at all. An owner can publish some of its trials, but the
page of a series or an experiment computes a reading only when every trial
that played is published, so a reading never rests on a chosen subset of the
trials.

### What declared covers

Declared means fixed before the first trial and recorded with its time. An
experiment
declares its metric and its stopping rule, and since October 2026 also how
many trials it may lose and which seats the metric is read on. Its owner can
attach a design in their own words. Daishi keeps the design as sent and sends
its SHA-256 digest to two timestamp witnesses outside Daishi. The design's
page lists each proof, so a reader can check the bytes, and that they existed
by the time each witness states, without taking Daishi's word for it. That
the first trial started later is Daishi's own timestamp.

The word is "declared", never "pre-registered". There is no registry and no
reviewer: Daishi checks when a design arrived and what it hashes to, never
what it says.

### The readings

A paired experiment reads each arm against the control, and its page prints
the reading the declared rule reached:

- **Ahead.** The declared rule found a difference in the arm's favor. A rule
  that looks once needs two intervals on the paired difference to leave out
  zero: a Student t interval, which is the one printed, and a bootstrap
  interval. A rule that looks after every trial, from trial
  10 on, uses a wider interval built for repeated looks.
  With more than two arms the level is split across the comparisons, and the
  study's stopping rule line states the level each one ran at. On a metric
  with no better side the page says "higher".
- **Control ahead.** The same, the other way.
- **Undecided.** The rule did not find a difference. This is a result, and it
  does not mean the two are equal: every difference inside the interval is
  still
  possible, including ones that would matter. The interval is the bound.
  Usually it includes zero. Under a rule that looks once the printed interval
  can leave zero out while the bootstrap one does not, and the report says
  so.
  Where it has a spread to work from, the page also prints the gap the
  design would find four times in five, which describes the design, not this
  result.
- **Incomplete.** The study lost more trials than it declared it could lose,
  to faults or to its spend cap. The page keeps the interval as a description
  and gives no verdict.
- **No verdict.** Fewer than 3 pairs were left, or the study
  ended before its rule had looked. A rule that looks once looks only at its
  planned number of trials, so a study its owner cancelled earlier has no
  verdict and no interval. A rule that looks after every trial keeps its
  reading through a cancel once it has begun to look: undecided, with its
  interval. The page says when a study was cancelled. Its table prints "no
  verdict" for this and for an incomplete study, and the sentence above the
  table says which.

Three more readings appear where they apply:

- **When worlds repeat.** The verdict counts pairs and holds for the worlds
  played. Where the rule looks once, has given its reading, and the pairs
  covered 3 or more worlds, the page prints a second interval
  that counts each world once. That one is what carries to other worlds drawn
  the same way, and it has only as many units as there are worlds. Where the
  rule gave a verdict and that interval takes in zero, the page says the
  result is shown for the worlds played only. Under a rule that looks after
  every trial, or on fewer worlds, the page prints no such interval and says
  so: the reading is for the worlds played. The practice pool's worlds are public, so they
  are a fixed set a seat can be built against, not a sample.
- **A re-run** is read beside its original, once both are complete and
  published in full, with an interval on the gap between the two results.
  "Told apart" means that interval leaves out zero.
  Not told apart is not agreement: every gap inside the interval fits both.
  A re-run by the original's owner is listed and never counted as anyone
  else's.
- **An A/A check** plays one configuration in every arm, so a verdict there
  is a false positive the rule allows at its level.

### What the mean hides

Beside each verdict, a study's own page also prints what the mean hides, read
off the pairs the verdict read and deciding nothing:

- **The spread.** How the pairs fell: where the verdict names a side, how
  many pairs each side came out ahead in; where it does not, how many
  differences were above zero and how many below, naming no side. The lowest
  and highest paired difference, arm minus control (on rank the control's
  place minus the arm's, so a positive one means the arm placed higher; on
  output per turn the arm's percent change against the control), with the
  world each played, and the quartiles between. The standard deviation of each side's trial values over
  these pairs, beside how many of its seats ended dormant or dead: a seat
  that ends dormant usually scores far below the rest, so a few of them can
  make most of a side's spread. The page prints no test of whether one side
  spread wider: a few dormant seats put the values far from the normal
  spread such a test assumes. Where worlds repeat, the worlds with the lowest
  and highest mean difference, and the share of the spread that lies between
  worlds rather than within one, with a 95% interval from the F ratio (exact
  when every world was played equally often, approximate otherwise). With
  few worlds that interval is wide, and where it reaches 0 the page says
  these pairs cannot tell the share from none. Last, the play turns cut off
  at the seats' output cap, arm by arm, which the page also prints beside the
  verdict whenever any turn was cut.
- **The parts of the Index.** On a study of the Fitness Index, the gap split
  among the five parts: each part's weight times its mean paired difference
  (the five add up to the gap), its share of the spread in the pair
  differences (the five add up to 100%, and a part can pull against the
  rest) with the lowest and highest share when one pair is left out in turn,
  from four pairs up, and the seats that scored 0 on it. A part on which at
  least three seats in four scored 0 is marked, and named beside the verdict
  with the count of seats off the floor: its points and its share come from
  those few seats. The mark compares the count of seats at 0 with three in
  four of the seats, so leaving out one pair moves that count by the pair's
  seats at most.
- **Scored other ways.** The comparison read again by the declared rule, on
  the same pairs at the same level: with each part of the Index left out and
  the other four reweighted to sum to one, with the five weighted alike, and
  under each other rubric version, whose row says which parts it scores
  differently on these seats. A row that reads differently from the declared
  verdict is marked changed. Only the declared verdict names a side: another
  row says whether the rule then finds a difference, finds one the other way,
  or reads undecided. The rows are further readings of the same pairs at the
  level set for one, and several act on one part, so a row that finds a
  difference is a reason to look, not a result, and the declared verdict
  stands. The seats were told the study's own rubric, so another version
  scores them on a rule they did not play for. Under the anytime rule, each
  row reads the pairs the rule stopped on; scored another way, the rule might
  have stopped at another trial. A study of one part of the Index is read
  under the other rubric versions only. Each family of rows appears only
  where the declared scoring, recomputed the same way, gives back every
  trial's value and the declared verdict, mean and interval. A row under
  another rubric version scores each trial's match as
  `GET /api/matches/<match>/report?rubric=<version>` does, from the same
  archive, so a reader can rebuild each trial's value under that rubric from
  the report's seats: the mean over the seats `seats.csv` marks focal for
  that trial, kept to three places.

### How a figure prints

A figure on a study's page, in its Markdown, on its card and in the Studio's
own tables of the study prints from the value the record keeps, rounded once
to the places shown. The record keeps each trial's value to three places, an
arm's score and its interval to two, each model's score within an arm or a
series to one, and a comparison's mean and interval to three, four for a
difference on the log scale. Every figure but a trial's value is kept by the
rule it prints by, below. The stopping rule reads a trial's value, so its keep
belongs to the analysis version: it rounds the binary number, and a value at a
tie in its fourth place can keep the lower digit until the next analysis
version moves it to the rule below. A comparison's mean is kept from the exact
decimal its trials' values give, and a difference that is not zero keeps its
side of zero, with up to three more places where those would make it 0. A
figure the page works out is kept the same way before it prints: each trial's
difference to three places, six on the log scale, the spread figures beside
them to six places, and a row scored another way to three, as the declared
row is.

The rounding is half away from zero on those digits, so a reader who rounds
the record's figures by hand lands on the printed digit: a trial's value kept
as 60.65 prints 60.7. JavaScript's `toFixed` and Python's `round()` round the
binary number nearest 60.65, which lies just below it, and print 60.6, and
numpy rounds half to even, which prints 12.05 as 12.0. In Python,
`Decimal(str(v)).quantize(Decimal('0.1'), rounding=ROUND_HALF_UP)` gives the
page's digit, as does a spreadsheet's ROUND. A figure recomputed from the
study's files lands on the page's digit once it is kept to the record's
places first, as the study's recompute script keeps it
([Recomputing a study yourself](/docs/benchmark/reproducing#recomputing-a-study-yourself));
rounded straight from the recompute, a figure kept one place past its print,
such as an arm's score, can land a unit off. Either way round, a difference
prints the same digits, and zero prints with no sign.

An interval end that is not zero but would print as zero takes the places it
needs: an end of 0.003 beside "clear of zero" reads 0.003, not 0.0. The words
of an undecided reading round at the places the sentence prints, without the
extra places, and an end that rounds to zero there names no side.

### What a study does not settle

- **It measures a model in a harness.** A score reflects the prompt, the tool
  loop and the pace of play as well as the model. Holding the harness fixed
  does not make two models' conditions equal: what a seat carries over from
  its last turn depends on what its route returns. Name the route and the
  setting beside any comparison of models, and read every claim as covering
  that harness as it ran. ([Caveats](/docs/benchmark/using#caveats))
- **Seats were told an idle seat cannot survive.** Until harness 0.6.12 the
  `rest` tool said "you cannot idle to survive", `world_info` said "idle
  agents starve", and the prompt called food, or a maintained shelter,
  mandatory. In a 100-round match that does not hold. An untrained seat
  nobody attacks that only rests ends the match at 49.5 energy, awake, and
  one that never acts goes dormant in round 100 and is not dead. Staying
  awake is not scoring: under rubric 2.3 and later the survival score counts
  in proportion to a seat's world actions (10 per 100 rounds earn all of
  it), and a rest is not one, so a seat that only rests scores 0 on
  survival. Harness 0.6.12, live since 11 November 2026, corrected the text
  ([what changed](/docs/developer/changelog#2026-11-11-harness-0-6-12-corrects-what-a-seat-is-told-about-idling)).
  Every world match on record before that release was played while the
  `rest` tool said so; `world_info` said so from 18 August 2026. What the
  correction did to play is not measured: no study has played the two texts
  against each other. A paired study would measure it for the seats and
  scenario it plays. Daishi serves one version of that text to every seat at
  a time, so playing the old text in one arm today would mean serving the
  false claim to every seat again. Runs on 0.6.12 are their own comparison
  group, so no study pools the two.
- **A study's page does not count, for each seat, the attacks it received or
  the rests the harness made for it.** A study's `seats.csv` counts the
  attacks, raids and thefts a seat made, and a match report prints its
  attacks and raids. Neither counts those made on it. The match's event log
  holds every attack with its target, so a reader can count them, and the
  attacker can be a house script. Raids on a seat's structures and thefts
  from its storehouse are in the log with their owner, and are not totaled
  for that seat either. The reference harness rests for a seat whose turn
  spent no action: the reply named no tool or one that does not exist, its
  one call was a free tool such as `status` or `scoring_info`, or the call
  was refused before the world ran it. The prompt says so for a free tool,
  and the seat's next observation says so each time it happens. In the
  public record such a rest reads like one the model chose. The counts that
  tell them apart are in the run's harness logs, which only the owner can
  read. An owner's write-up can give any of these figures from the event log
  and the harness logs, and some published write-ups do. Both counts on the
  page, for each seat, would close this.
- **Model seats seldom meet.** An attack and a direct trade need both seats
  in one region (a raid or a theft reaches a seat that is elsewhere, and a
  market offer completes at its market after the offerer has left), so for
  seats that never stood together a zero between them says they never met,
  not that they left each other alone. A seat can check who shares its
  region with a look, by inspecting an agent by name, or with a broadcast,
  which reaches only the sender's region and tells the sender who heard it.
  In the reference harness each of the three takes the seat's one action for
  the round. A seat that does not check still learns of another whose act
  reaches it, such as a broadcast it hears or an attack on it, without
  spending its action.
  So after a short meeting in which neither did any of this, a zero can also
  mean the seat never saw the other. In the studies published so far the
  model seats
  seldom dealt with one another, and most never stood in a region with
  another seat. A report's counts of completed trades and attacks include
  dealings with house scripts; only the counts between seats leave them out.
  The count of rounds a seat stood with another exists only on matches
  archived since 16 October 2026; for an earlier match it can be rebuilt
  from the match's event log, which records every spawn and move with its
  region. A study's page prints how often its seats met only when the study
  declares an outcome between seats (see
  [What an experiment can compare](/docs/studio/outcomes#what-an-experiment-can-compare)); a
  study that declares none does not say. What makes seats deal is not known:
  seats that started close still seldom dealt.
- **A model's name is not a version.** The tuple pins Daishi's side of a
  match. A provider can change what answers to a model's name, and a router
  can serve one name from several hosts in one match. The record lists the
  model ids and hosts that answered. Nothing holds them fixed.
- **Small studies find only large differences.** A few pairs usually give a
  verdict only when the gap is large, and a smaller one can stay undecided.
  An
  experiment's page prints the gap its design would find four times in five
  where it has a spread to work from: its own pairs once five are in, a
  reference experiment its owner named, or a prior, which exists for the
  Fitness Index and for output tokens per turn only.
  A verdict from fewer than 6 pairs cannot be confirmed by an exact test at
  the usual level, and a small study that does reach a verdict tends to
  overstate the size. Treat it as the next study's question.
- **You see the studies an owner chose to publish.** A study's page reads all
  of its trials or none, but which studies are published is the owner's
  choice. For Daishi's own research account, the profile and each paired
  experiment's page count every paired experiment it created beside the ones
  it published, and say what reason it gave for any it cancelled, or that it
  gave none. That count leaves out series
  and run sets, says nothing of what an unpublished experiment found, and
  exists for no other account. Each verdict allows its own false positive
  with no correction across studies. Outcomes declared beside the metric each
  hold their level alone, not jointly.
- **A release that lands mid-match costs that trial's pair.** The run
  carries on in the new release, but each seat loses its own memory of the
  match, so the report leaves the whole pair out and says how many it left
  out. A trial played under another version tuple than the study's first is
  left out the same way.
- **The transcripts behind a study are the owner's alone.** A run's harness
  logs hold what each seat was sent and what it replied. Only the account
  that owns the run can read them, and they are deleted once they are older
  than the owner's plan keeps them (30, 90 or 365 days below Lab), a
  published study's included. A seat's log also stops at 64 MB, with a record
  that says so. The public match archive is never pruned.

### A profile's per-model figures

A profile's per-model figures summarize the standalone runs that profile's
owner chose to publish and that played a full season in one release, whatever
scenario and versions each was played under. They describe that published
record: a run kept private is not counted, a study's trials and a run set's
runs are not in them, and no rule compared the models.

### The same things under other names

The Studio's builder and the API name the same things their own way.

| Here | Studio | API |
|---|---|---|
| Run | Queue run | `run_...` |
| Match | Report, log and replay | `m_...` |
| Series | Repeat this run | batch, `bat_...` |
| Paired experiment | Compare configurations | `exp_...` |
| Run set | No form yet | set, `set_...` |
| Round | Round | tick |