# Series and paired experiments

> Served 15 November 2026, when the published rubric was 2.4. A copy that names an older rubric or an earlier day is out of date; fetch it again.

### Series: repeat a run and get an interval

A run answers "who won this draw", not "what does this configuration score":
in the measurements behind the repeated-runs analysis, the spread between
repeats of one fixed configuration ran two to four times the spread between
world seeds, and repeats disagreed about the winner on every seed tested.
Set **Trials** in the builder and the Studio queues a **batch**: N trials of
one configuration, admitted against your plan once, played side by side up
to your plan's runs at once and reported together. Trials per batch: 2, 4, 10
or 50 by plan.

- **Each trial gets its own world** by default, which costs the same as
  repeating one map and supports a claim about the scenario rather than
  about that map. `seed_mode: "pinned"` holds the world and varies only the
  starting layout, for studying one map on purpose. A batch that pins the
  spawn seed as well is refused: every trial would replay the first.
- **A batch is admitted as one run, and its trials play as many.** Every
  trial is in line from the moment the batch is admitted, each on a fresh
  private world, and as many play at once as your plan's runs at once allow.
  As each trial starts, the rest of the batch rejoins the line as of then,
  behind anything queued since (a single run you launch after it, another
  account's run, the other arm of an experiment), so a long batch never
  holds the line for every trial.
- **The report leads with the interval.** Per model: a 95% Student t
  interval over trials (a model's seats in one trial share its world, so
  each trial counts once), the interquartile mean beside the mean (where
  they disagree, the gap is the tail), survival as a Wilson-bounded rate,
  best and worst trial, and a `provisional` flag below three scored
  trials. Where a model held more than one seat in a trial, its row also
  gives the design effect by trial: how much its seats in one trial moved
  together (1 is as good as independent draws), with its seats divided by
  it, what they are worth as independent seats. Nothing is trimmed: an agent that dies in a sixth of its trials
  has a death rate, which is a result, not an outlier to discard. The
  verdict line says plainly when the sample cannot support what a reader
  might take from it (still playing, below the floor, one world only, or
  trials that failed).
- **Spend and tokens sit beside the scores.** Per model: spend per seat
  per trial, spend per index point (the spend over the Fitness Index the
  same seats earned) and per win, and input and output tokens per seat per
  trial; each trial row carries its spend. Tokens are counted by each
  model's own tokenizer, so compare a model with itself on tokens and
  models with each other on spend. A seat the platform did not drive (a
  guest on its own keys) reads "not measured" and enters none of these
  figures, never as a zero. A figure is the provider's own charge where it
  reported one, else priced from the published table; "est." marks a call
  whose model the table has no price for, priced at a fallback rate.
- **The builder prices the whole batch** before launch: agent-turns, the
  inference range, the wall clock for all N trials (one trial's times the
  waves they take at your runs at once: ten trials at Pro's five at once
  take as long as two), and the interval N trials would buy at a reference
  spread, against what a single run buys. That interval is the report's own
  (Student's t on the spread between trials), so two or three trials buy a
  wider one than a single run's reference; four or more bring it under. The
  wall clock follows the slowest seat: a round waits for every seat, and
  each model and reasoning level is timed from published matches where one
  has been measured.

### Paired experiments

A series answers "what does this configuration score"; an **experiment**
answers "does this change matter". An experiment holds two to four
**arms** of one scenario. Each arm is an ordinary series batch of the same
number of trials, and the arms may differ in anything else: the roster
(one model against another, or one model with and without a skill, an
instruction or a reasoning effort), an environment knob, the season length
or the awareness frame. The first arm is the **control**, and every other
arm is compared against it. The report names what each arm changes, read
off the arms' own requests, so an ablation is the same object as a model
comparison.

**The same seeds, trial by trial.** Trial one of every arm plays the same
world seed and the same starting layout, trial two the next pair, and so
on: the seeds are drawn once, when the experiment is created, and no arm
may set its own. `seed_mode: "vary"` (the default) gives each trial a
fresh world, shared by every arm; `"pinned"` holds one world for every
trial and varies only the layout. A scenario that draws its worlds from a
seed pool plays trial i on the pool's world for that trial in every arm,
so it pairs the arms by itself but cannot hold one world: it refuses
`"pinned"`; use Custom World or a saved scenario to study one map. Its
practice pool holds eight worlds, so from trial nine on they come round
again, each time with a fresh layout; use Custom World when every trial
needs a world of its own. Sixteen trials there are eight maps played
twice, not sixteen maps: the comparison's interval, which counts pairs,
holds for those eight maps, and the report reads it a second way that
counts each map once (below), which is what carries to other maps. A scenario with one fixed world (the lab
scenarios) plays it on every trial, so an experiment on it is recorded
`"pinned"` and refuses `"vary"`. On a
shared seed the map and the layout cancel out of the comparison, and what
is left is the thing the arms differ in. The unit of evidence is therefore
the per-trial **difference** between an arm and the control, never the
pooled averages.

**The focal seats.** An experiment created from 20 October 2026 on declares
which seats its metric is read on. When every arm's change is a seat's (a
model, a skill, an instruction, a reasoning level or another setting of one
or more seats), those seats are **focal** and the metric is read on them
alone; the rest of the roster, the **background** seats, play the same
configuration in every arm, and the report reads them the same way and
prints their mean beside the focal mean, compared by nothing: it shows what
the change did around the focal seats, and no verdict rests on it. A change that reaches the whole match (an environment setting, the
season, the frame, or an A/A check) makes every seat focal and none
background. The report says which seats it read in words (`focal`, also on
the estimate, so the builder shows it before you queue), each arm carries
`background` beside `score`, and an experiment created before this date
reads every seat as it always did.

**What the report leads with.** The stopping rule's decision on that
difference, with its interval: the arm came out ahead of the control on
the metric (on the Fitness Index, it scored higher), the control came out
ahead, or **undecided**, which is a real outcome and is reported as such
rather than rounded to a winner. Below it, per arm: what
differs from the control, the scored trials, the mean with its 95%
interval, the cost, and the share of seats that said in their debrief
that they believed the match was an evaluation of them (see
[The awareness frame](/docs/studio/runs#the-awareness-frame)). **Spend and tokens** follows:
per arm the spend per finished trial, with its difference from the
control's, and the spend per index point, then per model in the arm the
input and output tokens and the spend per seat per trial, read the same
way as a batch's. Then the pairs: per trial,
the seeds, each arm's value and the difference. A trial that failed or was
cancelled in any arm, in which a seat gave up before the match ended (on
failed calls, or on turns its model declined), that the spend cap stopped
short of its season, or that a new server release picked up mid-match
(see "A deploy does not end your run" above), leaves its whole pair out,
and the report says how many were left out; a half-played match
never enters a difference. So does a trial played under another version
tuple than the study's first trial. Since 20 October 2026 every trial
records what it played under: the reference harness with its prompt
version and hash, the engine and its event schema, the house bots, the
build, and each screen and record written at play time. The report holds
every trial to the first one's tuple and counts a trial played under
another as lost, since nothing pools across versions: the report's
`versions` names the tuple it read and how many trials it left out for it.
Deploys are held while a study plays, so a study rarely spans two. A pair
is also left out when its two trials played different seeds, which only an
experiment created from 19 to 24 October 2026 can hold: a trial that
started in the seconds before its experiment's record was written drew its
own layout, a fault fixed on 24 October 2026. The pair's row on the card
names each arm's seeds without naming either arm as the one that strayed,
the card's Worlds line names those trials, and the pair counts against the
trials the design may lose (below) as a lost trial does, though the card
and the report say it was left out: both of its plays played whole. Daishi
has counted such pairs against the declared read only since that day,
after every study that can hold one was declared, and the card says so. A
fixed rule's stop recorded on such pairs is read again
without them, and the verdict quotes what the record read. When the trials left out fall unevenly across
the arms, the verdict says how many each arm lost: an arm that loses more
can look better on the pairs that remain, because the trials it lost may be
the ones it was playing badly. An arm whose comparison has already stopped
(next section) is not needed after its last pair, and the control only as
long as some comparison still is, so the trials no longer played leave
nothing out. Trials
are compared on the **Fitness Index** by default; `metric` can instead be
`rank` (place in the match, where lower is better, so the report flips the
sign and a positive difference always means the arm placed higher) or
`survived` (the share of an arm's seats neither dead nor dormant at the
end).

**The declared read.** An experiment created from 19 October 2026 on
declares, before its first trial, how many trials it may lose:
`loss_tolerance` (Studio: **Trials that may be lost**), from none up to the
trials less the three pairs a verdict needs, a tenth of the trials rounded
down when you set nothing. A trial is lost when an arm's trial failed or
was cancelled, lost a seat to an error, stopped short of its season at its
spend cap, was picked up mid-match by a new server release, played under
another version tuple than the first trial, or measured nothing on the
metric. A pair whose two plays started from different layouts is left out
(above) and counts against the tolerance the same way. A comparison that
loses more than that renders no
verdict at its declared read: the report reads **incomplete**, keeps the
interval on the pairs played as a description of them, and the experiment
ends `incomplete`. No lost trial is ever replaced and no trial is added
to make up for one: the design plays the trials it declared and reads what
it has, so a result can never rest on trials chosen after the losses were
seen. Within the tolerance the rule reads the pairs present, as it always
did, and the report counts the losses per arm before any number. The
study can also declare a spend cap for the whole of it,
`max_total_spend_usd` (Studio: **Spend cap for the whole study**): once the
trials played so far have spent it, nothing still in line starts, what is
left is cancelled under the cap rather than as your doing, a trial already
playing runs to its own per-run cap, and the experiment ends `spend_cap`;
the trials it stopped count as lost. The rule sentence on the report and
the card prints both. Beside every comparison that lost a pair, the report
prints its **tipping point**: the per-trial difference every lost pair
would have had to carry for the reading to change, found by adding that
many pairs at one value and reading the rule's interval again (for a
verdict, the value with the sign against it that first puts zero back in
the interval; for an undecided reading, the smallest value either way that
first takes zero out), beside the widest difference the pairs played did
carry. Each prints as a size with the side it leans toward ("3.53 toward
play A"), or on output per turn as the arm's percent change against the
control, with its sign. A verdict that one lost pair at a difference smaller than any seen
would undo is a verdict to read with care; one that no difference within
the metric's range would undo is not. An experiment created before the
declared read carries no tolerance and reads as it did.

**Run health.** Every driven seat's meter keeps two rate limits apart.
What the **world** refused of the seat's own actions under its seat rules,
counted from the match's own events: actions over a turn-based match's
per-tick budget, actions under a wall-clock match's pace floor, and actions
the world could not take (carrying too much, nothing here to gather). And
what the **provider** made the harness retry: the transient failures each
model call rode out, as the harness has written them on its usage and error
lines since version 0.6.7, by cause (an HTTP status such as 429 or 529, a
timeout, a dropped connection) with the time spent waiting. The first is
the model's own doing; the second is not, and it is the one that can reach
a comparison: the arms of an experiment share one key, so a provider that
throttles one arm's calls reaches the other arm's pairs through the shared
limit. The report sums each arm's health over its finished trials and reads
the **interference check**, provider retries per 100 model calls arm by
arm: **alike** when the highest rate is within 2 per 100 calls of the
lowest or within twice it, **uneven** otherwise, and unread until every arm
reports over 20 calls or more. Uneven is a reading, never a verdict: it
says the arms did not see the same provider, and the verdict paragraph
says so before any number. The Seats panel of a run, the developer API
(`GET /api/v1/runs/:id/seats`) and the archive's per-seat `observed` block
carry the same figures per seat.

**What the report cannot support.** Beside the verdict, which says its
limitations first in prose, the report carries them as a typed list,
`limitations`, one entry per kind in a fixed order: what the metric measures
and does not (`construct`); who stood behind the number (`access`: the
account that designed, funded and reads the study, the platform that built
the world, the rubric and the scorer, and whether any outside party reviewed
either; a study run by Daishi's own research account says so, an interested
party on both sides); how much was measured (`sample`, with the smallest gap
the design finds); what was left out (`selection`, the trials lost, by arm);
what it compares with (`version`); how it was measured (`measurement`: the
debrief and the scan as self-reports, guest seats not metered, the
interference reading); and where it holds (`scope`). The study's card
prints the list under its numbers, the Studio panel under the verdict, and a
match's report, export and page carry the same kinds for the match.

**Beside the rule's reading.** Two more readings, which decide nothing:
the verdict is the declared rule's. Under the fixed rule, an exact
sign-flip test of the paired differences, which assumes nothing about how
they spread. Five pairs that all go one way give p 0.0625, the smallest
five allow, so no exact test on five pairs or fewer can reach 0.05: a
verdict there rests on the t test's assumption that the differences
spread about normally, and the report says so. The estimate under Compare
configurations says the same before launch: the lowest p the exact test
can reach on the planned pairs (or maps), against the level each comparison
holds, and how many pairs it takes to reach that level, or how many maps
where the trials replay maps, which more trials do not add. Where the trials replayed
maps, the rule's interval holds for the maps played, and the report adds
the interval that counts each map once (cluster-robust Student t on the
number of maps less one, from three maps, fixed rule only), with the
exact test run over the maps. When that interval spans zero under a
verdict, the report and the study card say the verdict holds for those
maps and is not shown beyond them. It is never read as narrower than the
interval on the pairs: the two share a centre, and a result cannot be
surer for other maps than for the maps played. So where it comes out the
narrower, the card quotes the interval on the pairs, and its note on maps,
where it has one, says why; where it also clears zero, the report says it
is the narrower; and a reading the pairs leave undecided stays undecided.
This holds for a secondary outcome's reading as for the metric's. The anytime rule
gets neither that interval nor the exact test over the maps: it looks again
and again, and an interval or a test read at a stop the data chose would
not hold its level.

### Building an experiment

**Every arm must differ.** The server compares what each arm will play
with the control's: every seat as validated, defaults filled in (a seat's
name included, since the other agents see it), the environment as the
match resolves it (the scenario's pacing filled in, a 1x resource
multiplier dropped, the house-bot lineup expanded to its policies), the
season and the frame; not the label, the run name or the spend cap. An arm
that plays exactly what the control plays is refused
(`experiment_same_arms`), and `differs_from_control` lists what each arm
plays differently. To run identical arms on purpose, as a measure of the
noise between them, set `aa_check: true` (the Studio's **Identical arms on
purpose** box); an A/A check whose arms differ is refused, and its verdict
says any difference is noise. Its card's sentence gives, before any verdict,
how far the identical setups differed (the standard deviation of the paired
differences, once five pairs that played one layout are in; not on output
per turn, which compares on a log scale). A re-run of a published study whose arms, as
recorded, play one configuration is refused the same way, whether the
treated arm differed only in the owner's private skills or instructions
(which a re-run fields clean) or in nothing that reaches the match; start
a variant from the study's design instead. A setting the server does not
read (a misspelled `env` field, `env.fly`, an unknown seat, arm, run or
experiment field) is refused by name rather than dropped, over the API and
the Studio MCP alike, so a body never plays as something other than what
it says.

**Secondary outcomes.** `secondary_metrics` adds up to three outcomes
reported beside the one that decides, declared when the experiment is
created: any of the outcomes above except the metric itself. Each is read on the
same pairs with the same rule at the same level, and none stops anything:
when the deciding comparison stops, its secondaries are read on the pairs
it stopped on, and the trials still to play are cancelled as usual. Each
secondary interval holds its level on its own, not jointly with the
others, so with two arms and three secondaries the chance that at least
one excludes zero when nothing differs is up to about 15%. Read a secondary result as
something to confirm in an experiment that declares it as its metric.
Declaring output per turn as a secondary holds the arms to the same
models in the same seats, as it does as the metric.

**Queue slots and plan limits.** The arms run side by side, each admitted
against one run slot like any batch, so an experiment needs a plan that runs
as many at once as it has arms (the **Runs at once** line under
[Plans](/docs/studio/plans#plans)): Free runs one at a time and cannot hold an experiment,
Starter holds two arms, Pro and Lab up to four. Once admitted, the trials of
every arm play side by side up to your runs at once, and the arms take turns
in line so the pairs complete in index order. Trials per arm are bounded
by the plan's series ceiling (4 on Starter, 10 on Pro, 50 on Lab), so on
Starter and Pro an experiment plays every trial and looks once, and the
anytime rule below, which needs more than ten trials, takes Lab. Each arm is admitted against your plan
exactly as a batch would be, and the estimate prices the whole design
before launch: the cost summed over the arms, the wall clock with every
arm's trials sharing your runs at once, and the smallest gap the design can
detect. Cancelling an
experiment cancels every unfinished trial of every arm; see
[Stopping one yourself](/docs/studio/outcomes#stopping-rules) for what the report then reads.

**Creating one in the Studio.** The **Lab tools** card on the Overview and
**Build an experiment** on the Experiments view both open the builder; it is
the **Compare configurations** card under **New run**, below Queue run.
Experiments start on Starter: an experiment holds one of your runs at once
per arm and plays at least 3 trials, so it compares up to 2 arms on Starter
and up to 4 on Pro and Lab, and stopping early (the anytime rule, which needs
more than ten trials) is Lab's. The card says what your plan allows before
you build anything. Under **New run**, set up the control the
way you would any world run: the scenario, the roster, the environment,
the awareness frame. Set **Trials** under **Repeat this run** (at least 3)
and how the trials differ. Under **Compare configurations**, give it a
label and press **Add as arm**. Then change the one thing you are testing
(a model, a skill, the instructions, the frame, an environment setting)
and add that too, up to four arms; **Remove** takes one back. Each arm's
label, up to 32 characters, names it on the report, so no two may match;
the experiment takes its name from the builder's **Run name** (left blank,
it is named after the scenario and its number of arms). Choose the
**Stopping rule** (automatic picks anytime above ten trials and fixed
otherwise), what to **Compare on**, any outcome to **Also report**
beside it, and paste your **Declared design** if you wrote one (below).
The card prices the whole design as
you go and says plainly when your plan cannot hold it. Above **Queue
experiment**, one sentence gives the gap the design would catch four
times in five, says a smaller real difference will most likely read
undecided, and says what 10, 20 or 30 trials per arm would narrow it to,
where your plan allows them. **Queue experiment** starts it and opens the
**Live** page filtered to its worlds; its report fills in under **Experiments**,
which lists every experiment, shows each report as it grows and cancels
one. Once a trial has played, an experiment's panel opens on **How this
reads on your profile**: the study's card as a visitor would get it, built
over every trial that played as if it were published, above your declared
result, the design and **Publish study**, with the numbers after them. An
arm cannot set a world seed or seat a guest,
and every arm plays the control's scenario; the card says so when you try.
A target half-width and a per-experiment callback address are set through
the API. Three of the [templates](/docs/studio/runs#start-from-a-template) are experiments:
they set the trials, the metric and what to also report, and their
**Build the arms** button, at the top of the card, builds the arms from
your roster in one step. They also start the **Declared design** box on
their own outline (the question, the setup and arms, the result they read
and what they cannot show), with your prediction and what would change
your view left as marked placeholders. **Queue experiment** refuses while
a placeholder is left: fill both in, or clear the box to queue without a
design. Switching templates replaces only an empty box or an earlier
template's untouched outline, never your own words.

**Creating one by script or agent.** The developer API takes the same
thing (`POST /api/v1/experiments`, with `POST /api/v1/experiments/estimate`
to price one first), and so does the Studio MCP (`create_experiment`,
`estimate_experiment`). A body is a name, the trials per arm and two to
four run bodies, the same shape the run builder posts, each with an
optional `label`:

```json
{
  "name": "Does the notice change play?",
  "trials": 20,
  "arms": [
    { "label": "canonical", "scenario_id": "daishi:custom-base-v1",
      "roster": [{ "name": "seat-a", "model": "<provider>/<model>" }] },
    { "label": "monitored", "scenario_id": "daishi:custom-base-v1",
      "roster": [{ "name": "seat-a", "model": "<provider>/<model>" }],
      "frame": "monitored" }
  ]
}
```

A body of `replicates` (a published study's experiment id) and no design
fields queues that study again whole on your keys, as
[Re-running a study](/docs/studio/sharing#privacy-sharing-and-publishing) describes.

Optional fields: `seed_mode`, `metric` and `secondary_metrics` (above),
`stopping` and `target_half_width` (next section), `design` (below), and `callback_url`, an address that receives the signed
end-of-experiment notice described under
[When a run ends](/docs/studio/automation#when-a-run-ends), held to the same rules as the
account's webhook address there. The [developer docs](/docs/developer)
carry the full reference, the error codes and every field of the report.

**Your design and your declared result.** An experiment can carry two
things in your own words. The **design** is the document you write before
the first trial: what you predict, which result you will read and how, and
what would change your view. Paste it under **Declared design** when you
build the experiment, or attach it from the experiment's panel before the
first trial starts (`design` on the creation body, or
`POST /api/v1/experiments/:id/design`). Daishi keeps it as sent, with the
SHA-256 of its bytes and the time it arrived, and once a trial starts it
cannot be replaced; a document you replace before then is kept as an
earlier version, with its own digest and proofs, and the design page lists
it. The moment it arrives, its SHA-256 (never the text)
also goes to the two witnesses outside Daishi that stamp the digest chain:
an RFC 3161 timestamp authority, whose signed reply states the time by its
own clock, and the OpenTimestamps calendars, whose proof is usually finished
in a Bitcoin block within a few hours. A design sent with a new experiment is
normally witnessed before its first trial can start, since creating the
experiment waits a few seconds for the witnesses. Each trial's record
also carries the design's digest and arrival time as they stood when the
trial launched (`experiment` in its archive's `config.run`), so the record
on the chain names the design its launch read. Once a trial is
published, the study's card on your profile links to it, and its page
lets anyone download the exact bytes and check the digest, and download
each witness's proof and check it with the standard tools. Daishi checks
when the document arrived and what it hashes to, never what it says. A design attached after the first trial is
taken only when the experiment's name, fixed at creation, already carries
the first 32 or more hex digits of the document's SHA-256 (12 on
experiments created before 19 October 2026, which keep the rule they were
made under); its page then says it arrived late and how loosely a prefix
that length commits. The **note** is your declared result and whatever
you add later: dated entries, never edited or removed, so a reader sees
what you said first and what you added once the results were in. Write
them under **Your note** in the experiment's panel, or with
`POST /api/v1/experiments/:id/note`, at any time, a finished experiment
included. The study's card shows the first entry first, marked as yours,
above the sentence Daishi computes from the declared metric and rule, with
every entry and its date under it; the two are never merged.

**What the creation commits to.** The moment an experiment is created,
Daishi writes a creation record (its id, the time, its name, whether you
declared it a pilot, and the SHA-256 of the design it was sent with) and
hands the record's SHA-256 to the same two witnesses. Once a trial is
published, the record is at `/u/<handle>/studies/<id>/created.json`, its
proofs download from the study's card, and the card prints the
authority's signed time beside Daishi's own. Tick **A pilot** when you
build the experiment to declare a dry run whose result is not a finding:
it is said then and never after, and the card prints it.

**Where the seeds come from.** Nothing is drawn when you create an
experiment: its world and layout seeds come from drand, the League of
Entropy's public randomness beacon, read when the first trial is about to
start. Daishi takes the first round published after the design's witnessed
time (the authority's signed time for the design's digest; the creation
record's without a design; its own clock where no authority answered
within a minute) and derives each trial's seeds from that round's
randomness, the experiment's id and the trial number, the same way for
every arm. Until then the pairs table shows no seeds and the panel says so.
The card prints the round beside the design, and the seeds record is
public with the study at `/u/<handle>/studies/<id>/seeds-record.json`. If the
beacon does not answer within ten minutes of the first trial's turn, the
server draws the seeds itself and the record says so.