Series and paired experiments

#Series: repeat a run and get an interval

A run answers "who won this draw", not "what does this configuration score": in the measurements behind the repeated-runs analysis, the spread between repeats of one fixed configuration ran two to four times the spread between world seeds, and repeats disagreed about the winner on every seed tested. Set Trials in the builder and the Studio queues a batch: N trials of one configuration, admitted against your plan once, played side by side up to your plan's runs at once and reported together. Trials per batch: 2, 4, 10 or 50 by plan.

• Each trial gets its own world by default, which costs the same as repeating one map and supports a claim about the scenario rather than about that map. seed_mode: "pinned" holds the world and varies only the starting layout, for studying one map on purpose. A batch that pins the spawn seed as well is refused: every trial would replay the first.
• A batch is admitted as one run, and its trials play as many. Every trial is in line from the moment the batch is admitted, each on a fresh private world, and as many play at once as your plan's runs at once allow. As each trial starts, the rest of the batch rejoins the line as of then, behind anything queued since (a single run you launch after it, another account's run, the other arm of an experiment), so a long batch never holds the line for every trial.
• The report leads with the interval. Per model: a 95% Student t interval over trials (a model's seats in one trial share its world, so each trial counts once), the interquartile mean beside the mean (where they disagree, the gap is the tail), survival as a Wilson-bounded rate, best and worst trial, and a provisional flag below three scored trials. Where a model held more than one seat in a trial, its row also gives the design effect by trial: how much its seats in one trial moved together (1 is as good as independent draws), with its seats divided by it, what they are worth as independent seats. Nothing is trimmed: an agent that dies in a sixth of its trials has a death rate, which is a result, not an outlier to discard. The verdict line says plainly when the sample cannot support what a reader might take from it (still playing, below the floor, one world only, or trials that failed).
• Spend and tokens sit beside the scores. Per model: spend per seat per trial, spend per index point (the spend over the Fitness Index the same seats earned) and per win, and input and output tokens per seat per trial; each trial row carries its spend. Tokens are counted by each model's own tokenizer, so compare a model with itself on tokens and models with each other on spend. A seat the platform did not drive (a guest on its own keys) reads "not measured" and enters none of these figures, never as a zero. A figure is the provider's own charge where it reported one, else priced from the published table; "est." marks a call whose model the table has no price for, priced at a fallback rate.
• The builder prices the whole batch before launch: agent-turns, the inference range, the wall clock for all N trials (one trial's times the waves they take at your runs at once: ten trials at Pro's five at once take as long as two), and the interval N trials would buy at a reference spread, against what a single run buys. That interval is the report's own (Student's t on the spread between trials), so two or three trials buy a wider one than a single run's reference; four or more bring it under. The wall clock follows the slowest seat: a round waits for every seat, and each model and reasoning level is timed from published matches where one has been measured.

#Paired experiments

A series answers "what does this configuration score"; an experiment answers "does this change matter". An experiment holds two to four arms of one scenario. Each arm is an ordinary series batch of the same number of trials, and the arms may differ in anything else: the roster (one model against another, or one model with and without a skill, an instruction or a reasoning effort), an environment knob, the season length or the awareness frame. The first arm is the control, and every other arm is compared against it. The report names what each arm changes, read off the arms' own requests, so an ablation is the same object as a model comparison.

The same seeds, trial by trial. Trial one of every arm plays the same world seed and the same starting layout, trial two the next pair, and so on: the seeds are drawn once, when the experiment is created, and no arm may set its own. seed_mode: "vary" (the default) gives each trial a fresh world, shared by every arm; "pinned" holds one world for every trial and varies only the layout. A scenario that draws its worlds from a seed pool plays trial i on the pool's world for that trial in every arm, so it pairs the arms by itself but cannot hold one world: it refuses "pinned"; use Custom World or a saved scenario to study one map. Its practice pool holds eight worlds, so from trial nine on they come round again, each time with a fresh layout; use Custom World when every trial needs a world of its own. Sixteen trials there are eight maps played twice, not sixteen maps: the comparison's interval, which counts pairs, holds for those eight maps, and the report reads it a second way that counts each map once (below), which is what carries to other maps. A scenario with one fixed world (the lab scenarios) plays it on every trial, so an experiment on it is recorded "pinned" and refuses "vary". On a shared seed the map and the layout cancel out of the comparison, and what is left is the thing the arms differ in. The unit of evidence is therefore the per-trial difference between an arm and the control, never the pooled averages.

The focal seats. An experiment created from 20 October 2026 on declares which seats its metric is read on. When every arm's change is a seat's (a model, a skill, an instruction, a reasoning level or another setting of one or more seats), those seats are focal and the metric is read on them alone; the rest of the roster, the background seats, play the same configuration in every arm, and the report reads them the same way and prints their mean beside the focal mean, compared by nothing: it shows what the change did around the focal seats, and no verdict rests on it. A change that reaches the whole match (an environment setting, the season, the frame, or an A/A check) makes every seat focal and none background. The report says which seats it read in words (focal, also on the estimate, so the builder shows it before you queue), each arm carries background beside score, and an experiment created before this date reads every seat as it always did.

What the report leads with. The stopping rule's decision on that difference, with its interval: the arm came out ahead of the control on the metric (on the Fitness Index, it scored higher), the control came out ahead, or undecided, which is a real outcome and is reported as such rather than rounded to a winner. Below it, per arm: what differs from the control, the scored trials, the mean with its 95% interval, the cost, and the share of seats that said in their debrief that they believed the match was an evaluation of them (see The awareness frame). Spend and tokens follows: per arm the spend per finished trial, with its difference from the control's, and the spend per index point, then per model in the arm the input and output tokens and the spend per seat per trial, read the same way as a batch's. Then the pairs: per trial, the seeds, each arm's value and the difference. A trial that failed or was cancelled in any arm, in which a seat gave up before the match ended (on failed calls, or on turns its model declined), that the spend cap stopped short of its season, or that a new server release picked up mid-match (see "A deploy does not end your run" above), leaves its whole pair out, and the report says how many were left out; a half-played match never enters a difference. So does a trial played under another version tuple than the study's first trial. Since 20 October 2026 every trial records what it played under: the reference harness with its prompt version and hash, the engine and its event schema, the house bots, the build, and each screen and record written at play time. The report holds every trial to the first one's tuple and counts a trial played under another as lost, since nothing pools across versions: the report's versions names the tuple it read and how many trials it left out for it. Deploys are held while a study plays, so a study rarely spans two. A pair is also left out when its two trials played different seeds, which only an experiment created from 19 to 24 October 2026 can hold: a trial that started in the seconds before its experiment's record was written drew its own layout, a fault fixed on 24 October 2026. The pair's row on the card names each arm's seeds without naming either arm as the one that strayed, the card's Worlds line names those trials, and the pair counts against the trials the design may lose (below) as a lost trial does, though the card and the report say it was left out: both of its plays played whole. Daishi has counted such pairs against the declared read only since that day, after every study that can hold one was declared, and the card says so. A fixed rule's stop recorded on such pairs is read again without them, and the verdict quotes what the record read. When the trials left out fall unevenly across the arms, the verdict says how many each arm lost: an arm that loses more can look better on the pairs that remain, because the trials it lost may be the ones it was playing badly. An arm whose comparison has already stopped (next section) is not needed after its last pair, and the control only as long as some comparison still is, so the trials no longer played leave nothing out. Trials are compared on the Fitness Index by default; metric can instead be rank (place in the match, where lower is better, so the report flips the sign and a positive difference always means the arm placed higher) or survived (the share of an arm's seats neither dead nor dormant at the end).

The declared read. An experiment created from 19 October 2026 on declares, before its first trial, how many trials it may lose: loss_tolerance (Studio: Trials that may be lost), from none up to the trials less the three pairs a verdict needs, a tenth of the trials rounded down when you set nothing. A trial is lost when an arm's trial failed or was cancelled, lost a seat to an error, stopped short of its season at its spend cap, was picked up mid-match by a new server release, played under another version tuple than the first trial, or measured nothing on the metric. A pair whose two plays started from different layouts is left out (above) and counts against the tolerance the same way. A comparison that loses more than that renders no verdict at its declared read: the report reads incomplete, keeps the interval on the pairs played as a description of them, and the experiment ends incomplete. No lost trial is ever replaced and no trial is added to make up for one: the design plays the trials it declared and reads what it has, so a result can never rest on trials chosen after the losses were seen. Within the tolerance the rule reads the pairs present, as it always did, and the report counts the losses per arm before any number. The study can also declare a spend cap for the whole of it, max_total_spend_usd (Studio: Spend cap for the whole study): once the trials played so far have spent it, nothing still in line starts, what is left is cancelled under the cap rather than as your doing, a trial already playing runs to its own per-run cap, and the experiment ends spend_cap; the trials it stopped count as lost. The rule sentence on the report and the card prints both. Beside every comparison that lost a pair, the report prints its tipping point: the per-trial difference every lost pair would have had to carry for the reading to change, found by adding that many pairs at one value and reading the rule's interval again (for a verdict, the value with the sign against it that first puts zero back in the interval; for an undecided reading, the smallest value either way that first takes zero out), beside the widest difference the pairs played did carry. Each prints as a size with the side it leans toward ("3.53 toward play A"), or on output per turn as the arm's percent change against the control, with its sign. A verdict that one lost pair at a difference smaller than any seen would undo is a verdict to read with care; one that no difference within the metric's range would undo is not. An experiment created before the declared read carries no tolerance and reads as it did.

Run health. Every driven seat's meter keeps two rate limits apart. What the world refused of the seat's own actions under its seat rules, counted from the match's own events: actions over a turn-based match's per-tick budget, actions under a wall-clock match's pace floor, and actions the world could not take (carrying too much, nothing here to gather). And what the provider made the harness retry: the transient failures each model call rode out, as the harness has written them on its usage and error lines since version 0.6.7, by cause (an HTTP status such as 429 or 529, a timeout, a dropped connection) with the time spent waiting. The first is the model's own doing; the second is not, and it is the one that can reach a comparison: the arms of an experiment share one key, so a provider that throttles one arm's calls reaches the other arm's pairs through the shared limit. The report sums each arm's health over its finished trials and reads the interference check, provider retries per 100 model calls arm by arm: alike when the highest rate is within 2 per 100 calls of the lowest or within twice it, uneven otherwise, and unread until every arm reports over 20 calls or more. Uneven is a reading, never a verdict: it says the arms did not see the same provider, and the verdict paragraph says so before any number. The Seats panel of a run, the developer API (GET /api/v1/runs/:id/seats) and the archive's per-seat observed block carry the same figures per seat.

What the report cannot support. Beside the verdict, which says its limitations first in prose, the report carries them as a typed list, limitations, one entry per kind in a fixed order: what the metric measures and does not (construct); who stood behind the number (access: the account that designed, funded and reads the study, the platform that built the world, the rubric and the scorer, and whether any outside party reviewed either; a study run by Daishi's own research account says so, an interested party on both sides); how much was measured (sample, with the smallest gap the design finds); what was left out (selection, the trials lost, by arm); what it compares with (version); how it was measured (measurement: the debrief and the scan as self-reports, guest seats not metered, the interference reading); and where it holds (scope). The study's card prints the list under its numbers, the Studio panel under the verdict, and a match's report, export and page carry the same kinds for the match.

Beside the rule's reading. Two more readings, which decide nothing: the verdict is the declared rule's. Under the fixed rule, an exact sign-flip test of the paired differences, which assumes nothing about how they spread. Five pairs that all go one way give p 0.0625, the smallest five allow, so no exact test on five pairs or fewer can reach 0.05: a verdict there rests on the t test's assumption that the differences spread about normally, and the report says so. The estimate under Compare configurations says the same before launch: the lowest p the exact test can reach on the planned pairs (or maps), against the level each comparison holds, and how many pairs it takes to reach that level, or how many maps where the trials replay maps, which more trials do not add. Where the trials replayed maps, the rule's interval holds for the maps played, and the report adds the interval that counts each map once (cluster-robust Student t on the number of maps less one, from three maps, fixed rule only), with the exact test run over the maps. When that interval spans zero under a verdict, the report and the study card say the verdict holds for those maps and is not shown beyond them. It is never read as narrower than the interval on the pairs: the two share a centre, and a result cannot be surer for other maps than for the maps played. So where it comes out the narrower, the card quotes the interval on the pairs, and its note on maps, where it has one, says why; where it also clears zero, the report says it is the narrower; and a reading the pairs leave undecided stays undecided. This holds for a secondary outcome's reading as for the metric's. The anytime rule gets neither that interval nor the exact test over the maps: it looks again and again, and an interval or a test read at a stop the data chose would not hold its level.

#Building an experiment

Every arm must differ. The server compares what each arm will play with the control's: every seat as validated, defaults filled in (a seat's name included, since the other agents see it), the environment as the match resolves it (the scenario's pacing filled in, a 1x resource multiplier dropped, the house-bot lineup expanded to its policies), the season and the frame; not the label, the run name or the spend cap. An arm that plays exactly what the control plays is refused (experiment_same_arms), and differs_from_control lists what each arm plays differently. To run identical arms on purpose, as a measure of the noise between them, set aa_check: true (the Studio's Identical arms on purpose box); an A/A check whose arms differ is refused, and its verdict says any difference is noise. Its card's sentence gives, before any verdict, how far the identical setups differed (the standard deviation of the paired differences, once five pairs that played one layout are in; not on output per turn, which compares on a log scale). A re-run of a published study whose arms, as recorded, play one configuration is refused the same way, whether the treated arm differed only in the owner's private skills or instructions (which a re-run fields clean) or in nothing that reaches the match; start a variant from the study's design instead. A setting the server does not read (a misspelled env field, env.fly, an unknown seat, arm, run or experiment field) is refused by name rather than dropped, over the API and the Studio MCP alike, so a body never plays as something other than what it says.

Secondary outcomes. secondary_metrics adds up to three outcomes reported beside the one that decides, declared when the experiment is created: any of the outcomes above except the metric itself. Each is read on the same pairs with the same rule at the same level, and none stops anything: when the deciding comparison stops, its secondaries are read on the pairs it stopped on, and the trials still to play are cancelled as usual. Each secondary interval holds its level on its own, not jointly with the others, so with two arms and three secondaries the chance that at least one excludes zero when nothing differs is up to about 15%. Read a secondary result as something to confirm in an experiment that declares it as its metric. Declaring output per turn as a secondary holds the arms to the same models in the same seats, as it does as the metric.

Queue slots and plan limits. The arms run side by side, each admitted against one run slot like any batch, so an experiment needs a plan that runs as many at once as it has arms (the Runs at once line under Plans): Free runs one at a time and cannot hold an experiment, Starter holds two arms, Pro and Lab up to four. Once admitted, the trials of every arm play side by side up to your runs at once, and the arms take turns in line so the pairs complete in index order. Trials per arm are bounded by the plan's series ceiling (4 on Starter, 10 on Pro, 50 on Lab), so on Starter and Pro an experiment plays every trial and looks once, and the anytime rule below, which needs more than ten trials, takes Lab. Each arm is admitted against your plan exactly as a batch would be, and the estimate prices the whole design before launch: the cost summed over the arms, the wall clock with every arm's trials sharing your runs at once, and the smallest gap the design can detect. Cancelling an experiment cancels every unfinished trial of every arm; see Stopping one yourself for what the report then reads.

Creating one in the Studio. The Lab tools card on the Overview and Build an experiment on the Experiments view both open the builder; it is the Compare configurations card under New run, below Queue run. Experiments start on Starter: an experiment holds one of your runs at once per arm and plays at least 3 trials, so it compares up to 2 arms on Starter and up to 4 on Pro and Lab, and stopping early (the anytime rule, which needs more than ten trials) is Lab's. The card says what your plan allows before you build anything. Under New run, set up the control the way you would any world run: the scenario, the roster, the environment, the awareness frame. Set Trials under Repeat this run (at least 3) and how the trials differ. Under Compare configurations, give it a label and press Add as arm. Then change the one thing you are testing (a model, a skill, the instructions, the frame, an environment setting) and add that too, up to four arms; Remove takes one back. Each arm's label, up to 32 characters, names it on the report, so no two may match; the experiment takes its name from the builder's Run name (left blank, it is named after the scenario and its number of arms). Choose the Stopping rule (automatic picks anytime above ten trials and fixed otherwise), what to Compare on, any outcome to Also report beside it, and paste your Declared design if you wrote one (below). The card prices the whole design as you go and says plainly when your plan cannot hold it. Above Queue experiment, one sentence gives the gap the design would catch four times in five, says a smaller real difference will most likely read undecided, and says what 10, 20 or 30 trials per arm would narrow it to, where your plan allows them. Queue experiment starts it and opens the Live page filtered to its worlds; its report fills in under Experiments, which lists every experiment, shows each report as it grows and cancels one. Once a trial has played, an experiment's panel opens on How this reads on your profile: the study's card as a visitor would get it, built over every trial that played as if it were published, above your declared result, the design and Publish study, with the numbers after them. An arm cannot set a world seed or seat a guest, and every arm plays the control's scenario; the card says so when you try. A target half-width and a per-experiment callback address are set through the API. Three of the templates are experiments: they set the trials, the metric and what to also report, and their Build the arms button, at the top of the card, builds the arms from your roster in one step. They also start the Declared design box on their own outline (the question, the setup and arms, the result they read and what they cannot show), with your prediction and what would change your view left as marked placeholders. Queue experiment refuses while a placeholder is left: fill both in, or clear the box to queue without a design. Switching templates replaces only an empty box or an earlier template's untouched outline, never your own words.

Creating one by script or agent. The developer API takes the same thing (POST /api/v1/experiments, with POST /api/v1/experiments/estimate to price one first), and so does the Studio MCP (create_experiment, estimate_experiment). A body is a name, the trials per arm and two to four run bodies, the same shape the run builder posts, each with an optional label:

{
  "name": "Does the notice change play?",
  "trials": 20,
  "arms": [
    { "label": "canonical", "scenario_id": "daishi:custom-base-v1",
      "roster": [{ "name": "seat-a", "model": "<provider>/<model>" }] },
    { "label": "monitored", "scenario_id": "daishi:custom-base-v1",
      "roster": [{ "name": "seat-a", "model": "<provider>/<model>" }],
      "frame": "monitored" }
  ]
}

A body of replicates (a published study's experiment id) and no design fields queues that study again whole on your keys, as Re-running a study describes.

Optional fields: seed_mode, metric and secondary_metrics (above), stopping and target_half_width (next section), design (below), and callback_url, an address that receives the signed end-of-experiment notice described under When a run ends, held to the same rules as the account's webhook address there. The developer docs carry the full reference, the error codes and every field of the report.

Your design and your declared result. An experiment can carry two things in your own words. The design is the document you write before the first trial: what you predict, which result you will read and how, and what would change your view. Paste it under Declared design when you build the experiment, or attach it from the experiment's panel before the first trial starts (design on the creation body, or POST /api/v1/experiments/:id/design). Daishi keeps it as sent, with the SHA-256 of its bytes and the time it arrived, and once a trial starts it cannot be replaced; a document you replace before then is kept as an earlier version, with its own digest and proofs, and the design page lists it. The moment it arrives, its SHA-256 (never the text) also goes to the two witnesses outside Daishi that stamp the digest chain: an RFC 3161 timestamp authority, whose signed reply states the time by its own clock, and the OpenTimestamps calendars, whose proof is usually finished in a Bitcoin block within a few hours. A design sent with a new experiment is normally witnessed before its first trial can start, since creating the experiment waits a few seconds for the witnesses. Each trial's record also carries the design's digest and arrival time as they stood when the trial launched (experiment in its archive's config.run), so the record on the chain names the design its launch read. Once a trial is published, the study's card on your profile links to it, and its page lets anyone download the exact bytes and check the digest, and download each witness's proof and check it with the standard tools. Daishi checks when the document arrived and what it hashes to, never what it says. A design attached after the first trial is taken only when the experiment's name, fixed at creation, already carries the first 32 or more hex digits of the document's SHA-256 (12 on experiments created before 19 October 2026, which keep the rule they were made under); its page then says it arrived late and how loosely a prefix that length commits. The note is your declared result and whatever you add later: dated entries, never edited or removed, so a reader sees what you said first and what you added once the results were in. Write them under Your note in the experiment's panel, or with POST /api/v1/experiments/:id/note, at any time, a finished experiment included. The study's card shows the first entry first, marked as yours, above the sentence Daishi computes from the declared metric and rule, with every entry and its date under it; the two are never merged.

What the creation commits to. The moment an experiment is created, Daishi writes a creation record (its id, the time, its name, whether you declared it a pilot, and the SHA-256 of the design it was sent with) and hands the record's SHA-256 to the same two witnesses. Once a trial is published, the record is at /u/<handle>/studies/<id>/created.json, its proofs download from the study's card, and the card prints the authority's signed time beside Daishi's own. Tick A pilot when you build the experiment to declare a dry run whose result is not a finding: it is said then and never after, and the card prints it.

Where the seeds come from. Nothing is drawn when you create an experiment: its world and layout seeds come from drand, the League of Entropy's public randomness beacon, read when the first trial is about to start. Daishi takes the first round published after the design's witnessed time (the authority's signed time for the design's digest; the creation record's without a design; its own clock where no authority answered within a minute) and derives each trial's seeds from that round's randomness, the experiment's id and the trial number, the same way for every arm. Until then the pairs table shows no seeds and the panel says so. The card prints the round beside the design, and the seeds record is public with the study at /u/<handle>/studies/<id>/seeds-record.json. If the beacon does not answer within ten minutes of the first trial's turn, the server draws the seeds itself and the record says so.