Reading a study

#Reading a study

How to read a Daishi result names each unit and each kind of result in a line or two. This section holds the detail behind those lines: how a trial's world is drawn, what a pair is, what a study is once it is published, what "declared" covers, what each reading on an experiment's page means, and what a study does not settle.

#How a trial's world is drawn

A world is the map one match is played on, drawn from a seed. Where its seats start is drawn separately, so a later trial on the same map starts from a new layout. A trial's world comes from its scenario. A scenario that draws from the public practice pool walks its 8 worlds by trial number, so a study with more trials than that plays worlds more than once, and its page says so. A lab scenario plays one map on every trial, from a new starting layout each time. Custom World can give every trial a world of its own. Paired experiments says how each is drawn.

#What a pair is

A paired experiment is a comparison of two to four versions of one run, called arms, on one scenario. Each arm is a series with the same number of trials, and the first arm is the control. Every arm plays trial 1 on the same world from the same starting layout, and the same holds for each trial after it, so what the map does to every arm alike cancels out. What is left is the change plus each match's own luck, which is why the result is an interval over pairs and not one number. An arm's trial and the control's trial of the same number are a pair. The evidence is the difference inside each pair, never the two pooled averages. It answers "did this change move the metric here". It needs at least 3 pairs to give a verdict under a rule that looks once, and 10 under a rule that looks after every trial. The opening sentence of a study page calls an arm a setup and a pair a paired match. The page lists what differs between the arms, and nothing holds that to one thing.

Every score belongs to a seat, and seats that shared a match are counted together, because they shared a world.

#A study, once published

A study is what a series, a paired experiment or a run set is called once it is published: one card on its owner's profile and one page of its own. A study is read whole or not at all. An owner can publish some of its trials, but the page of a series or an experiment computes a reading only when every trial that played is published, so a reading never rests on a chosen subset of the trials.

#What declared covers

Declared means fixed before the first trial and recorded with its time. An experiment declares its metric and its stopping rule, and since October 2026 also how many trials it may lose and which seats the metric is read on. Its owner can attach a design in their own words. Daishi keeps the design as sent and sends its SHA-256 digest to two timestamp witnesses outside Daishi. The design's page lists each proof, so a reader can check the bytes, and that they existed by the time each witness states, without taking Daishi's word for it. That the first trial started later is Daishi's own timestamp.

The word is "declared", never "pre-registered". There is no registry and no reviewer: Daishi checks when a design arrived and what it hashes to, never what it says.

#The readings

A paired experiment reads each arm against the control, and its page prints the reading the declared rule reached:

• Ahead. The declared rule found a difference in the arm's favor. A rule that looks once needs two intervals on the paired difference to leave out zero: a Student t interval, which is the one printed, and a bootstrap interval. A rule that looks after every trial, from trial 10 on, uses a wider interval built for repeated looks. With more than two arms the level is split across the comparisons, and the study's stopping rule line states the level each one ran at. On a metric with no better side the page says "higher".
• Control ahead. The same, the other way.
• Undecided. The rule did not find a difference. This is a result, and it does not mean the two are equal: every difference inside the interval is still possible, including ones that would matter. The interval is the bound. Usually it includes zero. Under a rule that looks once the printed interval can leave zero out while the bootstrap one does not, and the report says so. Where it has a spread to work from, the page also prints the gap the design would find four times in five, which describes the design, not this result.
• Incomplete. The study lost more trials than it declared it could lose, to faults or to its spend cap. The page keeps the interval as a description and gives no verdict.
• No verdict. Fewer than 3 pairs were left, or the study ended before its rule had looked. A rule that looks once looks only at its planned number of trials, so a study its owner cancelled earlier has no verdict and no interval. A rule that looks after every trial keeps its reading through a cancel once it has begun to look: undecided, with its interval. The page says when a study was cancelled. Its table prints "no verdict" for this and for an incomplete study, and the sentence above the table says which.

Three more readings appear where they apply:

• When worlds repeat. The verdict counts pairs and holds for the worlds played. Where the rule looks once, has given its reading, and the pairs covered 3 or more worlds, the page prints a second interval that counts each world once. That one is what carries to other worlds drawn the same way, and it has only as many units as there are worlds. Where the rule gave a verdict and that interval takes in zero, the page says the result is shown for the worlds played only. Under a rule that looks after every trial, or on fewer worlds, the page prints no such interval and says so: the reading is for the worlds played. The practice pool's worlds are public, so they are a fixed set a seat can be built against, not a sample.
• A re-run is read beside its original, once both are complete and published in full, with an interval on the gap between the two results. "Told apart" means that interval leaves out zero. Not told apart is not agreement: every gap inside the interval fits both. A re-run by the original's owner is listed and never counted as anyone else's.
• An A/A check plays one configuration in every arm, so a verdict there is a false positive the rule allows at its level.

#What the mean hides

Beside each verdict, a study's own page also prints what the mean hides, read off the pairs the verdict read and deciding nothing:

• The spread. How the pairs fell: where the verdict names a side, how many pairs each side came out ahead in; where it does not, how many differences were above zero and how many below, naming no side. The lowest and highest paired difference, arm minus control (on rank the control's place minus the arm's, so a positive one means the arm placed higher; on output per turn the arm's percent change against the control), with the world each played, and the quartiles between. The standard deviation of each side's trial values over these pairs, beside how many of its seats ended dormant or dead: a seat that ends dormant usually scores far below the rest, so a few of them can make most of a side's spread. The page prints no test of whether one side spread wider: a few dormant seats put the values far from the normal spread such a test assumes. Where worlds repeat, the worlds with the lowest and highest mean difference, and the share of the spread that lies between worlds rather than within one, with a 95% interval from the F ratio (exact when every world was played equally often, approximate otherwise). With few worlds that interval is wide, and where it reaches 0 the page says these pairs cannot tell the share from none. Last, the play turns cut off at the seats' output cap, arm by arm, which the page also prints beside the verdict whenever any turn was cut.
• The parts of the Index. On a study of the Fitness Index, the gap split among the five parts: each part's weight times its mean paired difference (the five add up to the gap), its share of the spread in the pair differences (the five add up to 100%, and a part can pull against the rest) with the lowest and highest share when one pair is left out in turn, from four pairs up, and the seats that scored 0 on it. A part on which at least three seats in four scored 0 is marked, and named beside the verdict with the count of seats off the floor: its points and its share come from those few seats. The mark compares the count of seats at 0 with three in four of the seats, so leaving out one pair moves that count by the pair's seats at most.
• Scored other ways. The comparison read again by the declared rule, on the same pairs at the same level: with each part of the Index left out and the other four reweighted to sum to one, with the five weighted alike, and under each other rubric version, whose row says which parts it scores differently on these seats. A row that reads differently from the declared verdict is marked changed. Only the declared verdict names a side: another row says whether the rule then finds a difference, finds one the other way, or reads undecided. The rows are further readings of the same pairs at the level set for one, and several act on one part, so a row that finds a difference is a reason to look, not a result, and the declared verdict stands. The seats were told the study's own rubric, so another version scores them on a rule they did not play for. Under the anytime rule, each row reads the pairs the rule stopped on; scored another way, the rule might have stopped at another trial. A study of one part of the Index is read under the other rubric versions only. Each family of rows appears only where the declared scoring, recomputed the same way, gives back every trial's value and the declared verdict, mean and interval. A row under another rubric version scores each trial's match as GET /api/matches/<match>/report?rubric=<version> does, from the same archive, so a reader can rebuild each trial's value under that rubric from the report's seats: the mean over the seats seats.csv marks focal for that trial, kept to three places.

#How a figure prints

A figure on a study's page, in its Markdown, on its card and in the Studio's own tables of the study prints from the value the record keeps, rounded once to the places shown. The record keeps each trial's value to three places, an arm's score and its interval to two, each model's score within an arm or a series to one, and a comparison's mean and interval to three, four for a difference on the log scale. Every figure but a trial's value is kept by the rule it prints by, below. The stopping rule reads a trial's value, so its keep belongs to the analysis version: it rounds the binary number, and a value at a tie in its fourth place can keep the lower digit until the next analysis version moves it to the rule below. A comparison's mean is kept from the exact decimal its trials' values give, and a difference that is not zero keeps its side of zero, with up to three more places where those would make it 0. A figure the page works out is kept the same way before it prints: each trial's difference to three places, six on the log scale, the spread figures beside them to six places, and a row scored another way to three, as the declared row is.

The rounding is half away from zero on those digits, so a reader who rounds the record's figures by hand lands on the printed digit: a trial's value kept as 60.65 prints 60.7. JavaScript's toFixed and Python's round() round the binary number nearest 60.65, which lies just below it, and print 60.6, and numpy rounds half to even, which prints 12.05 as 12.0. In Python, Decimal(str(v)).quantize(Decimal('0.1'), rounding=ROUND_HALF_UP) gives the page's digit, as does a spreadsheet's ROUND. A figure recomputed from the study's files lands on the page's digit once it is kept to the record's places first, as the study's recompute script keeps it (Recomputing a study yourself); rounded straight from the recompute, a figure kept one place past its print, such as an arm's score, can land a unit off. Either way round, a difference prints the same digits, and zero prints with no sign.

An interval end that is not zero but would print as zero takes the places it needs: an end of 0.003 beside "clear of zero" reads 0.003, not 0.0. The words of an undecided reading round at the places the sentence prints, without the extra places, and an end that rounds to zero there names no side.

#What a study does not settle

• It measures a model in a harness. A score reflects the prompt, the tool loop and the pace of play as well as the model. Holding the harness fixed does not make two models' conditions equal: what a seat carries over from its last turn depends on what its route returns. Name the route and the setting beside any comparison of models, and read every claim as covering that harness as it ran. (Caveats)
• Seats were told an idle seat cannot survive. Until harness 0.6.12 the rest tool said "you cannot idle to survive", world_info said "idle agents starve", and the prompt called food, or a maintained shelter, mandatory. In a 100-round match that does not hold. An untrained seat nobody attacks that only rests ends the match at 49.5 energy, awake, and one that never acts goes dormant in round 100 and is not dead. Staying awake is not scoring: under rubric 2.3 and later the survival score counts in proportion to a seat's world actions (10 per 100 rounds earn all of it), and a rest is not one, so a seat that only rests scores 0 on survival. Harness 0.6.12, live since 11 November 2026, corrected the text (what changed). Every world match on record before that release was played while the rest tool said so; world_info said so from 18 August 2026. What the correction did to play is not measured: no study has played the two texts against each other. A paired study would measure it for the seats and scenario it plays. Daishi serves one version of that text to every seat at a time, so playing the old text in one arm today would mean serving the false claim to every seat again. Runs on 0.6.12 are their own comparison group, so no study pools the two.
• A study's page does not count, for each seat, the attacks it received or the rests the harness made for it. A study's seats.csv counts the attacks, raids and thefts a seat made, and a match report prints its attacks and raids. Neither counts those made on it. The match's event log holds every attack with its target, so a reader can count them, and the attacker can be a house script. Raids on a seat's structures and thefts from its storehouse are in the log with their owner, and are not totaled for that seat either. The reference harness rests for a seat whose turn spent no action: the reply named no tool or one that does not exist, its one call was a free tool such as status or scoring_info, or the call was refused before the world ran it. The prompt says so for a free tool, and the seat's next observation says so each time it happens. In the public record such a rest reads like one the model chose. The counts that tell them apart are in the run's harness logs, which only the owner can read. An owner's write-up can give any of these figures from the event log and the harness logs, and some published write-ups do. Both counts on the page, for each seat, would close this.
• Model seats seldom meet. An attack and a direct trade need both seats in one region (a raid or a theft reaches a seat that is elsewhere, and a market offer completes at its market after the offerer has left), so for seats that never stood together a zero between them says they never met, not that they left each other alone. A seat can check who shares its region with a look, by inspecting an agent by name, or with a broadcast, which reaches only the sender's region and tells the sender who heard it. In the reference harness each of the three takes the seat's one action for the round. A seat that does not check still learns of another whose act reaches it, such as a broadcast it hears or an attack on it, without spending its action. So after a short meeting in which neither did any of this, a zero can also mean the seat never saw the other. In the studies published so far the model seats seldom dealt with one another, and most never stood in a region with another seat. A report's counts of completed trades and attacks include dealings with house scripts; only the counts between seats leave them out. The count of rounds a seat stood with another exists only on matches archived since 16 October 2026; for an earlier match it can be rebuilt from the match's event log, which records every spawn and move with its region. A study's page prints how often its seats met only when the study declares an outcome between seats (see What an experiment can compare); a study that declares none does not say. What makes seats deal is not known: seats that started close still seldom dealt.
• A model's name is not a version. The tuple pins Daishi's side of a match. A provider can change what answers to a model's name, and a router can serve one name from several hosts in one match. The record lists the model ids and hosts that answered. Nothing holds them fixed.
• Small studies find only large differences. A few pairs usually give a verdict only when the gap is large, and a smaller one can stay undecided. An experiment's page prints the gap its design would find four times in five where it has a spread to work from: its own pairs once five are in, a reference experiment its owner named, or a prior, which exists for the Fitness Index and for output tokens per turn only. A verdict from fewer than 6 pairs cannot be confirmed by an exact test at the usual level, and a small study that does reach a verdict tends to overstate the size. Treat it as the next study's question.
• You see the studies an owner chose to publish. A study's page reads all of its trials or none, but which studies are published is the owner's choice. For Daishi's own research account, the profile and each paired experiment's page count every paired experiment it created beside the ones it published, and say what reason it gave for any it cancelled, or that it gave none. That count leaves out series and run sets, says nothing of what an unpublished experiment found, and exists for no other account. Each verdict allows its own false positive with no correction across studies. Outcomes declared beside the metric each hold their level alone, not jointly.
• A release that lands mid-match costs that trial's pair. The run carries on in the new release, but each seat loses its own memory of the match, so the report leaves the whole pair out and says how many it left out. A trial played under another version tuple than the study's first is left out the same way.
• The transcripts behind a study are the owner's alone. A run's harness logs hold what each seat was sent and what it replied. Only the account that owns the run can read them, and they are deleted once they are older than the owner's plan keeps them (30, 90 or 365 days below Lab), a published study's included. A seat's log also stops at 64 MB, with a record that says so. The public match archive is never pruned.

#A profile's per-model figures

A profile's per-model figures summarize the standalone runs that profile's owner chose to publish and that played a full season in one release, whatever scenario and versions each was played under. They describe that published record: a run kept private is not counted, a study's trials and a run set's runs are not in them, and no rule compared the models.

#The same things under other names

The Studio's builder and the API name the same things their own way.