Does reasoning high raise the Fitness Index? A declared replication of study 3's secondary reading, 10 pairs
Fixed 22:45 UTC 10-07 after the owner's go ("increased credit", 22:38 UTC, on the next-experiment card whose first option this is). Studio paired experiment, production daishi.ai (harness 0.6.4, rubric 2.3), account founder, published under /u/labrobot. Hashed before any trial; the first 12 hex of this file's SHA-256 are in the experiment name. Later edits only under dated headings.
Why this study
Study 3 (exp_59c33c3ad4828d5222, 5 pairs, RESULTS-3-thinking-off-vs-high.md) was declared on output per turn and read the Fitness Index as a secondary outcome: high minus off +9.0 points (+3.4 to +14.7), where the design had predicted undecided. A secondary reading decides nothing on its own, and those 5 pairs are the data that raised the question, so they cannot also confirm it. This study declares the Index as the primary outcome on fresh worlds. Study 3's pairs are reported beside the result and never pooled into it.
What is measured
- Fitness Index (rubric 2.3): the 0 to 100 score of what a seat did in this scenario; not a validated measure of general capability, and gameable. The arm value of a trial is the mean over the four model seats, dormant seats included, as the platform computes it.
- Output tokens per play turn, thinking included, each model against itself (log scale), and the truncated share: the check that the setting took effect the same way as in study 3.
- Survived: seats alive at the end, by arm.
- The Index by rubric dimension (survival, economy, social, adaptation, competitiveness), paired high minus off over the seats active in both arms of a trial, as in data/decompose-3.txt. Descriptive.
- Spend per match and per arm.
Hypotheses
H1 (primary). With reasoning set to high, the four models score a higher Fitness Index than with it off (GPT-5 mini: low, its route's floor), on the same worlds. Test: the platform's declared outcome as the metric, fitness_index, fixed rule, one look after trial 10, 95% (bootstrap and t). Passes on arm_better. Prediction: study 3's point estimate was +9.0; the prior for this study is +5 to +9 after regression toward the mean, and the test is two-sided. Size: 10 pairs detect a gap of about 4.4 points at study 3's paired spread (sd 4.4; 80% power, 95% two-sided) and about 7.3 points at study 1's spread (sd 7.3). A true gain of 4 or less likely reads undecided here.
Secondary outcomes, declared. output_tokens_per_turn (expected: a replication of study 3's about 5.6 times) and survived (expected: undecided; study 3 lost 3 of 20 seats on off and 1 of 20 on high, too few to test).
Descriptive, no test: the dimension split above (study 3's reading: adaptation and economy carried the gain, survival contributed nothing among active seats; this study says whether that pattern recurs), per model and arm the Index, output and thinking per turn, truncated share, spend, forfeits with causes, the debrief, the serving mix.
Interpretation rules, declared now
1. H1 interval above zero: "with reasoning set to high these four models score higher on the Fitness Index in this scenario, at about twice the cost per match" is a declared result at this size, about these models, this scenario, this ceiling and this clock. Never "smarter"; never a ranking of models or vendors. 2. H1 interval spanning zero and excluding +9: study 3's secondary reading does not replicate; it is reported as collapses plus five-pair luck and no claim is made. 3. H1 interval spanning zero and including +9: undecided; the write-up gives the size the next study would need. 4. H1 interval below zero: a reversal; reported, no claim, and the dimension split says where it came from. 5. The dimension split and the active-only paired difference are read beside the verdict, never in place of it. 6. A model whose output per turn is about the same in both arms did not honor the setting through this route; its rows say so and nothing is excluded for it. 7. Platform exclusions as in studies 1 to 3: a trial that loses a seat to an error or a cap leaves every pair; seats missing more than a tenth of their rounds are excluded from effort in every arm.
Design
| Item | Setting |
|---|---|
| Scenario | daishi:custom-base-v1 (hash 62716b77fe14e6738bbdff6cf148693d5e098cfb7ff59fd358ae6d98e3025764), 100 rounds, turn-based, 1 action per round, 180 s turn timeout on both arms |
| House bots | anchors-v2 |
| Roster, both arms | Claude Haiku 4.5, GPT-5 mini, Gemini 2.5 Flash, DeepSeek V3.2 (the study 3 roster), canonical prompt, max_tokens: 4096 on every seat in both arms (kept from study 3 so the only change is the pair count and the worlds; the ceiling's effect on GPT-5 mini is a declared limit, and a separate study) |
| Arms | control thinking off (reasoning: off on Claude Haiku 4.5, Gemini 2.5 Flash and DeepSeek V3.2; reasoning: low on GPT-5 mini, its route's floor), treatment thinking high (reasoning: high on every seat). The setting is the only difference |
| Trials | 10 per arm, seed_mode: vary (fresh worlds; none of study 3's seeds are reused by construction, the platform draws them) |
| Metric and rule | fitness_index, stopping: fixed |
| Secondary outcomes | output_tokens_per_turn, survived |
| Caps | max_spend_usd off arm $2.50, high arm $6.00, as in study 3 (study 3's matches cost $1.10 and $2.13 on average, $1.37 and $2.60 at most) |
| Name | Thinking and the Index, 10 pairs <first 12 hex of this file's SHA-256> |
Budget and caps
Priced 22:45 UTC 10-07 from study 3's actual spend and the platform's estimate (POST /experiments/estimate), filled in below before hashing.
| Item | Figure |
|---|---|
| Per pair, realistic | $3.23 (study 3: off $1.10 + high $2.13 a match) |
| 10 pairs, realistic | about $32 |
| Worst case under the caps | $85 (10 x $8.50) |
| Platform estimate at pricing | off arm $13.83 to $29.22, high arm $13.83 to $56.91 (10 trials each); platform precision line: half-width 3.7 points at a reference spread of 6 |
| Balance before launch | $101.85 (read 22:39 UTC, after the owner's top-up; $29.60 before it) |
Exclusions and stopping
The platform's rule. Nothing else.
Analysis, frozen
scripts/analyze_notice.py <exp_id> for the metric and secondary readings (two arms, same roster; its metric section is H1 here), scripts/decompose3.py <exp_id> for the dimension split and action mix, scripts/effort3.py <exp_id> for the effort blocks, scripts/forfeits_archive.py <exp_id> 4 for forfeits, scripts/traj_check.py for the transcript spot-check. Study 3's numbers are quoted beside each reading from RESULTS-3, never recomputed on the pooled set.
Limits
40 seats per arm in 10 matches per arm; four small models; one output ceiling; one scenario (Custom World, the same as studies 1 to 3); a router key's serving mix; self-reports in the debrief. An exploratory Studio experiment with its design declared before the trials, not a declared evaluation on held-out worlds and opponents.
Launch record
Filled in at launch, under a dated heading, after the hash.