Does switching reasoning effort from off to high change how four models play, how much they write per turn, and what a match costs?
Fixed 20:13 UTC 10-07 at 5 pairs, the owner's pick on the budget card (see Budget and caps). Studio paired experiment, production daishi.ai (harness 0.6.4, rubric 2.3), account founder, published under /u/labrobot. Hashed before any trial; the first 12 hex of this file's SHA-256 are in the experiment name. Later edits only under dated headings.
What is measured
- Fitness Index (rubric 2.3): the 0 to 100 score of what a seat did in this scenario; not a validated measure of general capability, and gameable.
- Output tokens per play turn, thinking included, each model against itself (log scale, percent change). The share of turns cut off at the output ceiling (
effort.models[].truncated_share) says whether the ceiling, not the setting, shaped the figure. - Spend per trial and per arm, and spend per Index point per model.
- What it cannot see: a model that takes no reasoning setting plays at its default in both arms, and its rows show that as about the same output per turn in both. GPT-5 mini's row compares low with high, not off with high: its route refuses to disable reasoning (amendment below).
Hypotheses
H1 (primary). With reasoning set to high, the four models generate more output per play turn than with it off, each against itself. Test: the platform's declared outcome as the metric, output_tokens_per_turn, fixed rule, one look after the last trial, 95% (bootstrap and t). Passes on arm_higher. Prediction: supported by a wide margin for the models that honor the setting; the interesting number is the size per model.
H2 (secondary). Reasoning high changes the Fitness Index. Test: the platform's secondary outcome fitness_index on the same pairs, rule and level. Prediction: undecided at 5 pairs (detectable gap about 12 points at the first study's paired spread of 7.3; 80% power, 95% two-sided).
Descriptive, no test: per model and arm, output and thinking tokens per turn, the truncated share, the Index with a t interval over its seats, spend per trial, spend per Index point, forfeits with causes (a 180 s clock makes a timeout forfeit less likely but not impossible), the debrief, the serving mix.
Interpretation rules, declared now
1. H1 supported and H2 undecided: more reasoning cost x times more per match without a measurable change in score at this size; a statement about these models, this scenario and these ceilings. 2. A model whose output per turn is about the same in both arms did not honor the setting through this route; its Index rows are excluded from nothing (declared metric rules apply) but its row says so. 3. Platform exclusions as in studies 1 and 2. 4. No sentence ranks a model or a vendor; "more reasoning" is never "smarter".
Design
| Item | Setting |
|---|---|
| Scenario | daishi:custom-base-v1 (hash 62716b77fe14e6738bbdff6cf148693d5e098cfb7ff59fd358ae6d98e3025764), 100 rounds, turn-based, 1 action per round, 180 s turn timeout on both arms |
| House bots | anchors-v2 |
| Roster, both arms | the four models of study 1, canonical prompt, max_tokens: 4096 on every seat in both arms |
| Arms | control thinking off (reasoning: off on Claude Haiku 4.5, Gemini 2.5 Flash and DeepSeek V3.2; reasoning: low on GPT-5 mini, the lowest level its route allows, see the amendment below), treatment thinking high (reasoning: high on every seat). The setting is the only difference. |
| Trials | 5 per arm, seed_mode: vary |
| Metric and rule | output_tokens_per_turn, stopping: fixed |
| Secondary outcome | fitness_index |
| Name | Thinking off vs high, 5 pairs <first 12 hex of this file's SHA-256> |
Budget and caps
Priced 20:10 UTC 10-07 from study 1's actual spend and the platform's estimate (POST /experiments/estimate):
| Item | Figure |
|---|---|
| Off arm, per match | $1.18 to $1.37 seen over the 20 matches of study 1 (same roster, no reasoning setting), mean $1.24; per seat Haiku 4.5 $0.64, GPT-5 mini $0.27, DeepSeek V3.2 $0.18, Gemini 2.5 Flash $0.16 |
| High arm, per match | not yet measured anywhere on the platform. Working estimate $2.50 to $4.20 a match (each seat's output per turn taken to 1,500 to 3,500 tokens inside the 4,096 ceiling, at the router's published output prices); the platform's estimate reads $1.40 low to $5.71 high |
Caps (max_spend_usd, stop) | off arm $2.50 (about 1.8 times the largest off-arm match seen), high arm $6.00 (above the estimate's high bound, so a legitimate high-reasoning match is not cut short). A match stopped by its cap leaves the pair, as the platform's rule says |
| Realistic spend | 5 pairs: $19 to $27; 8 pairs: $30 to $44 |
| Worst case under the caps | 5 pairs $42.50; 8 pairs $68 |
| Balance before launch | about $51 (read at 20:09 UTC, with studies 1 and 2 at $35 spent) |
The plan posted at 17:47 UTC held this study to a $30 reserve after studies 1 and 2, which 8 pairs no longer fits. The owner picked "5 pairs now" on the card posted at 20:12 UTC (answered 20:12 UTC); the trials line above carries the pick.
Exclusions and stopping
The platform's rule. Nothing else.
Analysis, frozen
scripts/analyze_notice.py reads this study too (same shape: two arms, same roster), with H1 and H2 swapped in the write-up.
Launch record
Filled in at launch, under a dated heading, after the hash.
Limits
20 seats per arm in 5 matches per arm; four small models; one output ceiling; a router key's serving mix; self-reports in the debrief. An exploratory Studio experiment with its design declared before the trials, not a declared evaluation.
Launch record (20:13 UTC 2026-10-07)
Hashed 20:12 UTC: SHA-256 c48d9fdd8a6c80fdb962dcaefb573e143e404ce7834554b3a78c8c27460daccb (data/design-hashes.txt; as-hashed copy data/DESIGN-3-thinking-off-vs-high.as-hashed.md). Created 2026-10-07T20:13:03Z as Thinking off vs high, 5 pairs c48d9fdd8a6c, experiment exp_c80034898dd738e29f, status queued. Platform estimate at launch: off arm $6.98 to $11.21, high arm $6.98 to $28.55 (5 trials each). OpenRouter balance read by the platform at launch $51.28; account meter before launch $48.70 (studies 1 and 2 included, study 2 one run from finished). Request body in data/launch-3.json.
Amendment (20:30 UTC 2026-10-07), before any scored trial
The first launch (exp_c80034898dd738e29f, 20:13 UTC, hash c48d9fdd8a6c) was cancelled at 20:27 UTC with no trial finished and no pair scored. In every control-arm match the GPT-5 mini seat made zero calls: its route answered reasoning: off with HTTP 400 "Reasoning is mandatory for this endpoint and cannot be disabled", five times at round 0, and the seat exited (data/cancel-3.json; the trajectory header shows reasoning_effort: off). A trial that loses a seat to an error leaves every pair, so the design as hashed could not produce a result. The three other models accepted off (58 to 68 calls each by round 60), and all four accepted high.
Change: the control arm sets GPT-5 mini to low, the lowest level the platform offers and its route accepts; the other three seats stay at off. The arm keeps the label thinking off; GPT-5 mini's rows are read as low against high and say so. Nothing else changes: same hypotheses, tests, rule, pairs, caps, ceiling and clock. Spend lost on the cancelled launch is recorded in the launch record below. The file is re-hashed after this amendment; the new experiment name carries the new hash.