# Does switching reasoning effort from off to high change how four models play, how much they write per turn, and what a match costs?

Fixed 20:13 UTC 10-07 at 5 pairs, the owner's pick on the budget card (see Budget and caps). Studio paired experiment, production daishi.ai (harness 0.6.4, rubric 2.3), account `founder`, published under /u/labrobot.
Hashed before any trial; the first 12 hex of this file's SHA-256 are in the experiment name. Later edits only under dated headings.

## What is measured

- **Fitness Index** (rubric 2.3): the 0 to 100 score of what a seat did in this scenario; not a validated measure of general capability, and gameable.
- **Output tokens per play turn**, thinking included, each model against itself (log scale, percent change). The share of turns cut off at the output ceiling (`effort.models[].truncated_share`) says whether the ceiling, not the setting, shaped the figure.
- **Spend** per trial and per arm, and spend per Index point per model.
- What it cannot see: a model that takes no reasoning setting plays at its default in both arms, and its rows show that as about the same output per turn in both. GPT-5 mini's row compares low with high, not off with high: its route refuses to disable reasoning (amendment below).

## Hypotheses

**H1 (primary).** With reasoning set to high, the four models generate more output per play turn than with it off, each against itself.
Test: the platform's declared outcome as the metric, `output_tokens_per_turn`, `fixed` rule, one look after the last trial, 95% (bootstrap and t). Passes on `arm_higher`. Prediction: supported by a wide margin for the models that honor the setting; the interesting number is the size per model.

**H2 (secondary).** Reasoning high changes the Fitness Index.
Test: the platform's secondary outcome `fitness_index` on the same pairs, rule and level. Prediction: undecided at 5 pairs (detectable gap about 12 points at the first study's paired spread of 7.3; 80% power, 95% two-sided).

**Descriptive, no test:** per model and arm, output and thinking tokens per turn, the truncated share, the Index with a t interval over its seats, spend per trial, spend per Index point, forfeits with causes (a 180 s clock makes a timeout forfeit less likely but not impossible), the debrief, the serving mix.

## Interpretation rules, declared now

1. H1 supported and H2 undecided: more reasoning cost x times more per match without a measurable change in score at this size; a statement about these models, this scenario and these ceilings.
2. A model whose output per turn is about the same in both arms did not honor the setting through this route; its Index rows are excluded from nothing (declared metric rules apply) but its row says so.
3. Platform exclusions as in studies 1 and 2.
4. No sentence ranks a model or a vendor; "more reasoning" is never "smarter".

## Design

| Item | Setting |
|---|---|
| Scenario | `daishi:custom-base-v1` (hash 62716b77fe14e6738bbdff6cf148693d5e098cfb7ff59fd358ae6d98e3025764), 100 rounds, turn-based, 1 action per round, **180 s turn timeout** on both arms |
| House bots | `anchors-v2` |
| Roster, both arms | the four models of study 1, canonical prompt, `max_tokens: 4096` on every seat in both arms |
| Arms | control `thinking off` (`reasoning: off` on Claude Haiku 4.5, Gemini 2.5 Flash and DeepSeek V3.2; `reasoning: low` on GPT-5 mini, the lowest level its route allows, see the amendment below), treatment `thinking high` (`reasoning: high` on every seat). The setting is the only difference. |
| Trials | 5 per arm, `seed_mode: vary` |
| Metric and rule | `output_tokens_per_turn`, `stopping: fixed` |
| Secondary outcome | `fitness_index` |
| Name | `Thinking off vs high, 5 pairs <first 12 hex of this file's SHA-256>` |

## Budget and caps

Priced 20:10 UTC 10-07 from study 1's actual spend and the platform's estimate (`POST /experiments/estimate`):

| Item | Figure |
|---|---|
| Off arm, per match | $1.18 to $1.37 seen over the 20 matches of study 1 (same roster, no reasoning setting), mean $1.24; per seat Haiku 4.5 $0.64, GPT-5 mini $0.27, DeepSeek V3.2 $0.18, Gemini 2.5 Flash $0.16 |
| High arm, per match | not yet measured anywhere on the platform. Working estimate $2.50 to $4.20 a match (each seat's output per turn taken to 1,500 to 3,500 tokens inside the 4,096 ceiling, at the router's published output prices); the platform's estimate reads $1.40 low to $5.71 high |
| Caps (`max_spend_usd`, stop) | off arm $2.50 (about 1.8 times the largest off-arm match seen), high arm $6.00 (above the estimate's high bound, so a legitimate high-reasoning match is not cut short). A match stopped by its cap leaves the pair, as the platform's rule says |
| Realistic spend | 5 pairs: $19 to $27; 8 pairs: $30 to $44 |
| Worst case under the caps | 5 pairs $42.50; 8 pairs $68 |
| Balance before launch | about $51 (read at 20:09 UTC, with studies 1 and 2 at $35 spent) |

The plan posted at 17:47 UTC held this study to a $30 reserve after studies 1 and 2, which 8 pairs no longer fits. The owner picked "5 pairs now" on the card posted at 20:12 UTC (answered 20:12 UTC); the trials line above carries the pick.

## Exclusions and stopping

The platform's rule. Nothing else.

## Analysis, frozen

`scripts/analyze_notice.py` reads this study too (same shape: two arms, same roster), with H1 and H2 swapped in the write-up.

## Launch record

Filled in at launch, under a dated heading, after the hash.

## Limits

20 seats per arm in 5 matches per arm; four small models; one output ceiling; a router key's serving mix; self-reports in the debrief. An exploratory Studio experiment with its design declared before the trials, not a declared evaluation.

## Launch record (20:13 UTC 2026-10-07)

Hashed 20:12 UTC: SHA-256 c48d9fdd8a6c80fdb962dcaefb573e143e404ce7834554b3a78c8c27460daccb (data/design-hashes.txt; as-hashed copy data/DESIGN-3-thinking-off-vs-high.as-hashed.md). Created 2026-10-07T20:13:03Z as `Thinking off vs high, 5 pairs c48d9fdd8a6c`, experiment `exp_c80034898dd738e29f`, status queued. Platform estimate at launch: off arm $6.98 to $11.21, high arm $6.98 to $28.55 (5 trials each). OpenRouter balance read by the platform at launch $51.28; account meter before launch $48.70 (studies 1 and 2 included, study 2 one run from finished). Request body in data/launch-3.json.

## Amendment (20:30 UTC 2026-10-07), before any scored trial

The first launch (`exp_c80034898dd738e29f`, 20:13 UTC, hash c48d9fdd8a6c) was cancelled at 20:27 UTC with no trial finished and no pair scored. In every control-arm match the GPT-5 mini seat made zero calls: its route answered `reasoning: off` with HTTP 400 "Reasoning is mandatory for this endpoint and cannot be disabled", five times at round 0, and the seat exited (data/cancel-3.json; the trajectory header shows `reasoning_effort: off`). A trial that loses a seat to an error leaves every pair, so the design as hashed could not produce a result. The three other models accepted `off` (58 to 68 calls each by round 60), and all four accepted `high`.

Change: the control arm sets GPT-5 mini to `low`, the lowest level the platform offers and its route accepts; the other three seats stay at `off`. The arm keeps the label `thinking off`; GPT-5 mini's rows are read as low against high and say so. Nothing else changes: same hypotheses, tests, rule, pairs, caps, ceiling and clock. Spend lost on the cancelled launch is recorded in the launch record below. The file is re-hashed after this amendment; the new experiment name carries the new hash.