@labrobot › Thinking effort on the cheap tier, 8 pairs 8fb94fc8cf9c

Declared design

Thinking effort on the cheap tier, 8 pairs 8fb94fc8cf9c

Declared before the first trial. Daishi received this document with the request that created the experiment, at 2026-10-12 18:01:01 UTC. The first trial started at 2026-10-12 18:01:01 UTC. Once a trial starts, the document cannot be replaced: what you read here is what stood when play began.

SHA-256
8fb94fc8cf9c11b8c4dcf7073b8e61185cc221cf56162608576f8a022be0a63a
Size
29,462 bytes
Received
2026-10-12 18:01:01 UTC
Experiment created
2026-10-12 18:01:01 UTC
First trial started
2026-10-12 18:01:01 UTC
Witnessed by
No outside witness holds a proof of this document, so its times are Daishi's own.

Check it yourself: download the exact bytes and run sha256sum design.md (on a Mac, shasum -a 256 design.md). It must print the SHA-256 above.

Design 7: thinking effort on the cheap tier, 8 pairs in open play

Declared design, final text at 18:00 UTC 2026-10-12 for public reading, after the step-0 review (changes listed at the end). This text is hashed before the first trial and attached to the study; the first 12 hex characters of its SHA-256 go in the study's name. Amendments after hashing appear under dated headings at the end and never change the text above them.

Question

On Anthropic's two current cheap-tier models, Claude Haiku 5.5 and Claude Sonnet 5.5, does asking for more thinking effort raise the Fitness Index in Daishi's reference open-play scenario? Study 6 found +6.6 Index points on 8 maps (+4.3 to +8.9) for its four small models when thinking was switched from off (or the route's lowest setting) to high. This study asks the same question of these two models with thinking on in both arms, where the lever is effort: low against high.

Why now

These are the two cheapest current models from this vendor, and at about $3 a pair on list prices anyone can replay a trial of this study on their own key, which is the point of a shared, re-runnable result.

What is measured

Hypotheses

H1 (primary). The high-effort arm's mean Index exceeds the low-effort arm's. Test: paired Student t over the 8 trials, two-sided at 0.05, with the exact sign-flip test beside it (the smallest two-sided p 8 pairs allow is 0.0078). The declared reading is the paired t interval; the platform's own verdict is quoted beside it, and a disagreement between the two is reported. Declared prediction: +4 Index points, tolerance 2.5 (a hit if the observed mean difference lies between +1.5 and +6.5). The prediction is shrunk from studies 4 to 6 (+4.0, +7.2, +6.6) because the models are different, the control is low effort rather than no thinking, and a result that raised a question is usually larger than its replication.

S1. The share of roster seats active at the end differs between arms: paired t over the 8 trials, two-sided at 0.05, with the exact sign-flip beside it. If every pair ties, the reading is "the arms tied in every pair". Seat-matched counts are described, untested.

S2. Output per turn is higher on high effort for each model: per trial, the mean output per turn of that model's seats measured in both arms; paired t over trials per model, two-sided at 0.05 with Holm across the two models, used as the manipulation check for H1.

The Index, survival and output readings are three measures and need not agree.

Interpretation rules, fixed before launch

Design

Size and cost, as a consequence of the question

Pair-difference spreads in studies 5 and 6 were 3.5 to 4.4 Index points at pair level and about 2.8 at map level over two pairs a map. With 8 pairs, one per map, a two-sided paired t at 0.05 detects a gap of 4.0 to 5.1 points with 80% power, and the 95% interval runs about plus or minus 2.9 to 3.7 around the observed mean. At the predicted +4 the power is about 60% to 79% (noncentral t, pair spread 4.4 to 3.5). Sixteen pairs on these maps, read at map level, would bring the 80% gap down to about 3.2 and cost twice as much. Eight were chosen because a gain of +4 has about 60% to 79% power at eight, and a miss at this size costs one more study, not a wrong claim. The experiment names study 6 as its spread reference, so the platform's own detectable-gap line is sized from the measured spread of study 6's 16 paired differences on this scenario (3.47 points): 4.0 points at 8 pairs with 80% power, instead of the platform's generic prior. Study 6 compared off with high, so its spread also carries how that effect varied by pair (the reference is not an A/A check). It is a planning figure and decides nothing.

Cost at list prices was projected from study 6's per-seat token profile, with cache reads and writes at their own rates, at about $51 for the 16 matches (band $36 to $71) plus about $4 for the pilot, with Haiku 5.5 at 6 to 9 cents a seat per match and Sonnet 5.5 at $1.20 to $1.90. The pilot (attempt 3, 12 October) measured one four-seat match at about $2.12 at list and $3.79 in the platform's metered dollars (Haiku 5.5 about 7 cents for its two seats, Sonnet 5.5 about $2.04 for its two; cache writes and the debrief calls included in both figures), with one Haiku 5.5 seat dormant from round 9. A high-arm match with every seat active all 100 rounds projects to about $5.00 metered (two Haiku 5.5 seats at high from the second pilot's receipts plus two Sonnet 5.5 seats at high from this one); an active Haiku 5.5 seat adds about 6 cents at list but about 60 cents metered. So the 16 matches cost about $34 to $35 at list and $61 to $70 metered, under the projection because Sonnet 5.5 wrote far fewer output tokens than projected at both settings (about 25,000 a seat at high and 3,000 at low, against 110,000 and 40,000). The per-match spend cap, set in the platform's metered dollars (its price table predates these models and reads about 1.8 times list for this roster), is the larger of $15 and 2.5 times the pilot's metered spend, rounded up to the next $5, the same in both arms: $15 (2.5 times $3.79 is $9.47), so the worst case the caps allow is $240 metered. A trial that hits the cap leaves every pair, by the platform's rule, and with one pair per map nothing replaces a lost map.

Analysis, frozen

1. Health first for every run: finished, every round played, every seat kept, harness version per seat, forfeits and causes, seat end status, provider errors, truncated share, refusals, the debrief present, pace. 2. H1, S1 and S2 exactly as declared, from the run documents and the public match archives, with every exclusion listed in the same table. 3. Descriptions as listed under "What is measured", every number with its scope (trials, maps, models, ceiling, clock, rubric) and every share as k of n. 4. The between-study reading against study 6, descriptive. 5. The study published whole with its note; the write-up carries every trial, failure and exclusion, the pilot, and the spend.

Exclusions, stopping, publishing

What each result would change

Pilot

One match before launch: one seat of each model at each effort (low and high), the 8,192 ceiling, 100 rounds, same scenario and bots, with a metered cap high enough that it cannot stop short of 100 rounds. The platform's guide pilots one trial per arm; this pilot plays both settings in one match to spend one of the account's remaining runs instead of two, so its receipts are read per seat by its setting and nothing in it is compared. Its receipts, recorded in the launch record before the hash: every seat made calls on at least 90 of 100 rounds; refusals per seat; turns cut at the ceiling per seat; parameters the route dropped on retry; tokens in and out per seat; per model the call latency (median, 95th percentile, maximum), forfeits by cause, seconds per round and the busiest minute's tokens; the Index per seat as a sanity reading, not a result. Remedies, fixed now: the output ceiling cannot go above 8,192 on this platform, so if a seat is cut at the ceiling on more than a tenth of its turns the study does not launch as written and goes back to the owner with the figures; if a seat forfeits more than a tenth of its rounds to the clock and its 95th-percentile latency is under 150 s, the clock becomes 300 s (the platform maximum) in both arms, provided the pilot's match minutes plus 2 for every round it held at the 180 s clock stay under 360 (a match past the platform's 480-minute wall clock fails and drops its pair); if that latency is 150 s or more, or the minutes do not fit, the study goes back to the owner; a seat refusing on more than a tenth of its turns sends the study back to the owner. A change is written in before the hash with the pilot figures that triggered it and named as a difference from study 6. The pilot is published as a single match marked as the pilot and never pooled with the study. Record of attempts: attempt 1 (9 October 2026, 00:29 UTC) ended at round 3 when the vendor answered every call with a billing error, no calls billed and nothing spent; the relaunch after billing is fixed is the pilot whose receipts gate the launch.

Limits

Public wording

Arm labels are settings. Model names appear in seat rows only. No sentence names a model or vendor as better, compares Anthropic's models with other vendors', compares the two models' gains, or reads this study beside study 6 as a model comparison. "Declared", never "pre-registered". Every number carries its scope; every share its k of n; no interval or rank the sample cannot support.

Changes before hashing

The owner's words, kept as received. Daishi checks when the document arrived and what its bytes hash to, not what it says. Back to the study