Does Claude Haiku 5.5 score higher the more it thinks, on harness 0.6.10?
Question
On harness 0.6.10, which shows an agent the whole result of a lookup once instead of its first 300 characters, does Claude Haiku 5.5 score higher on the Fitness Index the more it thinks? And how does the thinking setting change the way it spends its turns: acting, resting, or looking things up?
Why
- Earlier reads. On 14 October 2026, in this scenario, Haiku 5.5's own high effort moved its Index +0.27 points (95% −2.11 to +2.65). Haiku 5.5 spent 52 to 55% of its turns on free lookups at high, against 25% at low. An earlier study on 12 October, where both models' effort moved together, read Haiku 5.5's row at −7.3.
- Lookups cost a turn. Every lookup costs the agent its turn.
- The cut. Through harness 0.6.9 a lookup showed only the first 300 characters of its result. The scoring text runs to about 4,600.
- The fix, and what it leaves. Harness 0.6.10 shows the whole result once. Each call is still a single message: the last 15 actions at 300 characters each, the seat's own visible text from its previous turn (up to 1,500 characters), and that last result once. Thinking summaries are never shown back. So a seat that read the whole rules does not have them two turns later unless it copied them into its own text.
- What this design can and cannot say. It describes how the thinking setting changes play on that harness. No arm switches the cut on or off, so it cannot test whether the 300-character cut caused the earlier habit.
- Why the score is split. The scoring text now reaches every seat that asks for it, and it names which actions earn points. So this design also splits the score by its parts.
Setup
- Scenario. open-play-v1: the 8 practice maps and five house bots (random, greedy harvester, honest trader, aggressive raider, envoy). 100 rounds, one action a turn, a 180-second turn clock.
- Roster. Four Claude Haiku 5.5 seats on the native Anthropic route, with an output ceiling of 8,192 tokens and native tool calls.
- Arms. Only the thinking setting differs, and it is the same for every seat in an arm.
- Control "thinking low": adaptive thinking, effort low.
- "Thinking off": thinking disabled.
- "Thinking high": adaptive thinking, effort high.
- What comes with the setting. A seat with thinking off can plan only in visible text, and that text rides into its next observation. A seat that thinks and writes nothing carries nothing forward. In the pilots, off seats carried a note into 396 of 400 play turns (median 190 characters), against 49 of 400 at low and 35 of 400 at high. H1 and H2 read this as part of the setting's effect. It is also a rival pathway for every mechanism read below.
- Versions. Harness 0.6.10 (prompt v6), engine 0.3.3, rubric 2.3.
- Trials. 32 per arm. Each practice map is played 4 times, each time from a new spawn layout, and every arm plays the same layout on each trial.
- Seeds. Daishi draws them from the first public drand round after creation. Nobody picks the layouts.
- Pilots. One match per arm was played on 6 November 2026, before this design was locked, and declared as a pilot.
- The hypotheses and predictions below were written before the pilots and were not changed.
- The pilots set only the per-match spend caps.
- The reads under "Where the score comes from" were asked for in review before any pilot result existed.
- The carried-note count under "Mechanism read" was asked for in review while the pilots played, and was first read on them.
- The ceiling-cut count beside H1 and H2 was asked for in review after the pilots.
- The off pilot confirmed the setting took: no request field was dropped, and output ran 113 to 124 tokens a play turn, against 231 to 241 at low.
Decision rule, fixed before the first trial
- Metric. The Fitness Index, as the mean over the four seats in each trial. Every seat changes, so every seat is read.
- One look, after trial 32. Each treatment arm is compared with the control.
- The 0.05 level is split across the two comparisons, 0.025 each.
- An arm is ahead only when both the paired t interval and the bootstrap interval clear zero.
- Reported beside it: the map-level interval (8 maps, 7 degrees of freedom) and the exact sign-flip test over the maps. A claim beyond these 8 maps uses the map level.
- Loss tolerance. 3 trials per comparison. Past it, there is no verdict, and no trial is replaced.
- Secondary outcomes, read on the same pairs and stopping nothing:
- output per play turn, the check that the setting took (high above low above off);
- survival;
- messages per 100 rounds.
- Reported beside H1 and H2: turns cut at the 8,192-token output ceiling, per arm. Each cut turn is a lost action in its arm.
Hypotheses, predictions and surprise
- H1 (primary): thinking high against low.
- Prediction: high ahead by 4 points, a hit between +1 and +7.
- The +4 is a bet that the 0.6.10 change matters. Both earlier reads (−7.3 and +0.27) were lower.
- An interval that reaches zero reads Undecided: an effect inside it is neither shown nor ruled out.
- H2: thinking off against low.
- Prediction: off 2 points below low, a hit between −5 and +1.
- Surprise. High scores below low, with its interval below zero. We put that at about 1 in 8.
Where the score comes from (declared now, described without a test)
- The five parts. H1's and H2's gaps, each split into the five weighted dimension terms of the Index.
- Counts per seat and arm. Trained levels, messages sent, and completed trades.
- Rescoring. H1 is rescored twice in the platform's own scorer: once with trained levels set to zero, which removes both their adaptation points and the wealth they add, and once without the message term. These are accounting splits, not counterfactuals.
- Exposure. For each seat, whether and from which round its observation carried the whole scoring text, by arm.
- Reading rule. Suppose H1 clears zero and most of its weighted gap sits in the training and message terms. Then the reading is "high effort scored higher mostly through credits the scoring text names", not "it played the world better".
Mechanism read (declared, described)
- What is counted, per seat and arm, as shares of turns:
- scoring lookups;
- world lookups;
- chosen rests;
- looks;
- world actions;
- epilogue writes.
- The lookup share that the prediction and the 45% mark below use is every free-tool turn: scoring lookups, world lookups and epilogue writes, as 14 October counted its 52 to 55%. Scoring and world lookups are also shown on their own.
- Re-reads. Scoring-text calls after the seat's first whole read, and the median rounds between two such calls.
- Carried notes. Per arm, the share of play turns whose observation carried the seat's own text from its previous turn, and that text's median length.
- Prediction (written before the pilots): free lookups under 30% of turns at high, against 52 to 55% on 14 October.
- What a share can say.
- A share of 45% or more at high would show that the 300-character cut was not needed for the habit.
- A share below that says nothing about the cut, since the roster, the harness and the engine have all changed since 14 October.
- A habit that persists on 0.6.10 can be re-reading for lack of memory rather than truncation. The re-read count and the rounds between reads describe that.
- An off arm that scores near low may be planning in carried notes. That would not show that thinking does not matter.
Read beside the 14 October study (descriptive, never pooled)
- A Welch interval between this study's high-minus-low and that study's H1.
- The two differ in all of these:
- The harness. It was 0.6.7 there and 0.6.10 here, three versions apart: status lists own structures, offer notices appear, and the whole result shows once.
- The engine. 0.3.1 there, 0.3.3 here.
- The roster. There, two Haiku 5.5 seats played beside two Claude Sonnet 5.5 seats. Here, four Haiku 5.5 seats play.
- Spawn layouts, which every experiment draws fresh.
Raider read, declared
- On 14 October every early dormancy fell on the aggressive raider's first target.
- For each trial, the slot the raider attacks first in any arm is named. H1 is also described with that slot dropped from every arm of its trial.
Sizing
- Pair level. From the 14 October study's paired spread (SD 5.24 on its four-seat mean, where only the two Haiku 5.5 seats changed), 32 trials detect a gap of about 3.0 points at 80% power.
- Here every seat changes. If the four seats move independently, the four-seat SD is about 5.7 to 7.4, which is inferred, not measured. At 7.4 the gap is about 4.2.
- Power. At +4 it is 0.97 at SD 5.24 and 0.75 at SD 7.42. At +2 it is 0.43 and 0.22.
- Map level. On 14 October the two trials on a map barely agreed, and on that evidence the gap is about 3.5 to 4.9. The worst case, where the effect differs wholly by map, is 6.9 to 9.8.
- What that means. H1 may well read Undecided. Much of this design's value sits in H2 and the described reads.
What it cannot show
- Anything past one model, one scenario, these five house bots, 100 rounds and one harness version.
- Why a seat chose what it chose: thinking summaries are the vendor's summaries, not the reasoning itself.
- Capability in general: the Fitness Index is not a validated measure of capability, and it can be gamed.
- Other routes: an effort label maps to different mechanisms there.
- Anything about other vendors' models.