Does the high-reasoning Index gain hold under scarcity? Study 3's design on the Famine scenario, 16 pairs
Fixed 22:55 UTC 10-07 on the owner's word ("justify and proceed with your recommendation if your assessment is still valid", 22:46 UTC) after his rule that experiments are sized for the question, with cost stated as a consequence, not designed around credits. Studio paired experiment, production daishi.ai (harness 0.6.4, rubric 2.3), account founder, published under /u/labrobot. Hashed before any trial; the first 12 hex of this file's SHA-256 are in the experiment name. Later edits only under dated headings.
Why this study, and why now
Every published run on /u/labrobot (76 before today's replication) played one scenario, Custom World (daishi:custom-base-v1) with the v2 house bots. Study 3 read a +9.0 Index gain for reasoning high as a secondary outcome, and its decomposition (data/decompose-3.txt) put the whole gain in adaptation and economy, with survival contributing nothing: in Custom World no active model seat starved. Study 4 (exp_c4addab02aab3dab06, running) tests that gain as a declared primary on fresh Custom World worlds. This study asks the mission's question: does the gain hold when the conditions change, in a world where survival is live. It is a generalization test, not a replication; its interpretation is declared below as a matrix against study 4's verdict, so that neither result is read in light of the other after the fact.
Launched before study 4 reads out, by design: declaring the matrix now means no reading of study 4 shapes this design.
The scenario, and why 100 rounds
daishi:famine-v1 (library; hash recorded at launch): food pools at a quarter of their normal size and a quarter of their regeneration. The world's metabolism (src/domain/constants.ts): a seat starts at 100 energy, loses 1 a round passively plus 2 to 10 per action (move 3, gather 2, build 6, attack 10), and one food restores 5. A seat that acts every round burns about 4 energy a round, so without food it is dormant inside about 25 to 35 rounds. A normal plains pool holds 60 food and regrows 1 a round; under famine it holds 15 and regrows 0.25 a round, which is 1.25 energy a round against a seat's 4. One famine pool therefore sustains about a third of an active seat, and nine seats share a 12 x 12 world. Scarcity bites in the first 30 to 50 rounds of a 100-round season; the scenario's 1,000-round default is for leaderboard seasons, not for this question. 100 rounds also keeps this study comparable with studies 3 and 4 (100 rounds each), so the only change is the world's food.
The scenario draws its worlds from the platform's practice pool of eight; trial i plays world i, and the worlds come round again from trial 9. 16 pairs plays each of the eight worlds twice (a balanced block design); pinned is refused for pool scenarios. The scenario ships four house bots; env.anchors: ["anchors-v2"] fields the same five bots as studies 1 to 4 so the bots are not a second change.
What is measured
- Fitness Index (rubric 2.3), primary. Not a validated measure of general capability, and gameable. Arm value of a trial: mean over the four model seats, dormant included.
- Survived, secondary: seats alive at the end, by arm. Under famine this is expected to be live.
- Output tokens per play turn, secondary: the check that the setting took effect as in studies 3 and 4.
- Descriptive: the Index by rubric dimension (survival, economy, social, adaptation, competitiveness), paired high minus off over seats active in both arms; the round each seat went dormant; food gathered, eaten and traded and shelters built per seat from the archive's per-agent event counts; spend; forfeits; the debrief; the serving mix.
Hypotheses
H1 (primary). With reasoning set to high, the four models score a higher Fitness Index under famine than with it off (GPT-5 mini: low, its route's floor), on the same worlds. Test: the platform's declared outcome as the metric, fitness_index, fixed rule, one look after trial 16, 95% (bootstrap and t). Passes on arm_better. Two-sided. Prediction: none committed on the sign. Study 3's gain came from doing more economy and adaptation work; scarcity may reward that (planning food, sheltering, trading for calories) or erase it (nothing to gather, everyone starves, the Index compresses). The matrix below is the prediction. Size: 16 pairs detect a gap of about 3.3 points at study 3's paired spread (sd 4.4) and about 5.5 points at study 1's (sd 7.3), 80% power, 95% two-sided. The spread under famine is unknown and may be larger because survival is bimodal; 16 pairs rather than 8 is the hedge against that, and it is the smallest balanced size above 8 on an eight-world pool.
Secondary, declared. survived (two-sided; no prediction) and output_tokens_per_turn (expected: a replication of study 3's about 5.6 times, the sanity check that the setting took).
Interpretation matrix, declared now, read with study 4's H1 verdict
| Study 4 (Custom World) | This study (Famine) | Reading |
|---|---|---|
| above zero | above zero | the gain holds across two scenarios: a declared result about these models, both scenarios, this ceiling and clock |
| above zero | spans zero | the gain is scenario-bound to Custom World at this size; no claim about scarcity |
| spans zero | above zero | a scarcity-specific gain: a new hypothesis, not a confirmation of study 3 |
| spans zero | spans zero | the study 3 reading is retired at this size |
| either | below zero | a reversal under scarcity; reported with the dimension split and the dormancy rounds, no claim |
Rules carried over: a secondary reading decides nothing on its own; the dimension split and the active-only paired difference are read beside the verdict; a model whose output per turn is about the same in both arms did not honor the setting and its rows say so; platform exclusions as in studies 1 to 4 (a trial that loses a seat to an error or a cap leaves every pair; seats missing more than a tenth of their rounds are excluded from effort). Study 3's 5 pairs are quoted beside, never pooled. No sentence ranks a model or a vendor; "more reasoning" is never "smarter".
Design
| Item | Setting |
|---|---|
| Scenario | daishi:famine-v1 (library; content hash in the launch record), season_ticks: 100, turn-based, 1 action per round, 180 s turn timeout on both arms (env.turn_timeout_ms) |
| House bots | env.anchors: ["anchors-v2"], the five v2 bots of studies 1 to 4 |
| Roster, both arms | Claude Haiku 4.5, GPT-5 mini, Gemini 2.5 Flash, DeepSeek V3.2, canonical prompt, max_tokens: 4096 on every seat |
| Arms | control thinking off (reasoning: off on Claude Haiku 4.5, Gemini 2.5 Flash and DeepSeek V3.2; reasoning: low on GPT-5 mini), treatment thinking high (reasoning: high on every seat). The setting is the only difference |
| Trials | 16 per arm, seed_mode: vary (the pool's eight worlds, each twice) |
| Metric and rule | fitness_index, stopping: fixed |
| Secondary outcomes | survived, output_tokens_per_turn |
| Caps | max_spend_usd off arm $2.50, high arm $6.00, as in studies 3 and 4 |
| Name | Thinking under famine, 16 pairs <first 12 hex of this file's SHA-256> |
Cost, stated as a consequence
| Item | Figure |
|---|---|
| Per pair, working figure | $3.23 (studies 3 and 4's Custom World pairs: off $1.10 + high $2.13 a match); famine seats that go dormant early make fewer play turns, so this is likely an overestimate |
| 16 pairs, realistic | about $52 |
| Worst case under the caps | $136 (16 x $8.50) |
| Platform estimate at pricing | off arm $22.85 to $48.00, high arm $22.85 to $96.11 (16 trials each); platform precision line: half-width 2.9 points at a reference spread of 6 |
| Hot case (every match at the dearest seen in study 3, off $1.37 and high $2.60) | $64 |
| Balance | $92.30 at 22:50 UTC with study 4 still drawing (about $25 realistic, $35 hot, $75 under its caps). Launching this study while study 4 plays with only the realistic margin risks a provider 402 that fails matches in both studies, so the launch rule below applies |
| Launch rule | launch when the balance leaves at least $70 for this study beyond what study 4 can still draw: at once if the balance reads $130 or more (a top-up of about $40), otherwise when study 4 ends and the balance reads $70 or more; if neither, the owner is told the shortfall and nothing launches. The design is not shrunk to fit |
Analysis, frozen
scripts/analyze_notice.py <exp_id> for the metric and secondary readings, scripts/decompose3.py <exp_id> for the dimension split and action mix, scripts/effort3.py <exp_id> for the effort blocks, scripts/forfeits_archive.py <exp_id> 5 for forfeits, scripts/traj_check.py for the transcript spot-check; dormancy rounds and food counts from the archive's per-agent metrics. The matrix above is applied once, with study 4's verdict, in the results file.
Limits
64 seats per arm in 16 matches per arm on eight worlds; four small models; one output ceiling; two scenarios in total across studies 4 and 5, both with the same five bots; a router key's serving mix; self-reports in the debrief. An exploratory Studio experiment with its design declared before the trials, not a declared evaluation on held-out worlds and opponents.
Launch record
Filled in at launch, under a dated heading, after the hash.
Amendment (23:00 UTC 2026-10-07), before launch, from the post-study review thread
Nothing launched yet (the launch rule above had not been met); the file is re-hashed after this amendment and the experiment name carries the new hash.
1. World-level reading, added to H1. 16 pairs on an eight-world pool put two pairs on each world, and the platform's fixed rule treats every pair as independent (src/eval/sequential.ts, pair differences, no clustering), so a world-by-setting interaction would be counted twice. H1 is read as supported only when both clear: the platform's verdict (arm_better, as declared) and a world-level 95% interval, computed by averaging the two pair differences on each world and taking a Student t interval over the 8 world means (7 df), with an exact sign-flip test over the 8 world means (256 assignments) reported beside it. The platform's own cluster-robust interval (src/eval/stats.ts clusteredTInterval, world as the cluster) is quoted where the report exposes it. Size, corrected: the detectable gap at sd 4.4 is 3.3 points only if the effect does not vary by world; with world-by-setting variation it rises toward 5.1 (the 8-pair figure). 3.3 is the best case. 2. The effort check may rest on few seats. A seat that goes dormant keeps receiving calls for a while and then stops at an uneven point (study 3: a Haiku seat dormant at round 26 was called to 86, a DeepSeek seat dormant at 13 to 74), so it misses more than a tenth of its rounds and leaves the output-per-turn comparison in every arm. Under famine that could thin the manipulation check to a few seats, and dormant-phase calls cost money both arms pay alike. Added to the descriptive block: output per play turn over the rounds before each seat's first dormancy, per model and arm, from the run logs. Added to Limits. 3. Trained levels per seat and arm, added to the descriptive block. The rubric pays a trained level twice (12 adaptation points per level, src/domain/scoring.ts, and 5 wealth, FITNESS_VALUE_PER_LEVEL) and training costs food (TRAIN_BASE_FOOD_COST 2 per level times the next level). In study 3 the training term was about +2.3 of the +7.6 points among seats active in both arms (GPT-5 mini 3.0 levels at high against 0.2 at low; Gemini 1.2 against 0; Haiku 0.4 against 0; DeepSeek 0 in both). Under famine the same habit costs calories, so trained levels are a likely place for the two scenarios to diverge; counted from the archive's train events per seat. 4. Awareness scan. The platform's scan reads text and reasoning, and the high arm writes several times as much, so a count of flagged seats is confounded by exposure; it is reported with the output volume beside it, never as a rate of belief.
Amendment 2 (00:58 UTC 2026-10-08), after study 4 read out and before any famine data, from the post-study review of study 4
Nothing launched (balance $69.53 against the $70 rule; the owner's pick is pending). Study 4's H1 verdict is above zero (+3.98, 95% +2.14 to +5.81, 9 of 10 worlds), so rows 1, 2 and 5 of the matrix are the live ones. This amendment is made knowing study 4's result and knowing nothing about famine; every change makes a claim harder to reach, never easier. The file is re-hashed after it and the experiment name carries the new hash.
1. Matrix row 2 replaced. "Study 4 above zero, famine spans zero" cannot show that the scenarios differ: an interval that clears zero beside one that does not is not a difference (famine could read +2.5, -0.5 to +5.5, on an effect identical to study 4's). Declared now: the between-study difference, famine minus Custom World, as a Welch 95% interval on this study's 8 world means (amendment 1) against study 4's 10 pair differences (data/training-4-platform.txt: 4.13, 3.58, 6.77, 1.70, 5.95, -0.43, 6.15, 1.58, 7.35, 3.00). Row 2 now reads: "scenario-bound to Custom World" only when that interval excludes zero; otherwise "the gain is not shown under famine at this size, and the two scenarios are not shown to differ". The difference interval is reported beside rows 1 and 5 as well. 2. Dormancy reading, method fixed. Dormancy rounds come from each seat's own status (observation.your_status.status) on every turn of its trajectory, counting every dormant turn, episode and recovery (scripts/predorm4.py, rewritten 00:52 UTC), never from text in a turn's result (a look lists neighbors and house bots with their status, which the earlier version mistook for the seat's own; study 4: 10 of 19 dormancy episodes ended in a recovery). The archive's agent_dormant and agent_awake event counts are quoted beside. Under famine dormant neighbors will be common, so this is the only reading that is safe. 3. Awareness scan, exposure stated in the unit the scan reads. The scan reads returned text plus returned reasoning text, and how much reasoning text a route returns varies by model, so output tokens are the wrong proxy. Report characters scanned and flagged turns per million characters, per model and arm (study 4: 2.2M characters off against 10.6M high; 0.9 against 4.9 flags per million; 36 of the high arm's 52 flags were one model's reasoning). Never a rate of belief. 4. Training term, computed with the platform's scorer. Each arm's Index without the training term is recomputed with the platform's own scoring code (dist/eval/archivescore.js and dist/domain/scoring.js at the production commit; the review thread's scripts/s4-rescore-training.mjs, run here: data/training-4-platform.txt), with trained levels zeroed everywhere the rubric pays them (12 adaptation points and 5 wealth per level, wealth ranks recomputed). It is an accounting split, not a counterfactual: a seat that had not trained would have kept the food and the turn. Study 4: +2.51 (+0.39 to +4.64), 7 of 10 pairs positive; study 3: +6.73 (+1.07 to +12.39). 5. Cost line corrected. "Famine seats that go dormant early make fewer play turns, so this is likely an overestimate" is withdrawn: in study 4 dormant seats were still called 60 to 100 times each, 6.3% of all turns (495 of 7,867) were played while dormant at $3.31 of $31.96, and a dormant Haiku seat on high cost $1.12 against $0.88 for an awake one. The planning figure for this study is the hot case, $64, not the $52 realistic figure. The launch rule above is unchanged; at this amendment the balance ($69.53) covers the hot case with $5.53 to spare and is 47 cents under the rule, which is the owner's call. 6. Comparability hold. The matrix compares this study with study 4, so no harness or rubric change (the dormant-seat call fix included) should merge before this study reads out; display-only changes are fine.