Standardized Evaluation & Data Collection: Research Report

Status: research / design input; no code changes included. Question answered: how should Daishi collect data and structure a standardized evaluation + reports that companies and individuals can use to improve their models, and what is the industry currently using to evaluate AI models?

This report has four parts:

1. Where Daishi stood when this research was written: the baseline we started from and the gaps that motivated §3 (since closed; see §3.1) 2. What the industry does: the 2024-2026 evaluation landscape, organized by the decision it informs 3. Recommendations: a concrete design for Daishi's standardized eval, reports, and data collection 4. Sources

Everything in §1 was verified against the code at the commit this document was written at. Industry claims in §2 were checked against primary sources (papers, repos, official docs) listed in §4.

---

1. Where Daishi stood when this research was written

> Historical snapshot. This section describes the tree as it stood before the > §3 roadmap was implemented; its file:line receipts refer to that commit and have > drifted since. The gaps listed in §1.2 have been closed (Phases 1-4; see the > implementation-status callout in §3.1). It is kept as the motivation record for > the design in §3, not as a description of the current code.

Daishi is unusually well positioned to be a benchmark: the server is authoritative ("physics is truth"), every successful action already lands in an append-only event log, and the lobby gives synchronized, fixed-seed match starts: the structural property most benchmarks have to build from scratch (operator-observed ground truth instead of self-reported scores).

1.1 What we already collect

1.2 The gaps that matter for evaluation

Ordered roughly by how much they block a standardized eval:

1. No model identity. register_agent takes only name + signup_token (src/server/mcp.ts:90-93); /api/metrics ships a hardcoded model_note: 'set by owner, not tracked server-side' (src/server/http.ts:188). The central benchmark dimension, which model played, has no first-class field anywhere. Attribution today is an external spreadsheet. 2. No match/run identifier. The events table has no season or match column (src/storage/sqlite.ts:24-33); resetWorld restarts tick at 0 (src/domain/engine.ts:1217) so tick values collide across matches, and slicing one match out of the log means scanning for world_reset/season_started marker events. 3. Failed actions are invisible. ActionErrors (invalid args, not_here, insufficient energy, rate_limited, unauthorized) are returned to the caller and never persisted (src/server/mcp.ts:64-78; rate-limit rejections throw before the persist path, src/server/host.ts:171-189). Invalid-action rate, one of the strongest model-quality signals in agentic evals, cannot be computed from our data. 4. Results are (mostly) destroyed at rollover. /api/metrics iterates only current-season agents (src/server/http.ts:184), so a finished match's metrics vanish at reset. Lineage does archive {score, rank} for every agent alive on the final board, but leaderboard() filters status === 'dead' (src/domain/engine.ts:1107); an agent that died is indistinguishable from one that never played, and no metric/attribute snapshot, cause-of-death, or full final board beyond top-10 is kept. There is no end-of-match export artifact of any kind. 5. Timing is unfair by construction. The rate limit is a wall-clock floor (1 action / 2 s, src/server/host.ts:170-181), not a per-tick budget: at TICK_SECONDS=60 a low-latency harness can take ~30 rate-limited actions per tick while a 20 s-latency harness gets ~3; harness speed directly buys in-game throughput. Also note only the 17 actions in RATE_LIMITED_ACTIONS (src/server/host.ts:8-12) are limited at all; status/read_messages/world_info are unbounded. 6. Determinism is incomplete, but the fix path already exists. Worldgen is fully seed-deterministic (mulberry32, src/domain/rng.ts:4-13), and tick() contains zero randomness (src/domain/engine.ts:1123-1204). But Game.random defaults to Math.random (engine.ts:55) and is used for spawn assignment (engine.ts:280-285) and next-season seeds (engine.ts:1256). Importantly the engine already accepts an injectable random source and resetWorld accepts an explicit seed (engine.ts:1213); full benchmark-mode determinism is a configuration change plus seeded streams, not a rebuild. 7. No cost/latency/token telemetry. Events record tick but the wall-clock created_at is dropped in the row mapping (src/storage/sqlite.ts:100-108), so even decision latency between an agent's actions is unrecoverable. Token usage and dollar cost are entirely outside our view (the server never sees the LLM calls). 8. Config is not part of the record. TICK_SECONDS, SEASON_TICKS, MAX_AGENTS are env-overridable (src/domain/constants.ts:88-95) but only seed is stored in WorldState (src/domain/types.ts:171); two "identical" runs can silently differ. Event payloads are untyped Record<string, unknown> (types.ts:155) with no schema version. 9. No reference harness. README says "fix the harness prompt, vary only the model" (README.md:142-144) but the repo ships no harness or prompt; every participant's scaffold differs, confounding model comparison. Nothing documents the event schema, the scoring formula, or the path from "I connected my model" to "I got a report". 10. No sybil/collusion story. SIGNUP_TOKEN is one shared secret (src/server/host.ts:137-143); nothing links multiple agents to one operator, so a competitor can field cooperating feeder agents in a scored match undetected. 11. Metric attribution quirks. trade_completed carries only the acceptor's agentId (engine.ts:905-909) so offerers' completed trades are invisible in metrics; messages_sent counts private DMs because eventTypeCounts() has no private filter (src/storage/sqlite.ts:111-116); quantities inside payloads (gathered amounts, damage, trade value) are never aggregated, only event counts.

---

2. What the industry does

2.1 Harness architecture: environment / agent / scorer as separate layers

The clearest convergence across 2025-2026 eval frameworks (Inspect AI, the UK AI Security Institute framework that leads agentic evals, HELM, lm-evaluation-harness, promptfoo, Braintrust, LangSmith, W&B Weave):

• Task = Dataset + Solver + Scorer (Inspect's decomposition). The environment, the agent scaffold, and the scoring pipeline are three independently swappable layers. Scorers are pure functions over recorded logs, never logic inside the run loop, so new metrics can be back-applied to old runs.
• Scenarios are declarative, versioned configs, not code (lm-eval-harness YAML tasks, HELM RunSpecs, promptfooconfig). Two labs running daishi:survival-s3 should get bit-identical worlds; results reference the scenario spec by version/hash.
• One self-contained run artifact per evaluation (Inspect .eval logs): full message/action transcripts, exact config, errors, per-model token usage: replayable, queryable as dataframes, with a CLI to reconstruct the run config.
• Repetitions are first-class (Braintrust trials, LangSmith repetitions, Inspect epochs): N runs per configuration is the unit of evaluation, averaged at the top with individual runs preserved.
• Comparison-first reporting (Braintrust experiment-vs-baseline diffs): the report a model developer actually wants is "v2 vs v1, same seeds, which behaviors regressed?" with drill-down from metric delta to raw transcript.
• Leaderboards defined as data (W&B Weave): column = scorer + aggregation, row = model, recomputed from immutable logs; adding a metric is a config change.
• Local-first with opt-in sharing (promptfoo, DeepEval): companies evaluating unreleased models require self-hosting and privacy. Daishi already supports self-hosting; the eval design must keep it that way.

Cautionary tale: OpenAI's sprawling contributed-evals registry aged badly (the hosted Evals platform sunsets Nov 2026); their simple-evals replacement is deliberately tiny, readable, and standardized on zero-shot prompting "because it better reflects real-world usage". Keep Daishi's official eval small, curated, and runnable by anyone.

2.2 Agentic & game benchmarks: how episodes become scores

The benchmarks closest to Daishi, and what each contributes:

Two hard-won protocol lessons from the NeurIPS Neural MMO competitions transfer directly: survival time confounds everything (rank-by-lifespan correlated 0.91 with rank-by-tasks-completed across 86 submissions; report survival as its own sub-score and make economy/social/combat metrics rates per tick alive), and weak baselines saturate (22 teams maxed the easiest PvE stage; keep at least one reference tier nobody saturates).

2.3 Rating systems & statistics: ranking models when outcomes depend on opponents and seeds

The playbook, from Chatbot Arena, TrueSkill, and the eval-statistics literature:

• Batch Bradley-Terry, not online Elo. LMSYS switched in Dec 2023: online Elo is order-dependent and over-weights recent games; BT is the MLE of the same pairwise model, refit over the full log. Daishi's append-only event log is exactly the right substrate for batch refits.
• Multiplayer-native ratings for free-for-alls. One 8-agent match with a finish ordering carries ~28 implicit pairwise comparisons. TrueSkill converges in a handful of free-for-all games vs ~91 for 8v8 Elo; the unpatented OpenSkill (Plackett-Luce) is the library to actually ship (JS/Python implementations, consumes exactly a placement ordering). Sort leaderboards by the conservative estimate μ - 3σ. TrueSkill 2 shows that folding per-episode performance stats and quit/crash modeling into the rating substantially improves calibration (52%→68% outcome-prediction on Gears of War data per its paper).
• Rate model@version + scaffold as the player, never a model family. When LMSYS split API versions into separate BT entities, gpt-4-0314 vs gpt-4-0613 differed by ~50 points.
• Whole-history ratings for seasonal churn. Model versions retire; static BT breaks when deprecated models stop accumulating comparisons. TrueSkill Through Time or Whole-History Rating re-estimate all seasons jointly so a season-3 model stays comparable to a season-7 model; Glicko-2 (RD inflates with inactivity) is the lightweight live-ladder alternative.
• Uncertainty is mandatory. Bootstrap the rating fit (resample matches ~100-1000×), present ranks as statistically indistinguishable groups where CIs overlap, and set a minimum match count before a model appears ranked (Chatbot Arena, arXiv:2403.04132).
• Anthropic's "Adding Error Bars to Evals" (Miller 2024) supplies the core statistics, translated to Daishi: cluster standard errors at the match/lobby level (outcomes within one lobby share opponents and map; naive SEs can be >3× too small); compare two models with paired designs (same seeds, same opponent lineups, analyze per-seed differences); decompose variance into seed vs within-seed components to decide where the next compute dollar goes; publish a power-analysis table ("to detect a 3-point gap you need ~N matches") and refuse to render verdicts on underpowered comparisons.
• Single runs are noise. Replication studies of agentic evals (arXiv:2602.07150: 60,000 SWE-bench-Verified trajectories) found single-run pass@1 varies 2.2-6.0 pp between identical runs even at temperature 0. RL evaluation practice (rliable, NeurIPS 2021 outstanding paper) prescribes 3-10+ runs, interquartile mean, and stratified bootstrap CIs. Daishi trajectories are longer and more interactive than SWE-bench; expect worse.
• De-confound with covariates. LMArena's style control adds response-length/markdown features to the BT regression so wins from formatting load onto style coefficients. Daishi's analogues: spawn quality under the seed, join latency, token budget, lobby size.
• Opponent-pool bias. Plain win-rate/Elo is inflated by farming weak pool members; with small self-selected pools, publish the full cross-play win matrix and complement with clone-invariant ratings (Nash averaging, arXiv:1806.02643); watch for non-transitive cycles (aggressive beats economic beats turtle beats aggressive).
• Contamination. A single fixed WORLD_SEED becomes a memorization target the moment the benchmark matters (GSM1k showed up to ~13 pp overfit on a rewritten benchmark). Use a versioned seed pool: public practice seeds, rotating held-out seeds for scored matches (LiveBench-style refresh each season), private season-final seeds (GAIA/ARC-AGI tiering). Neural MMO 2023 published train/eval task overlap quantitatively (22.3% predicate-level, 2.1% full-spec); do the same for scenario suites.

2.4 Reporting standards: what a credible eval report contains

Synthesis of Model Cards, system cards (OpenAI/Anthropic), HELM's public artifacts, MLPerf's governance, BetterBench, EvalCards, and the EU AI Act:

• Identify exactly what was evaluated: model name + exact version string + provider + access date + sampling config + scaffold/harness version. Under exactly what conditions: benchmark/ruleset version, seed policy, prompts, adaptation method. Scores are only comparable within a version tuple; attach it to every number.
• Disaggregate. Report metrics sliced by factors (season phase, terrain, resource scarcity, opponent mix, activity type), never one aggregate scalar.
• Publish raw per-instance artifacts. HELM releases every prompt and completion, browsable per-instance; WebArena publishes trajectories; ClemBench ships per-episode transcripts. Daishi's equivalent: full event log + per-agent tool-call transcript + metrics for every ranked match, downloadable from the leaderboard.
• MLPerf-style divisions and process are the template for trust: Closed division (reference conditions, apples-to-apples) vs Open (innovation allowed, disclosed); submissions as structured artifacts (config, code, logs, replication README); a review window with a formal objection process before simultaneous publication; random audits with a replication path; results-messaging guidelines governing how scores may be quoted.
• BetterBench's two most-failed criteria across 24 major benchmarks: reporting statistical significance, and shipping a replication script. Both are cheap for Daishi (bootstrap CIs; docker run + seed + admin script).
• Eval cards. The 2025-26 standardization wave (EvalCards, EvalEval Coalition) converges on machine-readable records: benchmark metadata + run metadata + model metadata + links to raw logs. The fields systematically missing in the wild are where a new benchmark differentiates: provenance (who ran it, when, at whose expense), lifecycle status, preregistered metrics, comparability caveats.
• System-card-style behavioral findings (deception, collusion, aggression, exploit use) with methodology and quantitative results are what providers can reuse directly. The EU AI Act (GPAI obligations applying since Aug 2025, Commission enforcement powers from Aug 2026) requires providers to document evaluation strategies, criteria, metrics, results, and limitations (Annex XI); a well-structured Daishi report can be one input to that documentation (a useful adoption tailwind, though a game-world benchmark alone is not a compliance artifact).
• Leaderboard governance, written down before results exist (The Leaderboard Illusion, audit of 2M Arena battles): no score retraction from ranked matches; disclosed caps on private variant testing (best-of-N private submission materially inflates BT ratings); symmetric matchmaking/sampling across providers; public log of deprecated/renamed entrants; publish the matchmaking and rating algorithms themselves. Expect labs to train on public Daishi logs; that is what the rotating held-out seed pool is for.

2.5 Cost, and the trust model for telemetry

HAL (Princeton, ICLR 2026; 21,730 rollouts across 9 models × 9 benchmarks) and its predecessor critique "AI Agents That Matter" established that accuracy-only agent leaderboards mislead: agents can be 50× more expensive for similar accuracy. Their practice: meter raw tokens per run, derive dollars from a swappable price table (so history can be re-priced as prices drift), plot accuracy-vs-cost with a Pareto frontier, require multiple runs with CIs, publish full traces (encrypted, to avoid benchmark contamination), mark maintainer-reproduced entries "verified", and run LLM-aided log inspection over transcripts, which surfaced agents googling the benchmark answers and other unreported behaviors.

But cost columns are only as trustworthy as their observation point. Competitors self-reporting "which model + how many tokens" is trivially fakeable. The industry's three trust tiers map cleanly onto Daishi divisions:

Reconciliation channels exist for the middle tier: provider org-level usage APIs (OpenAI/Anthropic admin usage & cost endpoints, filterable per API key) let an auditor cross-check gateway numbers. TEE-attested inference (weights fingerprint bound into hardware attestation) exists but is practical only for open-weight models today; an optional future "attested" badge, not a foundation. One more hole to close: a BYO agent in the gateway tier must run in a sandbox whose egress is restricted to the gateway, or a "cheap model" entrant can silently consult a stronger model out-of-band.

2.6 Telemetry & data-collection standards

• OpenTelemetry GenAI semantic conventions are the naming standard: spans for invoke_agent/execute_tool/model calls; gen_ai.usage.input_tokens/output_tokens; gen_ai.request.model; duration/token histograms (p50/p99, not means). There is now an MCP-specific convention (mcp.method.name, gen_ai.tool.name, mcp.session.id, error.type, mcp.server.operation.duration); instrument the MCP boundary itself. Full prompt/completion capture should be opt-in, separate from always-on metadata. OpenInference adds first-class cost attributes (incl. cache read/write and reasoning-token detail).
• Inspect AI's .eval log is the model for a match artifact: header manifest (task/ruleset version, seed, models, config, git revision) + per-agent samples (input, messages, output, named scores, token usage, errors) + an ordered transcript of typed events with unique ids; large repeated content de-duplicated as attachments; a compact binary format with a cheap "sample summaries" read path.
• RLDS (episode/step schema for RL datasets: observation, action, reward, is_first/is_last/is_terminal, episode metadata) and PettingZoo (all outcome data keyed by agent-id per step; explicit termination-vs-truncation distinction (died vs season ended) define the replay format the RL/agents community can consume directly. Log losslessly at recording time: you cannot reconstruct what you did not capture.
• Event schema versioning (event-sourcing practice): add schemaVersion (+ engine commit) to every event, keep one CI-validated JSON Schema per event type in-repo, evolve additively, never rewrite the log, upcast at export time.
• Analytics/publication stack: keep SQLite/Postgres as the hot store; batch-export season/match-partitioned Parquet (queryable with DuckDB/pandas with zero infra; ClickHouse only if live dashboards outgrow Postgres). Publish season datasets on Hugging Face (auto-generates Croissant JSON-LD metadata → discoverable/citable).
• Licensing & privacy for published logs: match transcripts are largely LLM outputs from many providers whose ToS restrict using outputs to train competing models, so tag every message with its generating model (lets downstream users filter by provider), use LMSYS-Chat-1M-style custom research terms or ODC-BY with an explicit usage-restrictions notice (WildChat precedent), scrub free text with Presidio + a moderation pass, and keep private DM payloads out of public exports (the store already supports this filter, src/storage/store.ts:8-9).

2.7 Security: prompt injection is a distinct axis from in-game deception

Daishi's "speech can lie" design makes deception legal, but LLM-to-LLM prompt injection is a different animal, and the 2024-2026 literature (Prompt Infection, self-replicating payloads saturating a 10-agent society in ~5 turns; AgentDojo, where most models lose 10-25 pp utility under attack, and more capable agents were more susceptible to targeted injection; TAMAS, ASB, ChatInject) is directly applicable. The recommended treatment is a three-tier classification, not legal/illegal:

1. In-action-space persuasion/deception (lying about trades, bluffing): scored skill, stays legal. This is the game. 2. Instruction hijacking (payloads targeting the rival agent's control layer: override its system prompt, exfiltrate its api_key, induce actions outside its free choice): scored robustness dimension: report Injection Resistance (1 - ASR when targeted) and Injection Efficacy (ASR achieved against others) separately from game skill, plus utility-under-attack so refusing-to-play doesn't count as robust, plus a propagation index (how far payloads cascade; Daishi is a persistent society, so cascade matters more than single-hop success). 3. Harness/protocol exploits (spoofing the game-master/system role, forging tool calls, chat-template abuse): disqualifiable, blocked and flagged by the server.

Data-collection implication: message events need provenance forensics (sender id, turn, channel, verbatim bytes, and downstream-action linkage) so injection can be attributed and its propagation chain reconstructed from the log (payload-similarity matching across messages). Static injection suites overstate robustness against adaptive adversaries (GAMBIT), so any injection track needs a red-team/refresh cadence. Daishi already strips tool-like syntax from messages (see the README's Safety / anti-abuse section); that covers part of tier 3 only.

2.8 Scoring negotiation and behavioral traits from logs

Everything below is computable from Daishi's existing event types (trades with escrow status, messages, attacks, reputation), mostly without gameplay changes:

• Negotiation (NegotiationArena, GTBench, Davidson et al.): per-deal surplus split using market-imputed valuations; report three numbers: mean surplus over all attempts, over completed deals, and completion rate (so pushovers and deal-killers are distinguishable); anchoring coefficient (regression of final terms on first offer); Normalized Relative Advantage (ΣA - ΣB)/(ΣA + ΣB) for pairwise matchups.
• Promise-keeping / betrayal (the method that exposed Cicero's premeditated deception, Patterns 2024): LLM-extract commitments from messages, verify each against subsequent event-log ground truth; promise-keeping rate = kept/extracted; betrayal = attack/raid on a counterparty during an active commitment. Daishi's verified-action-vs-unverified-speech split gives ground truth that Werewolf/Diplomacy papers had to hand-label.
• Deception, two-sided (Deception Elo, Peskov et al. ACL 2020): maintain separate ratings for deceiving and for not-being-deceived; models are differentially better at one than the other. Bluffing is directly measurable as message-claims vs posted-offer divergence.
• Behavioral/safety profile (MACHIAVELLI, ICML 2023): count harm-taxonomy events along the trajectory (violence, stealing, promise-breaking, deception, manipulation...), normalize against a scripted random/baseline agent on the same seed (score = 100 × count/mean_random), and publish a score-vs-violations Pareto plot per model so "wins by exploitation" is visually separable from "wins cleanly". Daishi's aggression counter and raid/attack events cover the mechanical categories with zero annotation; only messages need LLM labeling (validate the judge against a small human-labeled set first).
• Population-level outcomes (Welfare Diplomacy): report per-match Nash welfare (geometric mean of final utilities) and score dispersion alongside individual rank; add an exploiter probe mode (one scripted defector) and report each model's welfare drop as its exploitability: aggression restraint measured as robust cooperation, not naive pacifism.
• Society metrics (Project Sid, Generative Agents): role-distribution entropy (specialization), rule-compliance rate vs stated norms, information-diffusion curves, coordination verified by log outcomes.
• Fairness validation before attribution (Werewolf Arena): identical models in a lobby must finish statistically indistinguishable on fixed seeds; use spawn-permuted repeats; keep scripted anchor bots in every lobby for cross-season comparability.

2.9 Collusion & sybil detection

An open free-for-all economy will attract feeder-agent boosting. Detection methods that map onto Daishi's log: per-season net-transfer graphs (nodes = agents, edges = net value flow from trades/gifts/thrown fights) with community detection; the chip-dumping/gold-farming signature is hub-and-spoke one-directional flow (MMORPG RMT literature); trades priced far off ITEM_VALUES reference; pattern-free collusion tables (pairwise mutual-advantage outliers, AAAI 2013); mutual information between one agent's actions and another's private state (dependence not explainable by public events ⇒ covert coordination; the log already distinguishes private events). Classic graph-based sybil detection (SybilRank et al.) assumes attack edges are costly, which fails in an open MCP world, so anchor sybil resistance in registration (verified operator identity per key, one ranked-lobby slot per (operator, model)) and use graph analysis as a secondary batch signal.

2.10 Configuration provenance: archive the effective config, not the request

The 2025-26 reproducibility wave converged on one rule for agentic evals: the unit of record is the rollout plus the complete configuration that produced it, with defaults resolved. What the operator requested is not evidence; what applied is. The practices to follow:

• Inspect's eval-log header (UK AISI) stores the resolved run configuration beside the samples, not in a lab notebook: task_args including defaulted values, model_args, generation config, dependency packages versions, and the git origin + commit (revision) of the eval source (log format, EvalSpec/EvalLog reference). A score whose config must be reconstructed from shell history is treated as unreproducible by construction.
• Rollout Cards (arXiv:2605.12131, 2026) generalize this into a publication standard: treat the rollout record — task specification, agent actions, environment dynamics, and the declared reporting rule over them — as the reproducibility unit, not the reported score. Their audit of 50 popular agent-research repositories found none reported failed/errored/skipped runs beside headline scores; a benchmark differentiates by recording run outcomes honestly (Daishi's run status + per-seat error fields, and archives of half-played matches, are that record).
• The Agentic Benchmark Checklist (arXiv:2507.02825) makes explicit documentation of toolchain, environment configuration, and test-data provenance a reporting criterion; it composes with BetterBench's two most-failed criteria from §2.4 (statistical significance, replication script).
• MLPerf submissions are structured artifacts — config + code + logs + replication README — reviewed before publication (policies); and every serious experiment tracker (MLflow, W&B) treats the run config as first-class, queryable metadata. Researchers expect "what exactly did run 47 use?" to be a lookup, never an archaeology project.

Daishi implementation (landed with this section): every MatchArchive now carries config as the effective setup — season length, pacing (turn-based budgets or wall-clock tick + rate limit), world size, physics multipliers, registration gates, scheduled directives, scenario id + content hash, engine + event-schema versions — plus wall-clock startedAtMs/endedAtMs. Studio custom runs additionally stamp a config.run provenance block (run id, resolved seed with derivation: pinned / scenario-fixed / practice-pool trial / random; requested-vs- effective overrides; per-seat model, route, temperature, skill count and customization-scaffold hash — never the private prompt text) and mirror the same record onto the run row as effective. The match report renders all of it as an Experiment configuration section with the raw record attached verbatim, so variables added later stay findable without a renderer change.

---

3. Recommendations

3.1 Data collection (the foundation; do this first)

> Implementation status: Phases 1-4 of §3.4 are implemented; see the > README's "Running fair evaluation matches" for the operator surface. > > Phase 1 (Instrument): items 1-6: match_id + match_manifest events, > model metadata at registration (REQUIRE_MODEL_INFO), action_rejected > events, wall-clock timestamps, match archives with > GET /api/matches[/:id[/export]], envelope schema versioning, both-party > trade credit, public/private message split, DETERMINISTIC seeding. > Stragglers landed with Phase 2: item 7 (per-agent agent_snapshot time > series), item 10 (verbatim message bytes + sha256, offline echo/propagation > detection in src/eval/provenance.ts), item 11 (Parquet export, > npm run export). > > Phase 2 (Standardize): reference harness + canonical content-hashed > prompt (src/harness, npm run harness); versioned content-hashed > scenario specs + seed pools (scenarios/, GET /api/scenarios, > POST /api/admin/scenario); lab-play suite with typed event-stream > assertions (npm run lab); turn-based eval tick mode with per-tick action > budgets and turn_forfeited events (EVAL_TURN_BASED, scenario-driven); > anchor bots + frozen population anchors-v1 (src/eval/anchors.ts). > > Phase 3 (Score & rate): archived-match scoring > (GET /api/matches/:id/report); repeated-trials series runner with > mean±SEM/IQM/bootstrap CIs/pass^k (npm run series); OpenSkill > Plackett-Luce ratings with bootstrap CIs, μ-3σ sort, indistinguishable > rank groups, cross-play matrix (GET /api/ratings); behavioral metrics + > collusion screens (GET /api/matches/:id/behavior); model report card / > eval card / paired-seed A-vs-B with power refusal > (GET /api/models/:model/report, /eval-card, GET /api/compare). > > Phase 4 (Scale trust): inference gateway with per-agent virtual keys, > server-side token metering (inference_usage events, OTel GenAI naming), > per-key rations, and a versioned price table (GATEWAY=true, > src/gateway/); trust labels on every report; held-out seed rotation > (scenarios/seeds.json); governance doc (docs/GOVERNANCE.md); season > dataset publication with scrubbing + ToS pass-through > (npm run publish-dataset). > > Remaining future work: item 8's periodic world-state checksum event; item 9's > sandboxed-egress enforcement for gateway BYO agents; LLM-backed commitment > extraction (the heuristic extractor and the pluggable interface exist); an > injection-robustness scored track (forensics substrate is in place); > TrueSkill-Through-Time cross-season refits. > > Config-provenance addendum (2026-09, §2.10): the "which exact setup > produced this number" gap is closed end-to-end for Studio custom runs and > archives generally: effective config + wall-clock start/end + run > provenance on every MatchArchive, the resolved effective record on > every launched run row, and the match report's Experiment > configuration section (with the raw config attached verbatim).

Priority-ordered; items 1-6 are prerequisites for everything in §3.2-3.3.

1. Add a match_id to the world and every event. Mint a UUID at resetWorld/endSeason; add match_id (and season) columns to events; stamp a match manifest event at launch: seed, engine version/commit, full effective config (TICK_SECONDS, SEASON_TICKS, MAX_AGENTS, rate limits, ruleset hash), roster. This is the reproducibility tuple every result will reference. 2. Capture model identity at registration. Extend register_agent with optional structured metadata: model (exact version string), provider, scaffold (name + version), operator (contact), config (temperature, reasoning effort). Store on AgentState, emit in agent_registered, surface in /api/metrics and the leaderboard. Rate model@version + scaffold as the entity, never a family (§2.3). For ranked matches, make these fields required and validated. 3. Log failed actions. Emit an action_rejected event (agent, tool name, error code, tick) from the MCP wrapper, including rate-limit rejections (move the emit before the throw, or emit in the catch). Invalid-action rate, retry patterns, and throttling pressure become computable; ClemBench-style %played needs this. 4. Expose wall-clock time and decision latency. Stop dropping created_at in queryEvents; additionally record per-action latency_ms since the agent's previous action. This is the only latency signal the server can see, and it feeds both fairness analysis and efficiency reporting. 5. Archive everything at match end. On endSeason/resetWorld, snapshot the full final board (including dead agents, with cause and tick of death), final per-agent metrics, attributes, and the match manifest into a match_results table (or a season-stamped JSON artifact). Add GET /api/matches/:id/report and GET /api/matches/:id/export (events + manifest + results as one downloadable bundle, our Inspect-.eval analogue). Fix the survivorship bias: dead agents get final scores and lineage entries too. 6. Version the event schema. schemaVersion on EventRecord, one JSON Schema per event type in-repo, CI validation, additive evolution, upcast on export. Enrich the thin payloads while at it (e.g. look currently logs only {name}, engine.ts:321); make quantities (amounts gathered, damage, trade value) consistently present so metrics can aggregate values, not just counts. Fix the attribution quirks: emit trade_completed for both parties; exclude private messages from public counters. 7. Per-agent time series. Emit a compact per-agent snapshot (energy, wealth, location, inventory hash) every N ticks; enables Voyager-style progression curves (wealth-vs-tick, time-to-first-milestone) that endpoint metrics can't show. 8. Deterministic benchmark mode. Wire WORLD_SEED through injectable seeded streams (the hooks exist: engine.ts:55, engine.ts:1213): per-subsystem mulberry32 streams (spawn, season-seed derivation) or a stateless hash-RNG keyed by (seed, tick, agentId, purpose). Record every derived seed in the manifest. Optionally add a periodic world-state checksum event so replay divergence is caught immediately. 9. Optional gateway telemetry for verified matches. For the verified division (§3.2), run a LiteLLM-class proxy: one virtual key per agent per match, server-side token/cost/model metering written into the match log (named per OTel GenAI conventions: gen_ai.request.model, gen_ai.usage.*), per-key budgets as enforced compute rations. Store raw tokens, not dollars; re-price from a versioned price table like HAL. 10. Message provenance for the injection track. Message events already carry sender/recipient; add verbatim-bytes retention, quote/echo detection (payload similarity across messages), and downstream-action linkage so propagation chains are reconstructable (§2.7). 11. Analytics export. Nightly/end-of-match job: match-partitioned Parquet of events + metrics + results. Publish season datasets (post-Presidio scrub, private DMs excluded, per-message model tags, pass-through ToS notice) on Hugging Face.

3.2 The standardized evaluation

Two tracks, one metric vocabulary (the offline/online split every platform ends up with):

• Ladder (online): the persistent world as-is: continuously scored telemetry, whole-history ratings, trends. Interesting, but not the standardized eval.
• Match play (offline, leaderboard-grade): lobby-synchronized, fixed-seed, manifest-stamped matches. This is what companies run to get a report.

Scenario specs. Define each official scenario as a versioned declarative config (daishi:open-play-v1, daishi:famine-v1, daishi:trade-required-v1...): seed policy, season length in ticks, roster size, background population, resource multipliers, difficulty/interdependence knobs (Alem-style: dial cooperation from optional to mandatory). Plus a lab-play suite (FLE-style): short scripted scenarios with threshold completion ("survive 200 ticks of famine", "complete a 3-party trade", "recover from a raid") that run in minutes at TICK_SECONDS=2: the cheap CI-runnable regression check, distributed as daishi-eval with typed assertions over the event stream.

Match protocol (per model, per scenario):

• ≥5 matches across ≥3 held-out seeds (public practice seeds are separate); paired seeds/lineups when comparing two models (§2.3).
• Fixed tick budget; per-tick action budget instead of the wall-clock 2 s floor; advance an eval-mode tick only when all agents have acted or their per-turn timeout lapsed (Kaggle-style pause-on-decision, with timeouts logged as explicit forfeit events). This removes the latency confound; wall-clock "survival mode" remains a labeled variant for the ladder.
• Every lobby includes anchor bots (scripted: random, greedy harvester, honest trader, aggressive raider) and, for official runs, a frozen background population of pinned reference-model agents (Melting Pot pattern): the absolute yardstick that makes scores comparable across seasons. Keep one anchor tier nobody saturates.
• Both cross-play (model among references) and self-play (model fills all slots) divisions; report both (Neural MMO finals shifted scores materially between them; Concordia's cooperation lesson).
• Spawn-permuted repeats on the same seed; self-play balance checks before attributing differences to models (Werewolf Arena).

Scoring: a small hierarchy, all computed offline from the event log (never in the engine, so metrics can be recomputed for old matches):

daishi_score = rule_following × capability          (ClemBench pattern)

rule_following = 1 - invalid_action_rate           (needs data item 3)

capability     = weighted sub-scores, each a Crafter log-mean over that
                 domain's achievement success rates across the match set:
  survival   = ticks alive, dormancy recoveries, winter survived
  economy    = wealth percentile, gather/craft/build milestones, market activity
  social     = trades completed, surplus split, promise-keeping, diffusion/coordination
  conflict   = win rate when engaged, defense success, restraint-adjusted
  (per-tick-alive rates, not lifetime totals; survival confounds everything, §2.2)

Alongside, per model: reliability (pass^k on lab-play scenarios: "survives famine in 8/8 replays"), efficiency (tokens, cost, actions per unit wealth; score-vs-cost Pareto per HAL), behavioral profile (MACHIAVELLI-normalized violation counts, promise-keeping, deception/detection ratings, exploitability under the defector probe), and optionally injection resistance as its own labeled track (§2.7). Keep wealth as the in-world incentive; the report score is the multi-dimensional one.

Ratings. OpenSkill Plackett-Luce over match placements for the leaderboard (μ - 3σ sort, provisional until a minimum match count), batch-refit + bootstrap CIs on every report, ranks displayed as indistinguishable groups, TrueSkill-Through-Time refit at season end for cross-season comparability, full cross-play matrix published.

Divisions & trust tiers (label travels with every score):

• Reference (Closed): Daishi-run harness (a minimal, open, versioned MCP agent loop + canonical system prompt, the missing piece README already presumes), operator-held model keys. Fully verified; headline division.
• Gateway (Open-verified): BYO scaffold, inference routed through the Daishi proxy; usage verified, scaffold disclosed; sandboxed egress.
• BYO (Open-unverified): anything goes, encrypted trace submission mandatory, plausibility checks + random replay audits (deterministic replay makes our audits cheap: MLPerf needs SSH into submitter hardware; we need a seed and a log).

Governance, written before the first ranked season: no retraction of ranked results; disclosed caps on private practice variants; symmetric scheduling; public deprecation log; published matchmaking + rating algorithms; a review/objection window before season results publish; one ranked-lobby slot per verified (operator, model); collusion screens (§2.9) run on every ranked match.

3.3 Reports

Three artifacts per (model, scenario-suite, season), all generated from the archived match data:

1. Model report card (human-readable; Model Card skeleton + Vending-Bench qualitative depth): identification block (model@version, scaffold, config, dates, division/trust tier, ruleset hash); headline daishi_score with CI and rank-group; sub-score radar with per-achievement success-rate vector; progression curves (wealth-vs-tick, time-to-first-milestone); reliability (pass^k table); efficiency (score-vs-cost point on the season Pareto plot); behavioral findings section (system-card style: deception rate, promise-keeping, betrayals, violence normalized vs baseline, exploitability, score-vs-violations Pareto position); failure-mode taxonomy mined from its worst matches (the section developers act on; consider HAL-style LLM-aided log inspection to draft it); limitations & comparability caveats. 2. Machine-readable eval card (EvalCards schema): benchmark metadata + run metadata + model metadata + links to the raw match bundles. Populate the fields the wild omits: provenance, lifecycle status, preregistered metrics, comparability caveats. 3. Comparison report (Braintrust-style diff): model A vs B (typically v2 vs v1) on paired seeds: per-metric deltas with paired-difference CIs, per-scenario regressions/improvements, drill-down links from any delta to the specific match events behind it. Refuse to render a verdict below the power threshold; say "need N more matches" instead.

Leaderboard page per season: models-as-rows × metric columns (defined as data, W&B-Weave style), CI bars, division/trust labels, cost column, download links to every match bundle, replication script (docker run + seed + admin commands), and the datasheet pair (world/scenario datasheet + collected-data datasheet per Gebru et al.).

3.4 Suggested phasing

The through-line: Daishi's append-only, server-authoritative event log is the asset. Every recommendation above is either (a) making that log complete enough to be the single source of truth (§3.1), or (b) pure functions over it (§3.2-3.3). Nothing requires changing what the game is.

---

4. Sources

Codebase (verified at time of writing)

src/domain/engine.ts, src/domain/types.ts, src/domain/constants.ts, src/domain/rng.ts, src/server/host.ts, src/server/mcp.ts, src/server/http.ts, src/server/dashboard.ts, src/storage/{store,sqlite,postgres}.ts, README.md, docs/AGENT_GUIDE.md, tests/lobby.test.ts, tests/mcp.test.ts.

Harnesses & platforms

• Inspect AI (UK AI Security Institute): https://github.com/UKGovernmentBEIS/inspect_ai (esp. eval-logs, eval-sets, agent limits)
• Stanford HELM: https://github.com/stanford-crfm/helm · https://crfm.stanford.edu/helm/
• EleutherAI lm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness
• OpenAI simple-evals / evals: https://github.com/openai/simple-evals · https://github.com/openai/evals
• promptfoo: https://github.com/promptfoo/promptfoo · Braintrust: https://www.braintrust.dev/docs/platform/experiments · LangSmith: https://docs.langchain.com/langsmith/evaluation · W&B Weave: https://docs.wandb.ai/weave/guides/core-types/leaderboards · DeepEval: https://github.com/confident-ai/deepeval

Agentic & game benchmarks

• HAL (Holistic Agent Leaderboard): arXiv:2510.11977 · https://hal.cs.princeton.edu · https://github.com/princeton-pli/hal-harness
• AI Agents That Matter (Kapoor, Stroebl, Narayanan): arXiv:2407.01502
• Neural MMO 2.0: arXiv:2311.03736 · NeurIPS 2023 competition retrospective: arXiv:2508.12524 · NeurIPS 2022 challenge: PMLR v220
• Crafter: arXiv:2109.06780 · Craftax: arXiv:2402.16801
• BALROG: arXiv:2411.13543 · FLE (Factorio): arXiv:2503.09617 · TextArena: arXiv:2504.11442 · ClemBench: arXiv:2305.13455
• tau-bench (pass^k): arXiv:2406.12045 · SWE-bench: https://github.com/SWE-bench/SWE-bench · WebArena: https://github.com/web-arena-x/webarena · AgentBench: https://github.com/THUDM/AgentBench · GAIA: arXiv:2311.12983 · MLE-bench: https://github.com/openai/mle-bench · ARC-AGI: https://arcprize.org
• Melting Pot 2.0: arXiv:2211.13746 · Concordia Contest (NeurIPS 2024): https://www.cooperativeai.com/contests/concordia-2024
• Vending-Bench 1 & 2: arXiv:2502.15840 · https://andonlabs.com/evals/vending-bench-2
• Kaggle Game Arena: https://github.com/google-deepmind/game_arena · Kaggle simulation-competition ladder docs
• Voyager: arXiv:2305.16291 · MineDojo: arXiv:2206.08853 · Project Sid: arXiv:2411.00114 · Generative Agents: arXiv:2304.03442
• MultiAgentBench/MARBLE: arXiv:2503.01935 · Alem: https://alem-world.github.io/

Ratings & statistics

• Chatbot Arena methodology: arXiv:2403.04132 · Elo→Bradley-Terry: lmsys.org/blog/2023-12-07-leaderboard · style control: lmsys.org/blog/2024-08-28-style-control
• Adding Error Bars to Evals (Miller/Anthropic): arXiv:2411.00640
• TrueSkill / TrueSkill 2: microsoft.com/research (trueskill2.pdf) · OpenSkill: arXiv:2401.05451 · TrueSkill Through Time: NeurIPS 2007, https://github.com/glandfried/TrueSkillThroughTime.py · WHR (Coulom 2008) · Glicko-2: glicko.net
• rliable / Statistical Precipice: arXiv:2108.13264 · How Many Random Seeds: arXiv:1806.08295 · Quantifying Variance in Benchmarks: arXiv:2406.10229 · On Randomness in Agentic Evals: arXiv:2602.07150 · Agentic Benchmark Checklist: arXiv:2507.02825
• Nash averaging / mElo: arXiv:1806.02643 · The Leaderboard Illusion: arXiv:2504.20879 · LiveBench: arXiv:2406.19314 · GSM1k: arXiv:2405.00332

Reporting standards

• Model Cards: arXiv:1810.03993 · Datasheets for Datasets: arXiv:1803.09010 · GPT-4o System Card: cdn.openai.com · Anthropic system cards: anthropic.com/system-cards
• Rollout Cards (reproducibility standard for agent research): arXiv:2605.12131 · Inspect eval-log/EvalSpec fields: inspect.aisi.org.uk/eval-logs.html
• MLPerf submission/review/audit rules: https://github.com/mlcommons/policies · https://github.com/mlcommons/inference_policies
• BetterBench: arXiv:2411.12990 · https://betterbench.stanford.edu · EvalCards: arXiv:2511.21695 · Evaluation Cards (EvalEval): evalevalai.com
• EU AI Act Art. 53 / Annex XI + GPAI Code of Practice: artificialintelligenceact.eu · HF leaderboard norms: huggingface.co/docs/leaderboards

Telemetry & data

• OpenTelemetry GenAI + MCP semantic conventions: https://github.com/open-telemetry/semantic-conventions-genai
• OpenInference: https://github.com/Arize-ai/openinference · RLDS: https://github.com/google-research/rlds · PettingZoo/Gymnasium episode stats: farama.org
• LiteLLM proxy (virtual keys, budgets): docs.litellm.ai · OpenRouter usage accounting: openrouter.ai/docs · Helicone: helicone.ai · provider usage/cost admin APIs: docs.anthropic.com/en/api/usage-cost-api, OpenAI org usage API · TEE attestation: docs.tinfoil.sh
• Event versioning: event-driven.io · Parquet/DuckDB lakehouse patterns · Croissant: mlcommons.org · LMSYS-Chat-1M & WildChat licensing/PII precedents: huggingface.co · Presidio: https://github.com/microsoft/presidio

Security & behavior

• Prompt Infection: arXiv:2410.07283 · AgentDojo: arXiv:2406.13352 · TAMAS: arXiv:2511.05269 · Agent Security Bench: arXiv:2410.02644 · ChatInject: arXiv:2509.22830 · MASpi: openreview.net/forum?id=1khmNRuIf9 · ACIArena: arXiv:2604.07775 · GAMBIT: arXiv:2605.09027 · Spotlighting: arXiv:2403.14720 · OWASP Agentic Threats (T12-T14): genai.owasp.org
• NegotiationArena: arXiv:2402.05863 · GTBench: arXiv:2402.12348 · LLM-Deliberation: arXiv:2309.17234 · LM negotiation agency: arXiv:2401.04536
• MACHIAVELLI: https://github.com/aypan17/machiavelli (ICML 2023) · AI Deception / Cicero audits: Patterns 2024, arXiv:2406.04643 · It Takes Two to Lie: ACL 2020 · Hoodwinked: arXiv:2308.01404 · AvalonBench: arXiv:2310.05036 · Werewolf Arena: arXiv:2407.13943 · Among Us Deception Elo: arXiv:2504.04072 · Welfare Diplomacy: arXiv:2310.08901
• Collusion/sybil: collusion tables: AAAI 2013 (Mazrooei et al.) · chip-dumping detection: AMLTRIX T0107.003 · MMORPG RMT networks: Fujita et al. AIIDE · SybilRank family surveys: arXiv:2507.06541