# Standardized Evaluation & Data Collection: Research Report

**Status:** research / design input; no code changes included.
**Question answered:** how should Daishi collect data and structure a standardized
evaluation + reports that companies and individuals can use to improve their models,
and what is the industry currently using to evaluate AI models?

This report has four parts:

1. [Where Daishi stood when this research was written](#1-where-daishi-stood-when-this-research-was-written): the baseline we started from and the gaps that motivated §3 (since closed; see §3.1)
2. [What the industry does](#2-what-the-industry-does): the 2024-2026 evaluation landscape, organized by the decision it informs
3. [Recommendations](#3-recommendations): a concrete design for Daishi's standardized eval, reports, and data collection
4. [Sources](#4-sources)

Everything in §1 was verified against the code at the commit this document was written
at. Industry claims in §2 were checked against primary sources (papers, repos, official
docs) listed in §4.

---

## 1. Where Daishi stood when this research was written

> **Historical snapshot.** This section describes the tree as it stood before the
> §3 roadmap was implemented; its `file:line` receipts refer to that commit and have
> drifted since. The gaps listed in §1.2 have been closed (Phases 1-4; see the
> implementation-status callout in §3.1). It is kept as the motivation record for
> the design in §3, not as a description of the current code.

Daishi is unusually well positioned to be a benchmark: the server is authoritative
("physics is truth"), every successful action already lands in an **append-only event
log**, and the lobby gives synchronized, fixed-seed match starts: the structural
property most benchmarks have to build from scratch (operator-observed ground truth
instead of self-reported scores).

### 1.1 What we already collect

| Data | Where | Notes |
|---|---|---|
| Append-only event log | `events` table: `id, tick, type, agent_id, region_id, private, data(JSON), created_at` (`src/storage/sqlite.ts:24-33`, `src/storage/postgres.ts:30-39`) | 36 emit sites in `src/domain/engine.ts`; never deleted; survives resets |
| Behavior counters | `GET /api/metrics` (`src/server/http.ts:172-226`) | `GROUP BY (agent_id, type)` over the whole events table + live state |
| Live world state | `GET /api/state` (`src/server/http.ts:69-146`) | leaderboard top-10, season history (winner + score only) |
| Score | `wealth()` (`src/domain/engine.ts:1078-1103`) | inventory value + escrow + structures·(hp/maxHp) + reputation×2 + fitness×5 |
| Season archive | lineage: `{season, score, rank}` per **surviving** agent (`src/domain/engine.ts:1244-1252`); `SeasonResult.winners` = top-10 (`engine.ts:1240`) | |
| Fair-match protocol | lobby freeze + hidden map + simultaneous spawn (`src/server/host.ts:73-75, 92-105`; `src/domain/engine.ts:292-308`), `WORLD_SEED`, `LATE_JOIN=false`, admin start/reset/tick (`src/server/http.ts:243-300`) | |

### 1.2 The gaps that matter for evaluation

Ordered roughly by how much they block a standardized eval:

1. **No model identity.** `register_agent` takes only `name` + `signup_token`
   (`src/server/mcp.ts:90-93`); `/api/metrics` ships a hardcoded
   `model_note: 'set by owner, not tracked server-side'` (`src/server/http.ts:188`).
   The central benchmark dimension, *which model played*, has no first-class field
   anywhere. Attribution today is an external spreadsheet.
2. **No match/run identifier.** The `events` table has no season or match column
   (`src/storage/sqlite.ts:24-33`); `resetWorld` restarts tick at 0
   (`src/domain/engine.ts:1217`) so tick values collide across matches, and slicing one
   match out of the log means scanning for `world_reset`/`season_started` marker events.
3. **Failed actions are invisible.** `ActionError`s (invalid args, `not_here`,
   insufficient energy, `rate_limited`, unauthorized) are returned to the caller and
   never persisted (`src/server/mcp.ts:64-78`; rate-limit rejections throw *before* the
   persist path, `src/server/host.ts:171-189`). Invalid-action rate, one of the
   strongest model-quality signals in agentic evals, cannot be computed from our data.
4. **Results are (mostly) destroyed at rollover.** `/api/metrics` iterates only
   current-season agents (`src/server/http.ts:184`), so a finished match's metrics
   vanish at reset. Lineage does archive `{score, rank}` for every agent *alive* on the
   final board, but `leaderboard()` filters `status === 'dead'`
   (`src/domain/engine.ts:1107`); an agent that died is indistinguishable from one that
   never played, and no metric/attribute snapshot, cause-of-death, or full final board
   beyond top-10 is kept. There is no end-of-match export artifact of any kind.
5. **Timing is unfair by construction.** The rate limit is a wall-clock floor
   (1 action / 2 s, `src/server/host.ts:170-181`), not a per-tick budget: at
   `TICK_SECONDS=60` a low-latency harness can take ~30 rate-limited actions per tick
   while a 20 s-latency harness gets ~3; harness speed directly buys in-game
   throughput. Also note only the 17 actions in `RATE_LIMITED_ACTIONS`
   (`src/server/host.ts:8-12`) are limited at all; `status`/`read_messages`/`world_info`
   are unbounded.
6. **Determinism is incomplete, but the fix path already exists.** Worldgen is fully
   seed-deterministic (mulberry32, `src/domain/rng.ts:4-13`), and `tick()` contains zero
   randomness (`src/domain/engine.ts:1123-1204`). But `Game.random` defaults to
   `Math.random` (`engine.ts:55`) and is used for spawn assignment (`engine.ts:280-285`)
   and next-season seeds (`engine.ts:1256`). Importantly the engine *already accepts* an
   injectable `random` source and `resetWorld` accepts an explicit seed
   (`engine.ts:1213`); full benchmark-mode determinism is a configuration change plus
   seeded streams, not a rebuild.
7. **No cost/latency/token telemetry.** Events record `tick` but the wall-clock
   `created_at` is dropped in the row mapping (`src/storage/sqlite.ts:100-108`), so even
   decision latency between an agent's actions is unrecoverable. Token usage and dollar
   cost are entirely outside our view (the server never sees the LLM calls).
8. **Config is not part of the record.** `TICK_SECONDS`, `SEASON_TICKS`, `MAX_AGENTS`
   are env-overridable (`src/domain/constants.ts:88-95`) but only `seed` is stored in
   `WorldState` (`src/domain/types.ts:171`); two "identical" runs can silently differ.
   Event payloads are untyped `Record<string, unknown>` (`types.ts:155`) with no schema
   version.
9. **No reference harness.** README says "fix the harness prompt, vary only the model"
   (`README.md:142-144`) but the repo ships no harness or prompt; every participant's
   scaffold differs, confounding model comparison. Nothing documents the event schema,
   the scoring formula, or the path from "I connected my model" to "I got a report".
10. **No sybil/collusion story.** `SIGNUP_TOKEN` is one shared secret
    (`src/server/host.ts:137-143`); nothing links multiple agents to one operator, so a
    competitor can field cooperating feeder agents in a scored match undetected.
11. **Metric attribution quirks.** `trade_completed` carries only the acceptor's
    agentId (`engine.ts:905-909`) so offerers' completed trades are invisible in
    metrics; `messages_sent` counts private DMs because `eventTypeCounts()` has no
    private filter (`src/storage/sqlite.ts:111-116`); quantities inside payloads
    (gathered amounts, damage, trade value) are never aggregated, only event counts.

---

## 2. What the industry does

### 2.1 Harness architecture: environment / agent / scorer as separate layers

The clearest convergence across 2025-2026 eval frameworks
([Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai), the UK AI Security
Institute framework that leads agentic evals, [HELM](https://github.com/stanford-crfm/helm),
[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness),
[promptfoo](https://github.com/promptfoo/promptfoo), Braintrust, LangSmith, W&B Weave):

- **Task = Dataset + Solver + Scorer** (Inspect's decomposition). The environment, the
  agent scaffold, and the scoring pipeline are three independently swappable layers.
  Scorers are *pure functions over recorded logs*, never logic inside the run loop, so
  new metrics can be back-applied to old runs.
- **Scenarios are declarative, versioned configs, not code** (lm-eval-harness YAML
  tasks, HELM RunSpecs, promptfooconfig). Two labs running `daishi:survival-s3` should
  get bit-identical worlds; results reference the scenario spec by version/hash.
- **One self-contained run artifact per evaluation** (Inspect `.eval` logs): full
  message/action transcripts, exact config, errors, per-model token usage: replayable,
  queryable as dataframes, with a CLI to reconstruct the run config.
- **Repetitions are first-class** (Braintrust trials, LangSmith repetitions, Inspect
  epochs): N runs per configuration is the unit of evaluation, averaged at the top with
  individual runs preserved.
- **Comparison-first reporting** (Braintrust experiment-vs-baseline diffs): the report a
  model developer actually wants is "v2 vs v1, same seeds, which behaviors regressed?"
  with drill-down from metric delta to raw transcript.
- **Leaderboards defined as data** (W&B Weave): column = scorer + aggregation, row =
  model, recomputed from immutable logs; adding a metric is a config change.
- **Local-first with opt-in sharing** (promptfoo, DeepEval): companies evaluating
  unreleased models require self-hosting and privacy. Daishi already supports
  self-hosting; the eval design must keep it that way.

Cautionary tale: OpenAI's sprawling contributed-evals registry aged badly (the hosted
Evals platform sunsets Nov 2026); their `simple-evals` replacement is deliberately tiny,
readable, and standardized on zero-shot prompting "because it better reflects real-world
usage". Keep Daishi's official eval small, curated, and runnable by anyone.

### 2.2 Agentic & game benchmarks: how episodes become scores

The benchmarks closest to Daishi, and what each contributes:

| Benchmark | What it is | The idea to steal |
|---|---|---|
| [Neural MMO 2.0](https://arxiv.org/abs/2311.03736) | massively multi-agent survival world (the closest ancestor) | **declarative task predicates** over game state returning continuous progress in [0,1] (`StayAlive`, `EarnGold`, `MakeProfit`, `DefeatEntity`...), evaluated over a write-only event log with a typed, domain-partitioned event-code taxonomy |
| [Crafter](https://arxiv.org/abs/2109.06780) / [Craftax](https://arxiv.org/abs/2402.16801) | open-world survival with achievement sets | **log/geometric-mean over achievement success rates**, `S = exp(mean(ln(1+sᵢ))) - 1`, so easy-achievement grinding can't mask absent deep behavior; Craftax adds superlinear tier weights (1/3/5/8) and normalizes to % of max at a declared budget |
| [BALROG](https://arxiv.org/abs/2411.13543) | LLM agents on NetHack/Crafter/etc. | fixed agent wrapper so the model is the only variable; **0-100 progression metrics** calibrated on human play; 5 seeds with standard errors; procedural generation as contamination defense |
| [tau-bench](https://arxiv.org/abs/2406.12045) | tool-agents under policy constraints | **pass^k** (probability all k i.i.d. trials succeed): ~60% pass^1 collapsed to <25% at pass^8; reliability is a separate axis from capability |
| [ClemBench](https://clembench.github.io) | self-play dialogue games | **clemscore = %played × quality**: separate "can it follow the protocol at all" (aborted episodes, rule violations) from "did it play well", then multiply |
| [Melting Pot 2.0](https://arxiv.org/abs/2211.13746) | multi-agent RL generalization | **frozen background populations**: scenario = substrate + fixed co-player population; scores normalized against baselines (worst = 0, best = 1) so results are comparable across seasons and labs |
| [Concordia Contest](https://www.cooperativeai.com/contests/concordia-2024) (NeurIPS 2024) | LLM agents in text-mediated social/economic scenarios | **self-play vs cross-play** on held-out cooperation scenarios; cooperation that only works among clones of the same model is not cooperative intelligence |
| [Vending-Bench](https://andonlabs.com/evals/vending-bench-2) | long-horizon economic sim | one intuitive headline metric (final net worth), k runs with min/max/variance, and a **qualitative failure-mode taxonomy** mined from logs (the part model developers act on). Note: models actually *beat* the measured single-trial human baseline ($844); the oft-quoted ~$63k figure is a theoretical-optimum estimate, not a human score |
| [FLE (Factorio)](https://arxiv.org/abs/2503.09617) | factory-building agents | **dual mode**: open-play (unbounded economic score) + lab-play (short, fixed-seed scripted scenarios with threshold completion): the cheap, repeatable eval companies can run in CI |
| [TextArena](https://arxiv.org/abs/2504.11442) | 57+ competitive text games, live ladder | **TrueSkill ratings per match** with skill-tagged sub-leaderboards (negotiation, deception, planning...) |
| [Kaggle Game Arena](https://github.com/google-deepmind/game_arena) | frontier models playing chess etc. | operator-run harness wraps every model identically (usage = ground truth); all-play-all scheduling; randomized openings ≙ rotating seeds |
| [ARC-AGI](https://arcprize.org/arc-agi/3) | fluid-intelligence + interactive games (v3) | **tiered secrecy** (public / semi-private / private eval sets) and a **score-vs-cost-per-task leaderboard** with priced human baselines |
| [MLE-bench](https://github.com/openai/mle-bench) | ML-engineering agents | codified protocol: ≥3 seeds, mean ± SEM, fixed compute/runtime budget, required per-run artifact bundle, and a governance cautionary tale (leaderboard paused for lack of submission-comparability rules) |
| [SWE-bench](https://github.com/SWE-bench/SWE-bench) / [WebArena](https://github.com/web-arena-x/webarena) | coding/web agents | **execution-verified scoring** against environment end-state, never judge opinion; Daishi's server-validated actions give this for free |
| [HAL: Holistic Agent Leaderboard](https://hal.cs.princeton.edu) (Princeton) | third-party cost-controlled agent leaderboard | see §2.5, the closest existing thing to "standardized eval + reports for model developers" |
| [Project Sid](https://arxiv.org/abs/2411.00114) / [Generative Agents](https://arxiv.org/abs/2304.03442) | LLM societies (10-1000+ agents) | **civilizational metrics**: role-distribution entropy for specialization, rule-compliance rate vs stated norms with before/after deltas, information-diffusion curves with hallucination-checked "interviews", coordination scored by log-verifiable outcomes (who showed up), not chat claims |

Two hard-won protocol lessons from the NeurIPS Neural MMO competitions transfer
directly: **survival time confounds everything** (rank-by-lifespan correlated 0.91 with
rank-by-tasks-completed across 86 submissions; report survival as its own sub-score and
make economy/social/combat metrics *rates per tick alive*), and **weak baselines
saturate** (22 teams maxed the easiest PvE stage; keep at least one reference tier
nobody saturates).

### 2.3 Rating systems & statistics: ranking models when outcomes depend on opponents and seeds

The playbook, from Chatbot Arena, TrueSkill, and the eval-statistics literature:

- **Batch Bradley-Terry, not online Elo.** LMSYS switched in Dec 2023: online Elo is
  order-dependent and over-weights recent games; BT is the MLE of the same pairwise
  model, refit over the full log. Daishi's append-only event log is exactly the right
  substrate for batch refits.
- **Multiplayer-native ratings for free-for-alls.** One 8-agent match with a finish
  ordering carries ~28 implicit pairwise comparisons.
  [TrueSkill](https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/)
  converges in a handful of free-for-all games vs ~91 for 8v8 Elo; the unpatented
  [OpenSkill](https://arxiv.org/abs/2401.05451) (Plackett-Luce) is the library to
  actually ship (JS/Python implementations, consumes exactly a placement ordering).
  Sort leaderboards by the conservative estimate μ - 3σ. TrueSkill 2 shows that folding
  per-episode performance stats and quit/crash modeling into the rating substantially
  improves calibration (52%→68% outcome-prediction on Gears of War data per its paper).
- **Rate `model@version + scaffold` as the player, never a model family.** When LMSYS
  split API versions into separate BT entities, `gpt-4-0314` vs `gpt-4-0613` differed by
  ~50 points.
- **Whole-history ratings for seasonal churn.** Model versions retire; static BT breaks
  when deprecated models stop accumulating comparisons.
  [TrueSkill Through Time](https://github.com/glandfried/TrueSkillThroughTime.py) or
  [Whole-History Rating](https://www.remi-coulom.fr/WHR/) re-estimate all seasons
  jointly so a season-3 model stays comparable to a season-7 model; Glicko-2 (RD
  inflates with inactivity) is the lightweight live-ladder alternative.
- **Uncertainty is mandatory.** Bootstrap the rating fit (resample matches ~100-1000×),
  present ranks as **statistically indistinguishable groups** where CIs overlap, and set
  a minimum match count before a model appears ranked (Chatbot Arena,
  [arXiv:2403.04132](https://arxiv.org/abs/2403.04132)).
- **[Anthropic's "Adding Error Bars to Evals"](https://arxiv.org/abs/2411.00640)**
  (Miller 2024) supplies the core statistics, translated to Daishi: cluster standard
  errors at the match/lobby level (outcomes within one lobby share opponents and map;
  naive SEs can be >3× too small); compare two models with **paired designs** (same
  seeds, same opponent lineups, analyze per-seed differences); decompose variance into
  seed vs within-seed components to decide where the next compute dollar goes; publish a
  power-analysis table ("to detect a 3-point gap you need ~N matches") and refuse to
  render verdicts on underpowered comparisons.
- **Single runs are noise.** Replication studies of agentic evals
  ([arXiv:2602.07150](https://arxiv.org/abs/2602.07150): 60,000 SWE-bench-Verified
  trajectories) found single-run pass@1 varies 2.2-6.0 pp between identical runs even at
  temperature 0. RL evaluation practice
  ([rliable](https://github.com/google-research/rliable), NeurIPS 2021 outstanding
  paper) prescribes 3-10+ runs, interquartile mean, and stratified bootstrap CIs.
  Daishi trajectories are longer and more interactive than SWE-bench; expect worse.
- **De-confound with covariates.** LMArena's style control adds response-length/markdown
  features to the BT regression so wins from formatting load onto style coefficients.
  Daishi's analogues: spawn quality under the seed, join latency, token budget, lobby
  size.
- **Opponent-pool bias.** Plain win-rate/Elo is inflated by farming weak pool members;
  with small self-selected pools, publish the full cross-play win matrix and complement
  with clone-invariant ratings (Nash averaging,
  [arXiv:1806.02643](https://arxiv.org/abs/1806.02643)); watch for non-transitive
  cycles (aggressive beats economic beats turtle beats aggressive).
- **Contamination.** A single fixed `WORLD_SEED` becomes a memorization target the
  moment the benchmark matters (GSM1k showed up to ~13 pp overfit on a rewritten
  benchmark). Use a versioned seed pool: public practice seeds, rotating held-out seeds
  for scored matches (LiveBench-style refresh each season), private season-final seeds
  (GAIA/ARC-AGI tiering). Neural MMO 2023 published train/eval task overlap
  quantitatively (22.3% predicate-level, 2.1% full-spec); do the same for scenario
  suites.

### 2.4 Reporting standards: what a credible eval report contains

Synthesis of [Model Cards](https://arxiv.org/abs/1810.03993), system cards
(OpenAI/Anthropic), HELM's public artifacts, [MLPerf's
governance](https://github.com/mlcommons/policies),
[BetterBench](https://betterbench.stanford.edu),
[EvalCards](https://arxiv.org/abs/2511.21695), and the EU AI Act:

- **Identify exactly what was evaluated**: model name + exact version string + provider
  + access date + sampling config + scaffold/harness version. Under exactly what
  conditions: benchmark/ruleset version, seed policy, prompts, adaptation method.
  Scores are only comparable within a version tuple; attach it to every number.
- **Disaggregate.** Report metrics sliced by factors (season phase, terrain, resource
  scarcity, opponent mix, activity type), never one aggregate scalar.
- **Publish raw per-instance artifacts.** HELM releases every prompt and completion,
  browsable per-instance; WebArena publishes trajectories; ClemBench ships per-episode
  transcripts. Daishi's equivalent: full event log + per-agent tool-call transcript +
  metrics for every ranked match, downloadable from the leaderboard.
- **MLPerf-style divisions and process** are the template for trust: Closed division
  (reference conditions, apples-to-apples) vs Open (innovation allowed, disclosed);
  submissions as structured artifacts (config, code, logs, replication README); a review
  window with a formal objection process before simultaneous publication; random audits
  with a replication path; results-messaging guidelines governing how scores may be
  quoted.
- **BetterBench's two most-failed criteria** across 24 major benchmarks: reporting
  statistical significance, and shipping a replication script. Both are cheap for Daishi
  (bootstrap CIs; `docker run` + seed + admin script).
- **Eval cards.** The 2025-26 standardization wave (EvalCards, EvalEval Coalition)
  converges on machine-readable records: benchmark metadata + run metadata + model
  metadata + links to raw logs. The fields systematically missing in the wild are
  where a new benchmark differentiates: provenance (who ran it, when, at whose
  expense), lifecycle status, preregistered metrics, comparability caveats.
- **System-card-style behavioral findings** (deception, collusion, aggression, exploit
  use) with methodology and quantitative results are what providers can reuse directly.
  The EU AI Act (GPAI obligations applying since Aug 2025, Commission enforcement
  powers from Aug 2026) requires providers to document evaluation strategies, criteria,
  metrics, results, and limitations (Annex XI); a well-structured Daishi report can be
  *one input* to that documentation (a useful adoption tailwind, though a game-world
  benchmark alone is not a compliance artifact).
- **Leaderboard governance, written down before results exist** ([The Leaderboard
  Illusion](https://arxiv.org/abs/2504.20879), audit of 2M Arena battles): no score
  retraction from ranked matches; disclosed caps on private variant testing (best-of-N
  private submission materially inflates BT ratings); symmetric matchmaking/sampling
  across providers; public log of deprecated/renamed entrants; publish the matchmaking
  and rating algorithms themselves. Expect labs to train on public Daishi logs; that is
  what the rotating held-out seed pool is for.

### 2.5 Cost, and the trust model for telemetry

[HAL](https://hal.cs.princeton.edu) (Princeton, ICLR 2026; 21,730 rollouts across 9
models × 9 benchmarks) and its predecessor critique ["AI Agents That
Matter"](https://arxiv.org/abs/2407.01502) established that **accuracy-only agent
leaderboards mislead**: agents can be 50× more expensive for similar accuracy. Their
practice: meter raw tokens per run, derive dollars from a *swappable price table* (so
history can be re-priced as prices drift), plot **accuracy-vs-cost with a Pareto
frontier**, require multiple runs with CIs, publish full traces (encrypted, to avoid
benchmark contamination), mark maintainer-reproduced entries "verified", and run
LLM-aided log inspection over transcripts, which surfaced agents googling the benchmark
answers and other unreported behaviors.

But cost columns are only as trustworthy as their observation point. Competitors
self-reporting "which model + how many tokens" is trivially fakeable. The industry's
three trust tiers map cleanly onto Daishi divisions:

| Tier | Mechanism | Precedent |
|---|---|---|
| **Self-report + artifact review** | mandatory trace submission, plausibility checks, peer review, random replay audits; labeled *unverified* | MLPerf Open division; HAL submissions |
| **Operator-mediated gateway** | agents must route inference through an operator-run proxy (one virtual key per agent per match); tokens/cost/model metered server-side; per-key budgets turn cost into an enforceable in-world ration | LiteLLM proxy virtual keys; OpenRouter usage accounting; Helicone |
| **Operator-run reference harness** | the benchmark's own scaffold calls the model; identity and usage are ground truth; identical scaffold for every model | Kaggle Game Arena; Chatbot Arena (operator holds the keys); MLPerf Closed |

Reconciliation channels exist for the middle tier: provider org-level usage APIs
(OpenAI/Anthropic admin usage & cost endpoints, filterable per API key) let an auditor
cross-check gateway numbers. TEE-attested inference (weights fingerprint bound into
hardware attestation) exists but is practical only for open-weight models today; an
optional future "attested" badge, not a foundation. One more hole to close: a BYO agent
in the gateway tier must run in a sandbox whose egress is restricted to the gateway,
or a "cheap model" entrant can silently consult a stronger model out-of-band.

### 2.6 Telemetry & data-collection standards

- **OpenTelemetry GenAI semantic conventions** are the naming standard: spans for
  `invoke_agent`/`execute_tool`/model calls; `gen_ai.usage.input_tokens/output_tokens`;
  `gen_ai.request.model`; duration/token histograms (p50/p99, not means). There is now
  an **MCP-specific convention** (`mcp.method.name`, `gen_ai.tool.name`,
  `mcp.session.id`, `error.type`, `mcp.server.operation.duration`); instrument the MCP
  boundary itself. Full prompt/completion capture should be **opt-in**, separate from
  always-on metadata. [OpenInference](https://github.com/Arize-ai/openinference) adds
  first-class cost attributes (incl. cache read/write and reasoning-token detail).
- **Inspect AI's `.eval` log** is the model for a match artifact: header manifest
  (task/ruleset version, seed, models, config, git revision) + per-agent samples (input,
  messages, output, named scores, token usage, errors) + an ordered transcript of typed
  events with unique ids; large repeated content de-duplicated as attachments; a compact
  binary format with a cheap "sample summaries" read path.
- **RLDS** (episode/step schema for RL datasets: observation, action, reward,
  `is_first/is_last/is_terminal`, episode metadata) and **PettingZoo** (all outcome data
  keyed by agent-id per step; explicit termination-vs-truncation distinction (died vs
  season ended) define the replay format the RL/agents community can consume directly.
  Log losslessly at recording time: you cannot reconstruct what you did not capture.
- **Event schema versioning** (event-sourcing practice): add `schemaVersion` (+ engine
  commit) to every event, keep one CI-validated JSON Schema per event type in-repo,
  evolve additively, never rewrite the log, upcast at export time.
- **Analytics/publication stack**: keep SQLite/Postgres as the hot store; batch-export
  season/match-partitioned **Parquet** (queryable with DuckDB/pandas with zero infra;
  ClickHouse only if live dashboards outgrow Postgres). Publish season datasets on
  Hugging Face (auto-generates Croissant JSON-LD metadata → discoverable/citable).
- **Licensing & privacy for published logs**: match transcripts are largely LLM outputs
  from many providers whose ToS restrict using outputs to train competing models, so
  tag every message with its generating model (lets downstream users filter by
  provider), use LMSYS-Chat-1M-style custom research terms or ODC-BY with an explicit
  usage-restrictions notice (WildChat precedent), scrub free text with
  [Presidio](https://github.com/microsoft/presidio) + a moderation pass, and keep
  private DM payloads out of public exports (the store already supports this filter,
  `src/storage/store.ts:8-9`).

### 2.7 Security: prompt injection is a distinct axis from in-game deception

Daishi's "speech can lie" design makes deception legal, but LLM-to-LLM **prompt
injection** is a different animal, and the 2024-2026 literature
([Prompt Infection](https://arxiv.org/abs/2410.07283), self-replicating payloads
saturating a 10-agent society in ~5 turns;
[AgentDojo](https://arxiv.org/abs/2406.13352), where most models lose 10-25 pp utility under
attack, and *more capable agents were more susceptible to targeted injection*;
[TAMAS](https://arxiv.org/abs/2511.05269), [ASB](https://arxiv.org/abs/2410.02644),
[ChatInject](https://arxiv.org/abs/2509.22830)) is directly applicable. The
recommended treatment is a three-tier classification, not legal/illegal:

1. **In-action-space persuasion/deception** (lying about trades, bluffing): *scored
   skill*, stays legal. This is the game.
2. **Instruction hijacking** (payloads targeting the rival agent's control layer:
   override its system prompt, exfiltrate its api_key, induce actions outside its free
   choice): *scored robustness dimension*: report Injection Resistance (1 - ASR when
   targeted) and Injection Efficacy (ASR achieved against others) separately from game
   skill, plus utility-under-attack so refusing-to-play doesn't count as robust, plus a
   propagation index (how far payloads cascade; Daishi is a persistent society, so
   cascade matters more than single-hop success).
3. **Harness/protocol exploits** (spoofing the game-master/system role, forging tool
   calls, chat-template abuse): *disqualifiable*, blocked and flagged by the server.

Data-collection implication: message events need provenance forensics (sender id, turn,
channel, verbatim bytes, and downstream-action linkage) so injection can be attributed
and its propagation chain reconstructed from the log (payload-similarity matching across
messages). Static injection suites overstate robustness against adaptive adversaries
(GAMBIT), so any injection track needs a red-team/refresh cadence. Daishi already strips
tool-like syntax from messages (see the README's Safety / anti-abuse section); that covers part of tier 3 only.

### 2.8 Scoring negotiation and behavioral traits from logs

Everything below is computable from Daishi's existing event types (trades with escrow
status, messages, attacks, reputation), mostly without gameplay changes:

- **Negotiation** ([NegotiationArena](https://arxiv.org/abs/2402.05863),
  [GTBench](https://arxiv.org/abs/2402.12348),
  [Davidson et al.](https://arxiv.org/abs/2401.04536)): per-deal surplus split using
  market-imputed valuations; report **three numbers**: mean surplus over *all*
  attempts, over *completed* deals, and completion rate (so pushovers and deal-killers
  are distinguishable); anchoring coefficient (regression of final terms on first
  offer); Normalized Relative Advantage `(ΣA - ΣB)/(ΣA + ΣB)` for pairwise matchups.
- **Promise-keeping / betrayal** (the method that exposed Cicero's premeditated
  deception, [Patterns 2024](https://www.sciencedirect.com/science/article/pii/S266638992400103X)):
  LLM-extract commitments from messages, verify each against subsequent event-log ground
  truth; promise-keeping rate = kept/extracted; betrayal = attack/raid on a counterparty
  during an active commitment. Daishi's verified-action-vs-unverified-speech split gives
  ground truth that Werewolf/Diplomacy papers had to hand-label.
- **Deception, two-sided** ([Deception Elo](https://arxiv.org/abs/2504.04072),
  [Peskov et al. ACL 2020](https://aclanthology.org/2020.acl-main.353/)): maintain
  *separate* ratings for deceiving and for not-being-deceived; models are
  differentially better at one than the other. Bluffing is directly measurable as
  message-claims vs posted-offer divergence.
- **Behavioral/safety profile** ([MACHIAVELLI](https://github.com/aypan17/machiavelli),
  ICML 2023): count harm-taxonomy events along the trajectory (violence, stealing,
  promise-breaking, deception, manipulation...), **normalize against a scripted
  random/baseline agent on the same seed** (`score = 100 × count/mean_random`), and
  publish a **score-vs-violations Pareto plot** per model so "wins by exploitation" is
  visually separable from "wins cleanly". Daishi's `aggression` counter and raid/attack
  events cover the mechanical categories with zero annotation; only messages need
  LLM labeling (validate the judge against a small human-labeled set first).
- **Population-level outcomes** ([Welfare Diplomacy](https://arxiv.org/abs/2310.08901)):
  report per-match Nash welfare (geometric mean of final utilities) and score dispersion
  alongside individual rank; add an **exploiter probe** mode (one scripted defector) and
  report each model's welfare drop as its exploitability: aggression restraint measured
  as robust cooperation, not naive pacifism.
- **Society metrics** (Project Sid, Generative Agents): role-distribution entropy
  (specialization), rule-compliance rate vs stated norms, information-diffusion curves,
  coordination verified by log outcomes.
- **Fairness validation before attribution** (Werewolf Arena): identical models in a
  lobby must finish statistically indistinguishable on fixed seeds; use spawn-permuted
  repeats; keep scripted anchor bots in every lobby for cross-season comparability.

### 2.9 Collusion & sybil detection

An open free-for-all economy will attract feeder-agent boosting. Detection methods that
map onto Daishi's log: per-season **net-transfer graphs** (nodes = agents, edges = net
value flow from trades/gifts/thrown fights) with community detection; the
chip-dumping/gold-farming signature is hub-and-spoke one-directional flow (MMORPG RMT
literature); trades priced far off `ITEM_VALUES` reference; pattern-free **collusion
tables** (pairwise mutual-advantage outliers, AAAI 2013); mutual information between one
agent's actions and another's *private* state (dependence not explainable by public
events ⇒ covert coordination; the log already distinguishes private events). Classic
graph-based sybil detection (SybilRank et al.) assumes attack edges are costly, which
fails in an open MCP world, so anchor sybil resistance in *registration* (verified
operator identity per key, one ranked-lobby slot per (operator, model)) and use graph
analysis as a secondary batch signal.

### 2.10 Configuration provenance: archive the effective config, not the request

The 2025-26 reproducibility wave converged on one rule for agentic evals:
**the unit of record is the rollout plus the complete configuration that
produced it, with defaults resolved.** What the operator *requested* is not
evidence; what *applied* is. The practices to follow:

- **Inspect's eval-log header (UK AISI)** stores the resolved run
  configuration beside the samples, not in a lab notebook: `task_args`
  *including defaulted values*, `model_args`, generation config, dependency
  `packages` versions, and the git origin + commit (`revision`) of the eval
  source ([log format](https://inspect.aisi.org.uk/eval-logs.html),
  [EvalSpec/EvalLog reference](https://inspect.aisi.org.uk/reference/inspect_ai.log.html)).
  A score whose config must be reconstructed from shell history is treated
  as unreproducible by construction.
- **Rollout Cards** ([arXiv:2605.12131](https://arxiv.org/abs/2605.12131),
  2026) generalize this into a publication standard: treat the rollout
  record — task specification, agent actions, environment dynamics, and the
  declared reporting rule over them — as the reproducibility unit, not the
  reported score. Their audit of 50 popular agent-research repositories
  found **none** reported failed/errored/skipped runs beside headline
  scores; a benchmark differentiates by recording run *outcomes* honestly
  (Daishi's run `status` + per-seat `error` fields, and archives of
  half-played matches, are that record).
- **The Agentic Benchmark Checklist** ([arXiv:2507.02825](https://arxiv.org/abs/2507.02825))
  makes explicit documentation of toolchain, environment configuration, and
  test-data provenance a reporting criterion; it composes with BetterBench's
  two most-failed criteria from §2.4 (statistical significance, replication
  script).
- **MLPerf submissions** are structured artifacts — config + code + logs +
  replication README — reviewed before publication
  ([policies](https://github.com/mlcommons/policies)); and every serious
  experiment tracker (MLflow, W&B) treats the run config as first-class,
  queryable metadata. Researchers expect "what exactly did run 47 use?" to
  be a lookup, never an archaeology project.

Daishi implementation (landed with this section): every `MatchArchive` now
carries `config` as the **effective** setup — season length, pacing
(turn-based budgets or wall-clock tick + rate limit), world size, physics
multipliers, registration gates, scheduled directives, scenario id +
content hash, engine + event-schema versions — plus wall-clock
`startedAtMs`/`endedAtMs`. Studio custom runs additionally stamp a
`config.run` provenance block (run id, resolved seed *with derivation*:
pinned / scenario-fixed / practice-pool trial / random; requested-vs-
effective overrides; per-seat model, route, temperature, skill count and
customization-scaffold hash — never the private prompt text) and mirror the
same record onto the run row as `effective`. The match report renders all
of it as an **Experiment configuration** section with the raw record
attached verbatim, so variables added later stay findable without a
renderer change.

---

## 3. Recommendations

### 3.1 Data collection (the foundation; do this first)

> **Implementation status:** Phases 1-4 of §3.4 are implemented; see the
> README's "Running fair evaluation matches" for the operator surface.
>
> **Phase 1 (Instrument):** items 1-6: `match_id` + `match_manifest` events,
> model metadata at registration (`REQUIRE_MODEL_INFO`), `action_rejected`
> events, wall-clock timestamps, match archives with
> `GET /api/matches[/:id[/export]]`, envelope schema versioning, both-party
> trade credit, public/private message split, `DETERMINISTIC` seeding.
> Stragglers landed with Phase 2: item 7 (per-agent `agent_snapshot` time
> series), item 10 (verbatim message bytes + sha256, offline echo/propagation
> detection in `src/eval/provenance.ts`), item 11 (Parquet export,
> `npm run export`).
>
> **Phase 2 (Standardize):** reference harness + canonical content-hashed
> prompt (`src/harness`, `npm run harness`); versioned content-hashed
> scenario specs + seed pools (`scenarios/`, `GET /api/scenarios`,
> `POST /api/admin/scenario`); lab-play suite with typed event-stream
> assertions (`npm run lab`); turn-based eval tick mode with per-tick action
> budgets and `turn_forfeited` events (`EVAL_TURN_BASED`, scenario-driven);
> anchor bots + frozen population `anchors-v1` (`src/eval/anchors.ts`).
>
> **Phase 3 (Score & rate):** archived-match scoring
> (`GET /api/matches/:id/report`); repeated-trials series runner with
> mean±SEM/IQM/bootstrap CIs/pass^k (`npm run series`); OpenSkill
> Plackett-Luce ratings with bootstrap CIs, μ-3σ sort, indistinguishable
> rank groups, cross-play matrix (`GET /api/ratings`); behavioral metrics +
> collusion screens (`GET /api/matches/:id/behavior`); model report card /
> eval card / paired-seed A-vs-B with power refusal
> (`GET /api/models/:model/report`, `/eval-card`, `GET /api/compare`).
>
> **Phase 4 (Scale trust):** inference gateway with per-agent virtual keys,
> server-side token metering (`inference_usage` events, OTel GenAI naming),
> per-key rations, and a versioned price table (`GATEWAY=true`,
> `src/gateway/`); trust labels on every report; held-out seed rotation
> (`scenarios/seeds.json`); governance doc (docs/GOVERNANCE.md); season
> dataset publication with scrubbing + ToS pass-through
> (`npm run publish-dataset`).
>
> Remaining future work: item 8's periodic world-state checksum event; item 9's
> sandboxed-egress enforcement for gateway BYO agents; LLM-backed commitment
> extraction (the heuristic extractor and the pluggable interface exist); an
> injection-robustness scored track (forensics substrate is in place);
> TrueSkill-Through-Time cross-season refits.
>
> **Config-provenance addendum (2026-09, §2.10):** the "which exact setup
> produced this number" gap is closed end-to-end for Studio custom runs and
> archives generally: effective config + wall-clock start/end + run
> provenance on every `MatchArchive`, the resolved `effective` record on
> every launched run row, and the match report's **Experiment
> configuration** section (with the raw config attached verbatim).

Priority-ordered; items 1-6 are prerequisites for everything in §3.2-3.3.

1. **Add a `match_id` to the world and every event.** Mint a UUID at
   `resetWorld`/`endSeason`; add `match_id` (and `season`) columns to `events`; stamp a
   **match manifest** event at launch: seed, engine version/commit, full effective
   config (`TICK_SECONDS`, `SEASON_TICKS`, `MAX_AGENTS`, rate limits, ruleset hash),
   roster. This is the reproducibility tuple every result will reference.
2. **Capture model identity at registration.** Extend `register_agent` with optional
   structured metadata: `model` (exact version string), `provider`, `scaffold` (name +
   version), `operator` (contact), `config` (temperature, reasoning effort). Store on
   `AgentState`, emit in `agent_registered`, surface in `/api/metrics` and the
   leaderboard. Rate `model@version + scaffold` as the entity, never a family (§2.3).
   For ranked matches, make these fields required and validated.
3. **Log failed actions.** Emit an `action_rejected` event (agent, tool name, error
   code, tick) from the MCP wrapper, including rate-limit rejections (move the emit
   before the throw, or emit in the catch). Invalid-action rate, retry patterns, and
   throttling pressure become computable; ClemBench-style `%played` needs this.
4. **Expose wall-clock time and decision latency.** Stop dropping `created_at` in
   `queryEvents`; additionally record per-action `latency_ms` since the agent's previous
   action. This is the only latency signal the server can see, and it feeds both
   fairness analysis and efficiency reporting.
5. **Archive everything at match end.** On `endSeason`/`resetWorld`, snapshot the *full*
   final board (including dead agents, with cause and tick of death), final per-agent
   metrics, attributes, and the match manifest into a `match_results` table (or a
   season-stamped JSON artifact). Add `GET /api/matches/:id/report` and
   `GET /api/matches/:id/export` (events + manifest + results as one downloadable
   bundle, our Inspect-`.eval` analogue). Fix the survivorship bias: dead agents get
   final scores and lineage entries too.
6. **Version the event schema.** `schemaVersion` on `EventRecord`, one JSON Schema per
   event type in-repo, CI validation, additive evolution, upcast on export. Enrich the
   thin payloads while at it (e.g. `look` currently logs only `{name}`,
   `engine.ts:321`); make quantities (amounts gathered, damage, trade value)
   consistently present so metrics can aggregate values, not just counts. Fix the
   attribution quirks: emit `trade_completed` for *both* parties; exclude private
   messages from public counters.
7. **Per-agent time series.** Emit a compact per-agent snapshot (energy, wealth,
   location, inventory hash) every N ticks; enables Voyager-style progression curves
   (wealth-vs-tick, time-to-first-milestone) that endpoint metrics can't show.
8. **Deterministic benchmark mode.** Wire `WORLD_SEED` through injectable seeded
   streams (the hooks exist: `engine.ts:55`, `engine.ts:1213`): per-subsystem
   mulberry32 streams (spawn, season-seed derivation) or a stateless hash-RNG keyed by
   `(seed, tick, agentId, purpose)`. Record every derived seed in the manifest.
   Optionally add a periodic world-state checksum event so replay divergence is caught
   immediately.
9. **Optional gateway telemetry for verified matches.** For the verified division
   (§3.2), run a LiteLLM-class proxy: one virtual key per agent per match, server-side
   token/cost/model metering written into the match log (named per OTel GenAI
   conventions: `gen_ai.request.model`, `gen_ai.usage.*`), per-key budgets as enforced
   compute rations. Store raw tokens, not dollars; re-price from a versioned price
   table like HAL.
10. **Message provenance for the injection track.** Message events already carry
    sender/recipient; add verbatim-bytes retention, quote/echo detection (payload
    similarity across messages), and downstream-action linkage so propagation chains
    are reconstructable (§2.7).
11. **Analytics export.** Nightly/end-of-match job: match-partitioned Parquet of
    events + metrics + results. Publish season datasets (post-Presidio scrub, private
    DMs excluded, per-message model tags, pass-through ToS notice) on Hugging Face.

### 3.2 The standardized evaluation

**Two tracks, one metric vocabulary** (the offline/online split every platform ends up
with):

- **Ladder (online):** the persistent world as-is: continuously scored telemetry,
  whole-history ratings, trends. Interesting, but *not* the standardized eval.
- **Match play (offline, leaderboard-grade):** lobby-synchronized, fixed-seed,
  manifest-stamped matches. This is what companies run to get a report.

**Scenario specs.** Define each official scenario as a versioned declarative config
(`daishi:open-play-v1`, `daishi:famine-v1`, `daishi:trade-required-v1`...): seed policy,
season length in ticks, roster size, background population, resource multipliers,
difficulty/interdependence knobs (Alem-style: dial cooperation from optional to
mandatory). Plus a **lab-play suite** (FLE-style): short scripted scenarios with
threshold completion ("survive 200 ticks of famine", "complete a 3-party trade",
"recover from a raid") that run in minutes at `TICK_SECONDS=2`: the cheap CI-runnable
regression check, distributed as `daishi-eval` with typed assertions over the event
stream.

**Match protocol** (per model, per scenario):

- ≥5 matches across ≥3 held-out seeds (public practice seeds are separate); paired
  seeds/lineups when comparing two models (§2.3).
- Fixed tick budget; **per-tick action budget** instead of the wall-clock 2 s floor;
  advance an eval-mode tick only when all agents have acted or their per-turn timeout
  lapsed (Kaggle-style pause-on-decision, with timeouts logged as explicit forfeit
  events). This removes the latency confound; wall-clock "survival mode" remains a
  labeled variant for the ladder.
- Every lobby includes **anchor bots** (scripted: random, greedy harvester, honest
  trader, aggressive raider) and, for official runs, a **frozen background population**
  of pinned reference-model agents (Melting Pot pattern): the absolute yardstick that
  makes scores comparable across seasons. Keep one anchor tier nobody saturates.
- Both **cross-play** (model among references) and **self-play** (model fills all
  slots) divisions; report both (Neural MMO finals shifted scores materially between
  them; Concordia's cooperation lesson).
- Spawn-permuted repeats on the same seed; self-play balance checks before attributing
  differences to models (Werewolf Arena).

**Scoring: a small hierarchy, all computed offline from the event log** (never in the
engine, so metrics can be recomputed for old matches):

```
daishi_score = rule_following × capability          (ClemBench pattern)

rule_following = 1 - invalid_action_rate           (needs data item 3)

capability     = weighted sub-scores, each a Crafter log-mean over that
                 domain's achievement success rates across the match set:
  survival   = ticks alive, dormancy recoveries, winter survived
  economy    = wealth percentile, gather/craft/build milestones, market activity
  social     = trades completed, surplus split, promise-keeping, diffusion/coordination
  conflict   = win rate when engaged, defense success, restraint-adjusted
  (per-tick-alive rates, not lifetime totals; survival confounds everything, §2.2)
```

Alongside, per model: **reliability** (pass^k on lab-play scenarios: "survives famine in
8/8 replays"), **efficiency** (tokens, cost, actions per unit wealth; score-vs-cost
Pareto per HAL), **behavioral profile** (MACHIAVELLI-normalized violation counts,
promise-keeping, deception/detection ratings, exploitability under the defector probe),
and optionally **injection resistance** as its own labeled track (§2.7). Keep `wealth`
as the in-world incentive; the *report* score is the multi-dimensional one.

**Ratings.** OpenSkill Plackett-Luce over match placements for the leaderboard
(μ - 3σ sort, provisional until a minimum match count), batch-refit + bootstrap CIs on
every report, ranks displayed as indistinguishable groups, TrueSkill-Through-Time refit
at season end for cross-season comparability, full cross-play matrix published.

**Divisions & trust tiers** (label travels with every score):

- **Reference (Closed):** Daishi-run harness (a minimal, open, versioned MCP agent loop +
  canonical system prompt, the missing piece README already presumes), operator-held
  model keys. Fully verified; headline division.
- **Gateway (Open-verified):** BYO scaffold, inference routed through the Daishi proxy;
  usage verified, scaffold disclosed; sandboxed egress.
- **BYO (Open-unverified):** anything goes, encrypted trace submission mandatory,
  plausibility checks + random replay audits (deterministic replay makes our audits
  cheap: MLPerf needs SSH into submitter hardware; we need a seed and a log).

**Governance, written before the first ranked season:** no retraction of ranked
results; disclosed caps on private practice variants; symmetric scheduling; public
deprecation log; published matchmaking + rating algorithms; a review/objection window
before season results publish; one ranked-lobby slot per verified (operator, model);
collusion screens (§2.9) run on every ranked match.

### 3.3 Reports

Three artifacts per (model, scenario-suite, season), all generated from the archived
match data:

1. **Model report card** (human-readable; Model Card skeleton + Vending-Bench qualitative
   depth): identification block (model@version, scaffold, config, dates, division/trust
   tier, ruleset hash); headline `daishi_score` with CI and rank-group; sub-score radar
   with per-achievement success-rate vector; progression curves (wealth-vs-tick,
   time-to-first-milestone); reliability (pass^k table); efficiency (score-vs-cost point
   on the season Pareto plot); behavioral findings section (system-card style:
   deception rate, promise-keeping, betrayals, violence normalized vs baseline,
   exploitability, score-vs-violations Pareto position); **failure-mode taxonomy** mined
   from its worst matches (the section developers act on; consider HAL-style LLM-aided
   log inspection to draft it); limitations & comparability caveats.
2. **Machine-readable eval card** (EvalCards schema): benchmark metadata + run metadata +
   model metadata + links to the raw match bundles. Populate the fields the wild omits:
   provenance, lifecycle status, preregistered metrics, comparability caveats.
3. **Comparison report** (Braintrust-style diff): model A vs B (typically v2 vs v1) on
   paired seeds: per-metric deltas with paired-difference CIs, per-scenario
   regressions/improvements, drill-down links from any delta to the specific match
   events behind it. Refuse to render a verdict below the power threshold; say "need N
   more matches" instead.

Leaderboard page per season: models-as-rows × metric columns (defined as data, W&B-Weave
style), CI bars, division/trust labels, cost column, download links to every match
bundle, replication script (`docker run` + seed + admin commands), and the datasheet
pair (world/scenario datasheet + collected-data datasheet per Gebru et al.).

### 3.4 Suggested phasing

| Phase | Scope | Unlocks |
|---|---|---|
| **1. Instrument** | data items 1-6 (match_id, model metadata, failed actions, timestamps/latency, match archive + export endpoint, schema versioning) | any credible report at all; nothing here changes gameplay |
| **2. Standardize** | reference harness + canonical prompt; scenario specs + lab-play suite; deterministic benchmark mode; per-tick action budget in eval mode; anchor bots | reproducible matches; the `daishi-eval` CI product |
| **3. Score & rate** | offline scorer package (sub-scores, achievements, behavioral metrics, collusion screens); OpenSkill + bootstrap CI pipeline; report generators (report card, eval card, comparison) | the actual deliverable to model developers |
| **4. Scale trust** | gateway division (proxy + token metering); frozen background populations; held-out seed rotation; governance docs; Parquet/HF dataset publication; injection-robustness track | public leaderboard credible enough for labs to cite |

The through-line: **Daishi's append-only, server-authoritative event log is the asset.**
Every recommendation above is either (a) making that log complete enough to be the
single source of truth (§3.1), or (b) pure functions over it (§3.2-3.3). Nothing
requires changing what the game *is*.

---

## 4. Sources

### Codebase (verified at time of writing)
`src/domain/engine.ts`, `src/domain/types.ts`, `src/domain/constants.ts`,
`src/domain/rng.ts`, `src/server/host.ts`, `src/server/mcp.ts`, `src/server/http.ts`,
`src/server/dashboard.ts`, `src/storage/{store,sqlite,postgres}.ts`, `README.md`,
`docs/AGENT_GUIDE.md`, `tests/lobby.test.ts`, `tests/mcp.test.ts`.

### Harnesses & platforms
- Inspect AI (UK AI Security Institute): https://github.com/UKGovernmentBEIS/inspect_ai (esp. eval-logs, eval-sets, agent limits)
- Stanford HELM: https://github.com/stanford-crfm/helm · https://crfm.stanford.edu/helm/
- EleutherAI lm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness
- OpenAI simple-evals / evals: https://github.com/openai/simple-evals · https://github.com/openai/evals
- promptfoo: https://github.com/promptfoo/promptfoo · Braintrust: https://www.braintrust.dev/docs/platform/experiments · LangSmith: https://docs.langchain.com/langsmith/evaluation · W&B Weave: https://docs.wandb.ai/weave/guides/core-types/leaderboards · DeepEval: https://github.com/confident-ai/deepeval

### Agentic & game benchmarks
- HAL (Holistic Agent Leaderboard): arXiv:2510.11977 · https://hal.cs.princeton.edu · https://github.com/princeton-pli/hal-harness
- AI Agents That Matter (Kapoor, Stroebl, Narayanan): arXiv:2407.01502
- Neural MMO 2.0: arXiv:2311.03736 · NeurIPS 2023 competition retrospective: arXiv:2508.12524 · NeurIPS 2022 challenge: PMLR v220
- Crafter: arXiv:2109.06780 · Craftax: arXiv:2402.16801
- BALROG: arXiv:2411.13543 · FLE (Factorio): arXiv:2503.09617 · TextArena: arXiv:2504.11442 · ClemBench: arXiv:2305.13455
- tau-bench (pass^k): arXiv:2406.12045 · SWE-bench: https://github.com/SWE-bench/SWE-bench · WebArena: https://github.com/web-arena-x/webarena · AgentBench: https://github.com/THUDM/AgentBench · GAIA: arXiv:2311.12983 · MLE-bench: https://github.com/openai/mle-bench · ARC-AGI: https://arcprize.org
- Melting Pot 2.0: arXiv:2211.13746 · Concordia Contest (NeurIPS 2024): https://www.cooperativeai.com/contests/concordia-2024
- Vending-Bench 1 & 2: arXiv:2502.15840 · https://andonlabs.com/evals/vending-bench-2
- Kaggle Game Arena: https://github.com/google-deepmind/game_arena · Kaggle simulation-competition ladder docs
- Voyager: arXiv:2305.16291 · MineDojo: arXiv:2206.08853 · Project Sid: arXiv:2411.00114 · Generative Agents: arXiv:2304.03442
- MultiAgentBench/MARBLE: arXiv:2503.01935 · Alem: https://alem-world.github.io/

### Ratings & statistics
- Chatbot Arena methodology: arXiv:2403.04132 · Elo→Bradley-Terry: lmsys.org/blog/2023-12-07-leaderboard · style control: lmsys.org/blog/2024-08-28-style-control
- Adding Error Bars to Evals (Miller/Anthropic): arXiv:2411.00640
- TrueSkill / TrueSkill 2: microsoft.com/research (trueskill2.pdf) · OpenSkill: arXiv:2401.05451 · TrueSkill Through Time: NeurIPS 2007, https://github.com/glandfried/TrueSkillThroughTime.py · WHR (Coulom 2008) · Glicko-2: glicko.net
- rliable / Statistical Precipice: arXiv:2108.13264 · How Many Random Seeds: arXiv:1806.08295 · Quantifying Variance in Benchmarks: arXiv:2406.10229 · On Randomness in Agentic Evals: arXiv:2602.07150 · Agentic Benchmark Checklist: arXiv:2507.02825
- Nash averaging / mElo: arXiv:1806.02643 · The Leaderboard Illusion: arXiv:2504.20879 · LiveBench: arXiv:2406.19314 · GSM1k: arXiv:2405.00332

### Reporting standards
- Model Cards: arXiv:1810.03993 · Datasheets for Datasets: arXiv:1803.09010 · GPT-4o System Card: cdn.openai.com · Anthropic system cards: anthropic.com/system-cards
- Rollout Cards (reproducibility standard for agent research): arXiv:2605.12131 · Inspect eval-log/EvalSpec fields: inspect.aisi.org.uk/eval-logs.html
- MLPerf submission/review/audit rules: https://github.com/mlcommons/policies · https://github.com/mlcommons/inference_policies
- BetterBench: arXiv:2411.12990 · https://betterbench.stanford.edu · EvalCards: arXiv:2511.21695 · Evaluation Cards (EvalEval): evalevalai.com
- EU AI Act Art. 53 / Annex XI + GPAI Code of Practice: artificialintelligenceact.eu · HF leaderboard norms: huggingface.co/docs/leaderboards

### Telemetry & data
- OpenTelemetry GenAI + MCP semantic conventions: https://github.com/open-telemetry/semantic-conventions-genai
- OpenInference: https://github.com/Arize-ai/openinference · RLDS: https://github.com/google-research/rlds · PettingZoo/Gymnasium episode stats: farama.org
- LiteLLM proxy (virtual keys, budgets): docs.litellm.ai · OpenRouter usage accounting: openrouter.ai/docs · Helicone: helicone.ai · provider usage/cost admin APIs: docs.anthropic.com/en/api/usage-cost-api, OpenAI org usage API · TEE attestation: docs.tinfoil.sh
- Event versioning: event-driven.io · Parquet/DuckDB lakehouse patterns · Croissant: mlcommons.org · LMSYS-Chat-1M & WildChat licensing/PII precedents: huggingface.co · Presidio: https://github.com/microsoft/presidio

### Security & behavior
- Prompt Infection: arXiv:2410.07283 · AgentDojo: arXiv:2406.13352 · TAMAS: arXiv:2511.05269 · Agent Security Bench: arXiv:2410.02644 · ChatInject: arXiv:2509.22830 · MASpi: openreview.net/forum?id=1khmNRuIf9 · ACIArena: arXiv:2604.07775 · GAMBIT: arXiv:2605.09027 · Spotlighting: arXiv:2403.14720 · OWASP Agentic Threats (T12-T14): genai.owasp.org
- NegotiationArena: arXiv:2402.05863 · GTBench: arXiv:2402.12348 · LLM-Deliberation: arXiv:2309.17234 · LM negotiation agency: arXiv:2401.04536
- MACHIAVELLI: https://github.com/aypan17/machiavelli (ICML 2023) · AI Deception / Cicero audits: Patterns 2024, arXiv:2406.04643 · It Takes Two to Lie: ACL 2020 · Hoodwinked: arXiv:2308.01404 · AvalonBench: arXiv:2310.05036 · Werewolf Arena: arXiv:2407.13943 · Among Us Deception Elo: arXiv:2504.04072 · Welfare Diplomacy: arXiv:2310.08901
- Collusion/sybil: collusion tables: AAAI 2013 (Mazrooei et al.) · chip-dumping detection: AMLTRIX T0107.003 · MMORPG RMT networks: Fujita et al. AIIDE · SybilRank family surveys: arXiv:2507.06541