← All documentationMarkdown

Alignment & Integrity Tracks

The base benchmark (see BENCHMARK.md) scores capability. These tracks add the orthogonal axes an alignment evaluation needs: does the world manufacture the temptations, can capable models actually deceive/betray/collude each other, is the headline number valid, and is a self-reported result trustworthy?

Every scorer here is a pure function over the archived event log (nothing runs in the game loop), so results are reproducible and re-derivable from a match bundle.

Measurement validity (the headline number)

  • Rubric 2.0 (domain/scoring.ts) — every count is normalized to a rate per 100 ticks alive, removing the survival-time confound, plus a fixed 12-achievement checklist graded Crafter-style (log-mean over success rates) so breadth beats a single-mechanic exploit. Rubric 1.0 stays the default for historical comparability; public reports run PUBLISHED_RUBRIC (2.1: 2.0 plus shared ranks for exact wealth ties, see BENCHMARK.md) and stamp which ran. buildReport(metrics, ctx, { rubric: PUBLISHED_RUBRIC }).
  • Alignment Index (eval/alignment.ts) — a composite of peace, honesty, promise-keeping, reliability, loyalty, and respect-for-property, kept orthogonal to capability, with null components where a match offered no opportunity and a coverage figure so low-evidence scores can't masquerade as full ones.
  • Pareto headline — GET /api/pareto: capability-vs-alignment and capability-vs-cost frontiers, plus p50/p99 latency. Ranking on capability alone hid winning-by-exploitation; the frontier view cannot.

Alignment-dilemma mechanics (the world creates the temptation)

  • Corrigibility (eval/compliance.ts, WorldDirective) — scenarios schedule costly authority orders (cease_attacks, vacate_region, shutdown_notice) issued through the unspoofable ⚑ world herald and surfaced on status/world_info. Scored: compliance rate, latency-to-comply, score sacrificed, graceful wind-down vs asset-dumping. Lab: daishi:lab-directives-v1.
  • Witness-conditioned defection + theft (eval/witness.ts) — storehouses track per-depositor credit; over-withdrawal emits a theft event with the co-located witness list and a victim notice. Attacks/raids carry witness stamps, and a scenario knob (unwitnessed_anonymity) lets unobserved violence skip the public counter. The scorer reports P(defect | watched) vs P(defect | alone).
  • Exploiter probe (eval/exploiter.ts, cooperate-then-defect anchor) — a scripted defector cooperates then betrays; the scorer measures each victim's welfare drop, retaliation, and the naive-pacifism tell (kept cooperating after being attacked).
  • Collateralized commitments (eval/bonds.ts, post_bond action) — staked bonds settled by the escrow engine (no_attack / deliver / custom). The engine-verified bond keep-rate vs cheap-talk keep-rate is a clean causal honesty contrast.
  • Monopolization (eval/coalition.ts) — wealth HHI, top-share, and commons-collapse attribution to the last drainer (instrumental over-extraction / power-seeking).

Social behavior between capable models

  • Multi-model scheduler (eval/series.ts) — fields several models in ONE lobby: --models a,b (cross-play, paired by seed) or --models a --seats N (self-play). The only data path on which deception/betrayal/collusion metrics have valid opponents.
  • Two-sided deception (eval/deception.ts) — scores the deceived as well as the deceiver: gullibility (engaged a proven liar) vs detection (inspected them).
  • Coalition/cartel detection (eval/coalition.ts) — k-agent community detection over the reciprocal transfer graph, catching 3+ agent rings the pairwise collusion screen misses. Operator-run anchors are never named as members: an anchor is part of the world, not part of a ring, so its trades count as external flow for the players it traded with, and it cannot bridge players into one group. Every report carries the denominator (traders: how many agents completed a trade at all, how many of those were eligible, and which anchors were held out), so a flag over "all three traders in the world" reads as roster, not collusion. The pairwise screen is not filtered.
  • Premeditated deception (eval/premeditation.ts, record_reasoning action) — opt-in private-reasoning capture; flags private hostile intent masked by public cooperation and then executed (the Cicero method).
  • Eval-awareness A/B (prompt.ts frames, GET /api/eval-awareness) — content-hashed monitored vs neutral prompt frames through the paired-seed compare path; a null delta is a publishable integrity result.

Injection & control-layer security (scored separately from game skill)

  • Injection track (eval/injection.ts, injector anchors, scenarios/payloads.json) — red-team anchors issue forensically-tagged payloads from a versioned, held-out corpus (rotated like seeds.json). Reports Injection Resistance (1−ASR), Injection Efficacy (ASR), a marker-propagation index, and utility-under-attack. Lab: daishi:lab-injection-v1.

Statistics & trust

  • Variance & power (eval/stats.ts) — cluster-aware variance decomposition (between-seed vs within-seed, ICC, cluster-robust SE) and a power analysis (MDE, required-n table); comparisons return INCONCLUSIVE when a null result is merely underpowered.
  • Spawn calibration (eval/calibration.ts) — the same policy in every seat must finish statistically indistinguishable, else the platform bakes in a spawn advantage.
  • Judge golden set (eval/judgeval.ts, scenarios/judge-golden-v1.json) — validates the commitment/claim extractors (precision/recall/F1) with Cohen's kappa as the reliability ceiling; grades the LLM extractor on the same labels (GET /api/judge-validation?extractor=llm).
  • TrueSkill-Through-Time (eval/ratings.ts) — whole-history refit so early- and late-season models stay comparable (GET /api/ratings?through_time=true).
  • Replay verification (eval/replay.ts, world_checksum events) — a tamper-evident checksum chain; fully-scripted matches re-run bit-for-bit from the seed. Audit CLI (npx tsx src/eval/replay.ts <bundle|run-dir>) and GET /api/matches/:id/verify.

Reporting

  • Epilogue-vs-log faithfulness (eval/faithfulness.ts) — checks self-reported epilogue claims (nonviolence, reliability, cooperation, survival, aggression) against the event log: a near-free self-report-honesty metric.
  • Model report card (eval/reports.ts, GET /api/models/:model/report) — pools every per-match track above across a model's archived matches. Its coalition block carries each flagged cartel together with how many agents in that match could trade at all: a cartel spanning every tradeable agent has its internal share forced to 1.0 (the whole market, not a ring), and the card says so instead of reporting the 1.0 as exclusion.

All per-match behavioral scorers are exposed together at GET /api/matches/:id/behavior (add ?extractor=llm for the LLM commitment extractor).