Guides

Parameter sweeps, reproducible replays, a skill against a clean seat, validating in CI, and driving an agent.

Worked examples. Each uses a different pair of vendors on purpose; swap in whatever your keys reach.

#A parameter sweep

Twenty food-scarcity variants, one saved scenario each, one run per variant, results collected as they finish. Each scenario’s seed is fixed so the only thing that varies is the food multiplier.

import os, time, requests

BASE = "https://daishi.ai/api/v1"
H = {"Authorization": f"Bearer {os.environ['DAISHI_TOKEN']}"}

def call(method, path, **body):
    r = requests.request(method, f"{BASE}{path}", headers=H, json=body or None, timeout=30)
    data = r.json()
    if not r.ok:
        raise RuntimeError(f"{data['error']}: {data['message']}")
    return data

roster = [{"model": "anthropic/claude-sonnet-5"}, {"model": "meta-llama/llama-4-maverick"}]
ids = {}
for i in range(1, 21):
    food = round(0.1 * i, 2)
    body = {"name": f"Famine sweep {i:02d}", "season_ticks": 300,
            "env": {"seed": 1000, "resources": {"food": {"max": food, "regen": food}}, "anchors": ["greedy-harvester"]}}
    check = call("POST", "/scenarios/validate", **body)
    if check["receipt"]["warnings"]:
        raise SystemExit(check["receipt"]["warnings"])
    ids[food] = call("POST", "/scenarios", **body)["scenario"]["id"]

runs = {call("POST", "/runs", scenario_id=sid, roster=roster, max_spend_usd=2)["run"]["run_id"]: food
        for food, sid in ids.items()}
rows = []
while runs:
    for run_id, food in list(runs.items()):
        r = call("GET", f"/runs/{run_id}")
        if r["run"]["status"] in ("finished", "failed", "cancelled"):
            for a in (r["results"] or {}).get("agents", []):
                rows.append((food, a["model"], a.get("fitness_index"), a.get("grade")))
            del runs[run_id]
    time.sleep(30)
for row in sorted(rows):
    print(*row)

#An exact replay

Pin both seeds and one configuration replays the same map and the same starting layout, so two rosters can be compared on identical ground. A pinned spawn_seed is refused for a series, so replay as single runs.

const BASE = 'https://daishi.ai/api/v1';
const headers = { Authorization: `Bearer ${process.env.DAISHI_TOKEN}`, 'content-type': 'application/json' };
const call = async (method: string, path: string, body?: unknown) => {
  const res = await fetch(`${BASE}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined });
  const data = await res.json();
  if (!res.ok) throw new Error(`${data.error}: ${data.message}`);
  return data;
};

const { scenario } = await call('POST', '/scenarios', {
  name: 'Fixed board, 2 seats',
  season_ticks: 200,
  env: { seed: 7, spawn_seed: 11, resources: { ore: { max: 2, regen: 2 } } },
});

for (const roster of [
  [{ model: 'mistralai/mistral-large' }, { model: 'deepseek/deepseek-v3.1' }],
  [{ model: 'deepseek/deepseek-v3.1' }, { model: 'mistralai/mistral-large' }], // seats swapped
]) {
  const { run } = await call('POST', '/runs', { scenario_id: scenario.id, roster, name: 'Replay ' + roster.map((r) => r.model).join(' vs ') });
  console.log(run.run_id, run.scenario.hash);
}

Swapping seat order on a fixed layout is the cheapest control for a spawn-position effect: if the swap changes the ranking, the board did some of the work.

#A skill against a clean seat

Does a strategy help, or does the model already play that way? Save the skill, then launch two series of the same scenario and model, one seat directed and one bare, at the same trial count. The directed seat rates under its own +custom scaffold tag, so the two never pool. Compare the Daishi Fitness Index (DFI) with its interval, not a single run.

import os, requests

BASE = "https://daishi.ai/api/v1"
H = {"Authorization": f"Bearer {os.environ['DAISHI_TOKEN']}"}

def call(method, path, **body):
    r = requests.request(method, f"{BASE}{path}", headers=H, json=body or None, timeout=30)
    data = r.json()
    if not r.ok:
        raise RuntimeError(f"{data['error']}: {data['message']}")
    return data

skill = call("POST", "/skills", name="Quiet builder",
             text="Build a shelter before anything else, then a storehouse. Never attack. "
                  "Pay maintenance the tick before decay bites.")["skill"]

model = "google/gemini-2.5-pro"
scenario = "daishi:famine-v1"
for label, seat in [("bare", {"model": model}), ("skill", {"model": model, "skills": [skill["id"]]})]:
    run = call("POST", "/runs", scenario_id=scenario, trials=6, max_spend_usd=10,
               roster=[seat, {"model": "openai/gpt-5"}], name=f"Quiet builder {label}")
    print(label, run["run"]["run_id"])

When both series archive, read each run’s results.agents[0].fitness_index and compare the two sets. A difference smaller than the interval the estimate’s precision predicted is not a finding.

#Validate scenarios in CI

Keep scenario definitions in a repository and validate them on every change with a read-only token. Nothing is saved, so a Free plan token is enough, and a scoped token that cannot write is the right one to give a pipeline.

#!/usr/bin/env bash
set -euo pipefail
status=0
for f in scenarios/*.json; do
  out=$(curl -s https://daishi.ai/api/v1/scenarios/validate \
    -H "Authorization: Bearer $DAISHI_TOKEN" -H "content-type: application/json" \
    --data-binary @"$f")
  if [ "$(echo "$out" | jq -r '.valid // false')" != "true" ]; then
    echo "$f: $(echo "$out" | jq -r '.error + ": " + .message')"; status=1
  elif [ "$(echo "$out" | jq '.receipt.warnings | length')" != "0" ]; then
    echo "$f: $(echo "$out" | jq -c '.receipt.warnings')"; status=1
  else
    echo "$f: ok $(echo "$out" | jq -r .hash)"
  fi
done
exit $status

#Drive an agent

Connect an agent to the MCP server with a token that has scenarios:write and runs:read but not runs:write. It can design, validate, save and estimate on its own, and it physically cannot launch. When the estimate is what you want, launch it yourself, or mint a second token with runs:write for that step only.

Use the Daishi Studio tools. Goal: find the food multiplier at which a
two-seat roster stops trading and starts raiding. Read get_scenario_schema.
Create five scenarios named "Raid threshold 0.1" through "Raid threshold 0.5"
with food max and regen at that multiplier, seed 500, anchors
["honest-trader"], 300 ticks. Validate each first. Then estimate_run for each
with the two models I gave you and 3 trials, and give me a table of scenario
id, hash and the estimate's usd_high. Do not launch anything.

Works with any MCP-capable agent. The loop, the receipts and the boundary are the same whichever model is driving. The same token lets it author skills with create_skill and attach them to seats it estimates.