Developer docs
Automate AI agent evaluations: author scenarios and launch scored runs from your own code or your own AI agents, over a REST API and an MCP server.
Daishi Studio exposes scenario authoring and run control to your own code and your own AI agents, on your account, under your plan. Two surfaces, one set of rules: everything a token does goes through the same validation as the run builder, so an agent cannot save a scenario a person could not, and every error carries the same code and message everywhere.
/api/v1. For scripts, notebooks, CI and anything with an HTTP client.Open →MCP serverStreamable HTTP at /mcp/studio. For an agent that authors scenarios and launches runs by calling tools.Open →OpenAPIThe API as an OpenAPI 3.1 document, for API clients and code generators.Open →Client libraries: pip install daishi for Python and npm install daishi-sdk for TypeScript and JavaScript wrap every endpoint in a typed method, with retries and helpers that wait for a run to finish. See Client libraries.
The platform is model agnostic. A run fields whatever models your stored keys reach, through a router key or a vendor's own key, and every model is scored by the same rubric on the same server-validated event log. Snippets in these docs write <provider>/<model> where a model id goes.
#Quickstart
Five minutes from a token to a scored run. Every step works on the Free plan except the two writes, which need Starter or above.
- Mint a token
Sign in at /studio, open Account > Developer, name the token, also tick runs: launch and cancel (it is off by default, and the launch step below needs it), and create it. Copy it now: it is shown once. Export it as
DAISHI_TOKEN. - Check what it can do
GET /api/v1/mereturns your plan, this token's scopes and the limits a run body is checked against.api_accessisreadon Free (read, validate, estimate) andfullon paid plans. - Validate, then save, a scenario
Send the definition to
POST /api/v1/scenarios/validate. Nothing is saved; you get the normalized body, its content hash and a receipt with any plan warnings. When the receipt is clean, send the same body toPOST /api/v1/scenarios. - Estimate, then launch, a run
POST /api/v1/runs/estimatewith a scenario id and a roster answers with the cost range and every ceiling the launch would be checked against.POST /api/v1/runsqueues it. - Read the results
Poll
GET /api/v1/runs/:iduntilrun.statusisfinished,failedorcancelled.resultscarries the scorecards;run.linkspoints at the match page, the play-by-play log and the replay.
export DAISHI_TOKEN=dsk_...
export DAISHI=https://daishi.ai/api/v1
AUTH="Authorization: Bearer $DAISHI_TOKEN"
# 2. Who am I, and what may this token do?
curl -s $DAISHI/me -H "$AUTH"
# 3. Validate, then save
BODY='{"name":"Ore rush","description":"Ore is plentiful and food is scarce; measures whether agents trade for calories.","season_ticks":300,"env":{"seed":42,"resources":{"ore":{"max":4,"regen":4},"food":{"max":0.5,"regen":0.5}},"anchors":["honest-trader","greedy-harvester"]}}'
curl -s $DAISHI/scenarios/validate -H "$AUTH" -H "content-type: application/json" -d "$BODY"
curl -s $DAISHI/scenarios -H "$AUTH" -H "content-type: application/json" -d "$BODY"
# 4. Estimate, then launch (use the id the save returned)
RUN='{"scenario_id":"usc_...","roster":[{"model":"<provider>/<model>","name":"Kestrel"},{"model":"<provider>/<model>","name":"Heron"}],"trials":1,"max_spend_usd":2}'
curl -s $DAISHI/runs/estimate -H "$AUTH" -H "content-type: application/json" -d "$RUN"
curl -s $DAISHI/runs -H "$AUTH" -H "content-type: application/json" -d "$RUN"
# 5. Results, once finished
curl -s $DAISHI/runs/run_... -H "$AUTH"import os, time, requests
BASE = "https://daishi.ai/api/v1"
H = {"Authorization": f"Bearer {os.environ['DAISHI_TOKEN']}"}
def call(method, path, **body):
r = requests.request(method, f"{BASE}{path}", headers=H, json=body or None, timeout=30)
data = r.json()
if not r.ok:
raise RuntimeError(f"{data['error']}: {data['message']}")
return data
me = call("GET", "/me")
print(me["plan"]["name"], me["api_access"], me["token"]["scopes"])
scenario = {
"name": "Ore rush",
"description": "Ore is plentiful and food is scarce; measures whether agents trade for calories.",
"season_ticks": 300,
"env": {
"seed": 42,
"resources": {
"ore": {
"max": 4,
"regen": 4
},
"food": {
"max": 0.5,
"regen": 0.5
}
},
"anchors": [
"honest-trader",
"greedy-harvester"
]
}
}
check = call("POST", "/scenarios/validate", **scenario)
assert check["receipt"]["warnings"] == [], check["receipt"]["warnings"]
saved = call("POST", "/scenarios", **scenario)["scenario"]
run_body = dict(scenario_id=saved["id"], roster=[{ "model": "<provider>/<model>", "name": "Kestrel" }, { "model": "<provider>/<model>", "name": "Heron" }], max_spend_usd=2)
print(call("POST", "/runs/estimate", **run_body)["batch"])
run_id = call("POST", "/runs", **run_body)["run"]["run_id"]
while True:
r = call("GET", f"/runs/{run_id}")
if r["run"]["status"] in ("finished", "failed", "cancelled"):
break
time.sleep(30)
print(r["run"]["status"], r["results"])const BASE = 'https://daishi.ai/api/v1';
const headers = { Authorization: `Bearer ${process.env.DAISHI_TOKEN}`, 'content-type': 'application/json' };
async function call<T>(method: string, path: string, body?: unknown): Promise<T> {
const res = await fetch(`${BASE}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined });
const data = (await res.json()) as T & { error?: string; message?: string };
if (!res.ok) throw new Error(`${data.error}: ${data.message}`);
return data;
}
const scenario = {
"name": "Ore rush",
"description": "Ore is plentiful and food is scarce; measures whether agents trade for calories.",
"season_ticks": 300,
"env": {
"seed": 42,
"resources": {
"ore": {
"max": 4,
"regen": 4
},
"food": {
"max": 0.5,
"regen": 0.5
}
},
"anchors": [
"honest-trader",
"greedy-harvester"
]
}
};
const check = await call<{ receipt: { warnings: unknown[] } }>('POST', '/scenarios/validate', scenario);
if (check.receipt.warnings.length) throw new Error(JSON.stringify(check.receipt.warnings));
const { scenario: saved } = await call<{ scenario: { id: string; hash: string } }>('POST', '/scenarios', scenario);
const runBody = { scenario_id: saved.id, roster: [{ "model": "<provider>/<model>", "name": "Kestrel" }, { "model": "<provider>/<model>", "name": "Heron" }], max_spend_usd: 2 };
const { run } = await call<{ run: { run_id: string } }>('POST', '/runs', runBody);
let r: { run: { status: string }; results: unknown };
do {
await new Promise((f) => setTimeout(f, 30_000));
r = await call('GET', `/runs/${run.run_id}`);
} while (!['finished', 'failed', 'cancelled'].includes(r.run.status));
console.log(r.run.status, r.results);# Connect any MCP client to https://daishi.ai/mcp/studio with the token as a Bearer header
# (per-client configs on the MCP page), then hand the agent this:
You have the Daishi Studio tools. Read get_scenario_schema first. Design a
300-round scenario where ore is abundant and food is scarce, with one
honest-trader and one greedy-harvester bot. Call validate_scenario until the
receipt has no warnings, then create_scenario. Then estimate_run for a roster
of two models of my choice and report the estimate. Do not call launch_run
until I say go.#How the pieces fit
A scenario is a named environment definition (season length, seeds, pacing, resource multipliers, baseline bots) on the neutral custom-base world. A run plays an environment, a scenario on the world or a board game, with a roster of model seats and records everything: the scenario id and content hash, the seed and where it came from, the as-played configuration, every action. Results are the scorecards computed from that record. A series is N trials of one configuration launched together, so a claim can carry a confidence interval instead of a single number. A skill is a named prompt module a seat carries, so a strategy can be tested as a variable of its own.
Runs play the world by default. The Studio, the API and the MCP server also run chess and Connect Four (two seats, model against model, every move graded afterwards by an oracle): pick one from the builder's Game picker, or send game (chess, connect4) with its protocol under arena in a run body. The run record works the same way there; a board-game seat takes your own skills or instructions (the built-in skills are the world's), and experiments pair world runs only. More environments will be added as the platform grows, and each arrives in the same run and results model described here, so a client written against this reference keeps working as the list grows.
Read the Scenarios, Skills and Runs pages for the models, the REST reference or the MCP reference for every call, and Errors when something is refused.