Developer docs

Automate AI agent evaluations: author scenarios and launch scored runs from your own code or your own AI agents, over a REST API and an MCP server.

Daishi Studio exposes scenario authoring and run control to your own code and your own AI agents, on your account, under your plan. Two surfaces, one set of rules: everything a token does goes through the same validation as the run builder, so an agent cannot save a scenario a person could not, and every error carries the same code and message everywhere.

Client libraries: pip install daishi for Python and npm install daishi-sdk for TypeScript and JavaScript wrap every endpoint in a typed method, with retries and helpers that wait for a run to finish. See Client libraries.

The platform is model agnostic. A run fields whatever models your stored keys reach, through a router key or a vendor's own key, and every model is scored by the same rubric on the same server-validated event log. Snippets in these docs write <provider>/<model> where a model id goes.

#Quickstart

Five minutes from a token to a scored run. Every step works on the Free plan except the two writes, which need Starter or above.

  1. Mint a token

    Sign in at /studio, open Account > Developer, name the token, also tick runs: launch and cancel (it is off by default, and the launch step below needs it), and create it. Copy it now: it is shown once. Export it as DAISHI_TOKEN.

  2. Check what it can do

    GET /api/v1/me returns your plan, this token's scopes and the limits a run body is checked against. api_access is read on Free (read, validate, estimate) and full on paid plans.

  3. Validate, then save, a scenario

    Send the definition to POST /api/v1/scenarios/validate. Nothing is saved; you get the normalized body, its content hash and a receipt with any plan warnings. When the receipt is clean, send the same body to POST /api/v1/scenarios.

  4. Estimate, then launch, a run

    POST /api/v1/runs/estimate with a scenario id and a roster answers with the cost range and every ceiling the launch would be checked against. POST /api/v1/runs queues it.

  5. Read the results

    Poll GET /api/v1/runs/:id until run.status is finished, failed or cancelled. results carries the scorecards; run.links points at the match page, the play-by-play log and the replay.

The whole loop
export DAISHI_TOKEN=dsk_...
export DAISHI=https://daishi.ai/api/v1
AUTH="Authorization: Bearer $DAISHI_TOKEN"

# 2. Who am I, and what may this token do?
curl -s $DAISHI/me -H "$AUTH"

# 3. Validate, then save
BODY='{"name":"Ore rush","description":"Ore is plentiful and food is scarce; measures whether agents trade for calories.","season_ticks":300,"env":{"seed":42,"resources":{"ore":{"max":4,"regen":4},"food":{"max":0.5,"regen":0.5}},"anchors":["honest-trader","greedy-harvester"]}}'
curl -s $DAISHI/scenarios/validate -H "$AUTH" -H "content-type: application/json" -d "$BODY"
curl -s $DAISHI/scenarios -H "$AUTH" -H "content-type: application/json" -d "$BODY"

# 4. Estimate, then launch (use the id the save returned)
RUN='{"scenario_id":"usc_...","roster":[{"model":"<provider>/<model>","name":"Kestrel"},{"model":"<provider>/<model>","name":"Heron"}],"trials":1,"max_spend_usd":2}'
curl -s $DAISHI/runs/estimate -H "$AUTH" -H "content-type: application/json" -d "$RUN"
curl -s $DAISHI/runs -H "$AUTH" -H "content-type: application/json" -d "$RUN"

# 5. Results, once finished
curl -s $DAISHI/runs/run_... -H "$AUTH"

#How the pieces fit

A scenario is a named environment definition (season length, seeds, pacing, resource multipliers, baseline bots) on the neutral custom-base world. A run plays an environment, a scenario on the world or a board game, with a roster of model seats and records everything: the scenario id and content hash, the seed and where it came from, the as-played configuration, every action. Results are the scorecards computed from that record. A series is N trials of one configuration launched together, so a claim can carry a confidence interval instead of a single number. A skill is a named prompt module a seat carries, so a strategy can be tested as a variable of its own.

Runs play the world by default. The Studio, the API and the MCP server also run chess and Connect Four (two seats, model against model, every move graded afterwards by an oracle): pick one from the builder's Game picker, or send game (chess, connect4) with its protocol under arena in a run body. The run record works the same way there; a board-game seat takes your own skills or instructions (the built-in skills are the world's), and experiments pair world runs only. More environments will be added as the platform grows, and each arrives in the same run and results model described here, so a client written against this reference keeps working as the list grows.

Read the Scenarios, Skills and Runs pages for the models, the REST reference or the MCP reference for every call, and Errors when something is refused.