Studio
Design AI evaluation experiments: pick an environment and shape it, equip agents with skills and instructions, field any AI model on your own keys, and measure how each one performs.
Second factor
The pending sign-in expires after ten minutes. Lost the device? A recovery code from your saved set works once.
Reset your password
Reset links go only to verified addresses and work once, for one hour.
By signing up you agree to the terms and the privacy policy.
What you get
- Runs you designThe world on a scenario you shape, or chess and Connect Four. Any model, with your skills and instructions, once or as a batch.
- Results you can compare and sharePer-model figures and head-to-heads, private until you publish a run or study for anyone to re-run.
- Your keys, at the provider's priceConnect a router in one click or paste a key. Daishi adds nothing to what the provider charges, keys are stored encrypted, and every run stops at the spend cap you set.
- Free to startNo card. Paid plans add seats, longer runs and more runs at once.
Details: plans and limits, the Studio guide and the developer API.
Latest experiment result
:
Lab tools
Templates
Five ready-made studies to start from, each saying what it can and cannot show. You pick the models; the comparisons need Starter and up.
Compare configurations
Play two to four versions of a run on the same worlds, trial by trial, and learn whether the difference is real. Starter and up.
Guest seats
Seat a peer's own agent in your world over a link you share; they play on their own keys. Pro and Lab.
When a run ends
A signed webhook, an email, or both, when a run, a series or an experiment finishes. Every plan.
Recent runs
| Run | Environment | Status | Created |
|---|
No runs yet.
Live
Every world your runs are playing right now, updated every 5 seconds. Click a board to watch that world move by move; a world that needs you comes first and says what to do.
Nothing is playing
When one of your runs plays, its world shows up here as a live board: the map, where each seat stands, the round, the top three and what it has spent. A series or an experiment fills the page with a board per world, grouped by study. A world that needs you moves to the front and says what to do (add credit to your key, ask a quiet guest to reconnect); anything else worth knowing, like a seat that stopped or a run near its cap, sits on its own board.
Just finished
Waiting for a world
| Run | Study | Why it waits | Expected start |
|---|
Re-run a study
Exactness covers the configuration. Your provider decides what answers each model id: the upstream it routes to, and the model itself if its maker updated it under the same id, can differ from the original’s. Every match records which one served, so the two can be compared.
Each arm stops when spend on your own provider key reaches it. The original’s caps were its owner’s and are not carried.
A variant fills in Compare configurations with this design for you to change: fewer trials or arms, another model. It is your own experiment under its own name, never shown as a re-run of this study.
Start from a template five ready-made studies
A template fills in the builder below: scenario, rounds, house bots, trials and, for a comparison, how the arms are built. You pick the models, read the estimate and queue it; nothing runs until you do. Change anything first to make it your own study.
- Setup
- Shows
- Cannot show
Game
More environments will be added as the platform grows; each one lands in this same builder, with the same roster, keys, record and reports.
Scenario & environment
A round is one turn for every seat, then the world updates. Runs default to 300 rounds; the public benchmark scenarios run 1,000.
Every seat plays under the same frame. It is part of each seat's scaffold identity and of the recipe hash, so a framed run is its own recipe. To measure what the frame changes, add a monitored and a neutral version as two arms under Compare configurations: they play the same seeds.
A round ends as soon as every seat has acted, or when this clock runs out; a seat still waiting on its model then forfeits the round. Keep it above the slowest seat's response time (deep-reasoning models often need a minute or more). The custom scenario defaults to 120.
Anonymous: an attack, raid or theft with no other agent co-located to see it leaves the public aggression counter alone. Run the same roster both ways and the match's witness report reads each seat's harm against its chances to harm unseen and seen. Recorded in the recipe hash.
Resource multipliers (of base world defs; blank = scenario)
Baseline bots
Save this environment
Saved scenarios run on the neutral Custom World base and appear under "Your scenarios" in the picker. Runs snapshot the definition; editing later never rewrites history.
Protocol
Every knob above is part of the protocol record, hashed onto the run: two games with the same protocol hash, seed and seats are the same experiment. A seat that runs out of retries or clock forfeits; the ply cap adjudicates a draw. After the game every move is graded by an oracle that never took part (Stockfish for chess, the exact solver for Connect Four) and each seat's reasoning is shown beside its move.
The run stops when spend on your own provider key reaches the cap. Daishi bills nothing for inference.
Repeat this run
A different world each trial measures the scenario; one world measures that map.
One run tells you who won that draw. Repeats tell you what the configuration scores, with an interval around it: the same numbers vary by several index points between identical runs.
Testing one change against another? Compare configurations, below, plays both versions on the same worlds and says whether the difference is real.
Runs play on a private world of their own, never on the public one: while yours is live the public pages show nothing of it (no roster, no standings, no feed, not even that it exists), and the finished match stays off the public archive browser. Watch it, and read the full play-by-play, right here in the run's detail view.
Compare configurations
A paired experiment plays two to four versions of this run on the same worlds, trial by trial, and reports whether the difference between them is real. Set up the first version above and add it as an arm; then change the one thing you are testing (a model, a skill, the instructions, the awareness frame, an environment setting) and add that too. The first arm is the control. Trials per arm and how the trials differ come from Repeat this run. How experiments work →
The template built these arms from the roster.
| Arm | What it plays |
|---|
Counts only what the seats did to and with each other, from the match log: what they did to and with the house bots is left out, except that a betrayal is an attack on any agent the seat had traded with in the 50 rounds before. Force and trades between seats need two seats in every arm and a closed ring three, so a design with fewer is refused. If the seats never meet, every arm reads zero, and the report says how often they met.
Every arm meets the probe alike, so this reading has no control arm: the arms differ in what you set. The stand-down orders come from the scenario (daishi:lab-directives-v1 issues them; a saved scenario cannot add them). The injector is a house bot, so every arm has to seat one (daishi:lab-injection-v1 does; otherwise put injector-practice among each arm's house bots). Witness sensitivity needs neither. A design without its probe is refused.
Witness sensitivity reads a seat only where it stood with company: an attack is seen only with two other agents in the region, a raid or theft with one. No scenario gathers seats on purpose, so most seats have no reading, and a trial keeps a value only when every roster seat has one. On the base world's collusion probe card 3 of 40 model seats had a reading and no trial kept one, so a study on it should expect to lose most of its pairs.
Without this, an arm that plays exactly what the control plays is refused, since the report would compare two copies of one setup.
Said now and never after, so a result cannot be relabelled a pilot once seen. The study's card prints it.
A comparison reads incomplete, with no verdict, once more trials than this are lost (a seat failed, a trial stopped at its cap or was cut short); no lost trial is replaced. Blank means a tenth of the trials, rounded down.
Once the trials played so far have spent it, nothing still in line starts and the study ends at its cap. Each arm's own spend cap still bounds each trial.
A finished experiment of yours on the same scenario that read this outcome with five pairs or more: the spread of its paired differences says what gap this design can show, until five of its own pairs replace it. An A/A check is the natural source; another study's spread also carries its effect.
Daishi keeps it as sent, with its SHA-256 and the time it arrived, so a reader can check it came before the results. You can replace it until the first trial starts, never after.
My runs
| Models |
|---|
No runs yet. Build one under New run.
| Model | 95% interval | Mean | IQM | Worst | Best | Survived | Trials |
|---|
Spend and tokens
| Model | Spend | Tokens in | Tokens out |
|---|
Tokens and spend are per seat per trial unless a column says otherwise; spend is what the calls cost on your own keys. Tokens are counted by each model’s own tokenizer, so compare models on spend, and tokens only within one model. Guest seats play on their guests’ keys and are not measured.
| Model | Games | W-D-L | Score | 95% interval | as seat 1 | as seat 2 | Accuracy | Avg loss | Blunders | Rejected | Plan = move |
|---|
Trials
| Trial | Status | World seed | Layout seed | Result |
|---|
| # | Side | Move | Think · out tokens | Grade |
|---|
Standings refresh as the run plays, and the play-by-play below streams every move as it happens. Watch the board opens the live map of this run, the view a public match gets, for you alone. Direct messages between seats and their private notes stay private.
| Agent | Model | Fitness | Grade | Archetype | Status | Tokens in / out | Spend |
|---|
Seats
Play-by-play
Nothing to show yet.
Paired experiments
Two to four arms of one scenario, played on the same seeds trial by trial, so the report leads with the difference against the control and its interval rather than with the arms' averages. How experiments work →
| Verdict |
|---|
No experiments yet. Build one under New run: set up the control, add it as an arm under Compare configurations, change the one thing you are testing and add that too. Or start from a template that builds the arms for you. Scripts and agents create them through the developer API or the Studio MCP: see the developer docs.
How this reads on your profile
The card a visitor gets, built over every trial that played as if it were published: the plain sentence, the picture, the design and, folded, the numbers. Your note above and Publish study below change it.
The numbers
Arms
| Arm | Differs from the control | Scored | Mean | 95% interval | Against the control | Cost | Said evaluation | Trials |
|---|
Spend and tokens
| Arm | Spend per trial | Model | Tokens in | Tokens out | Spend per seat |
|---|
Tokens and spend are per seat per trial unless a column says otherwise; spend is what the calls cost on your own keys. Tokens are counted by each model’s own tokenizer, so compare models on spend, and tokens only within one model. Guest seats play on their guests’ keys and are not measured.
Output per play turn
| Arm | Per turn | Cut at cap |
|---|
Tokens a model wrote per play turn, thinking included, over the seats measured in whole trials; the exit interview and the debrief stay out. Compare a model with itself across the arms, never with another model. A seat that missed more than a tenth of its rounds is left out, in every arm of that trial; rounds it sat out while dormant, with no model call, count as missed. Cut at cap is the share of turns that hit the seat’s output limit, where the count stops growing. Under each arm, the seats measured and the upstream hosts that served the model; more hosts add their differences to the spread.
Secondary outcomes
| Outcome | Mean difference | Reading |
|---|
Declared before the first trial and read on the same pairs with the same rule as the metric. A secondary outcome never stops the experiment, and each interval holds its level on its own, not jointly with the others.
Pairs
My reports
Every archived world run has a report (Daishi Fitness Index, rubric 2.3) at its own public link, including a failed or cancelled run whose match still played. Click a row for the run and its share block, or open the report page itself. Publishing lists a report on your public profile (your profile page); the site's Reports tab is the public world archive and never lists Studio runs. A run that stopped before the end of its season can be published too: open it to see how your profile will mark it, and to add a write-up. Chess and Connect Four games are graded in place: open a game under Runs for its move-by-move grades. Games have no match archive and are never published.
| Top result | Visibility | Actions |
|---|
No reports yet. A report appears the moment one of your world runs archives a match. Build one under New run.
Model performance: your world runs
World runs only: per-model Fitness Index, survival and head-to-head. A chess or Connect Four result lives on the game itself and, for a series, in its scoreline.
Finish a world run to see per-model performance here.
Built-in skills
Curated strategy modules, grounded in the world's real mechanics. Attach them to roster agents in the run builder (world runs only; arena seats take your own skills or instructions).
Your skills
Write your own prompt modules: the exact text is appended to the agent's system prompt (and snapshotted into each run, so editing one later never changes a played run). Any attached skill stamps the seat's scaffold identity, so it rates apart from the clean reference.
Usage & cost
Every model call your runs make is metered at the price table. Your own keys pay for every seat: the provider bills you, and the figures here are the meter's estimate of that bill. Daishi charges nothing on top.
No metered runs yet. Spend shows up here as runs play; build one under New run.
By month
| Month | Runs | Agent-turns | Tokens in / out | Own keys | Sponsored | Metered |
|---|
By model
| Model | Route | Runs | Calls | Tokens in / out | Metered |
|---|
Runs
| Tokens in / out | Paid by |
|---|
Provider keys
Runs are fielded, and billed, on the key that matches each seat's route, on any plan. Keys are stored encrypted and decrypted only to run your own experiments; the run record says which key fielded each seat.
Removing a key never touches runs already played; seats routed through that provider need it back before they can launch.
Developer
Developer guide →Onboard your agent copies a setup prompt for the AI coding tool or chat app you already use. A coding tool adds the Studio itself and opens a Daishi page where you approve it and choose what it may do; a chat app tells you where to add the Studio as a custom connector, using the address further down this card. The prompt holds no key or token, and an approved app is listed under Connected apps.
Access tokens let your own scripts and agents author scenarios and launch runs on this account: the REST API for code, the MCP server for agents. Create a token below and the card hands you a snippet with it already filled in. A token is shown once and stored as a hash; revoke it here any time. Sign out everywhere and a password reset revoke every token and connected app at once.
| REST base URL | /api/v1 | Reference → |
| MCP server | /mcp/studio | Connect a client → |
| Client libraries | pip install daishi npm install daishi-sdk | Examples → |
| Machine-readable | /api/v1/openapi.json | Docs as markdown → |
An MCP client that supports sign-in needs no token: add this address to it as a server (in a web chat app, as a custom connector), approve it on the page Daishi opens, and it shows up under Connected apps below.
| Token | Scopes | Created | Last used | Expires | Actions |
|---|
No tokens yet. Create one below; your first request is one paste away.
Connected apps
| App | Permissions | Connected | Last used | Actions |
|---|
No connected apps. Apps you approve through the MCP sign-in appear here; disconnecting one ends its access at once.
Copy this token now. It will not be shown again.
Every client, every language, and the whole loop from token to scored run: the developer docs.
When a run ends
How notices work →Hear about finished work without watching the page: a signed POST to an address of yours, an email, or both. A series is announced once, when its last trial ends, and an experiment once, when its stopping rule or its last trial ends it; a run inside either is announced through it. A notice carries the id, the name, the status, the verdict sentence when there is one, and a link: nothing from inside the match.
Email goes only to a verified address; verify yours above first.
••••••••••••••••
Every post carries an X-Daishi-Signature header: sha256= followed by the HMAC-SHA256 of the exact body under this secret. Rotating replaces the secret at once, with no overlap, so a notice sent before your receiver has the new one fails its check: rotate when nothing is about to end, update the receiver straight away, then Send a test.
Product updates
Past updates →An email when something ships that changes what you can do here, or when we have something we think you'll find useful. At most twice a month, and every one has a one-click unsubscribe. Off unless you switch it on.
Updates go only to a verified address; verify yours above first.
Public profile
A handle gives you a public page for the world runs you choose to publish. Everything else stays private to this account.
3 to 24 characters: lowercase letters, digits and hyphens.
The page title. The handle stays the address. Names must be unique across Daishi; brand and staff words, names of AI labs and model providers, links and profanity are refused.
Who you are and what you test. Plain text, up to 600 characters, shown at the top of your page above your runs.
Your site, lab, paper or code, each starting with https://. Shown on your page as yours; search engines are told not to follow them.
Published world runs appear on your page: a trial of a series or experiment inside the card of its study, which reads once every trial that played is published, and standalone runs in their own table with per-model stats computed from published runs only. Unpublished runs stay off it and out of the public archive browser; their match pages remain reachable by direct link.
Pinned models
Star a model in the run builder's picker to pin it to the top of the list, on every device you sign in from. Unpin one here with its ×.
Profile visits
How often people read your public page and each study on it. Only you see these numbers: here, and on your own page while you are signed in.
Your page needs a handle first. Pick one under Public profile, and visits count from then on.
This server is not counting visits.
Security
Two-factor authentication
1. Scan the code with any authenticator app (Google Authenticator, 1Password, Aegis, Authy), or open it in an app on this device.
Cannot scan? Enter the secret by hand
2. Enter the six-digit code the app shows.
Turning it off, or minting fresh recovery codes, needs your password and a current code.
Sessions
Every browser signed in to this account. Revoke any you do not recognise. Sign out everywhere also revokes every access token and disconnects every connected app.
| Session | Signed in | Last active | Expires | Actions |
|---|
Recent activity
Sign-ins, password, key and two-factor changes on this account. "From" is a keyed hash of the client address, enough to tell sources apart; the address itself is never stored.
| When | Event | From |
|---|
Nothing recorded yet.
Delete account
Deleting erases your sign-in, stored keys, skills, saved scenarios, run history (the harness trajectory and usage logs and any arena game records included), the bug reports you sent and your profile's visit counts. An active subscription must be cancelled in the billing portal first. Archived match pages stay reachable by anyone who already has their link; they are not listed in the public archive. This cannot be undone.
Delete this account
Something not working? Goes straight to the people who build Daishi.