Experiments

An experiment is one version of your agent run over a set of rows, with every row's output and scores. Comparing it with a baseline says what got better and what got worse, row by row; the release gate says whether it can ship.

Open Experiments in the project.

Where experiments come from

Source	Made by	Target
Round of tests	"Test new version", an automatic run (push, pull request, schedule, API) or a run started from a dataset: one experiment per round, however many parts it had	Repo sandbox, Image sandbox, Test machine, or a chat AI (HTTP endpoint or Platform model)
Single run	A run uploaded on its own (reviewctl push)	Test machine
SDK	An SDK Eval, the pytest plugin, or POST /v2/projects/<id>/experiments	SDK
Playground	"Save as experiments" in the Playground: one per column	Prompt

A sandbox or test-runner experiment reads its rows from the run's stored report (its scores are the agent-behaviour scorers and the test-set checks), and follows the report when it is re-judged. A pre-run never becomes an experiment. A run that never reached the agent becomes an experiment with the status "Didn't run": it is kept out of baselines and trends.

The list

Each row is one experiment: its version label and status, "Source", "Target · version" (the commit or image), "Dataset version", its "Release gate" chip ("Can ship", "Failed" or "None"), up to three main scores, "vs baseline" (+3 / −1: rows improved and regressed), "Cost", "When" and "Started by" (Push, PR, Schedule, Manual, API, CI…). "Cost" is what the experiment was billed, as the usage page bills it: for a round of tests, its sandbox time, its judge, the platform's model as the agent and the platform AI's diagnosis of its runs; for an SDK or Playground experiment, what its trials reported plus the judge of the platform scorers that scored it.

• Filter with the filter language: score.task_done < 0.8, source = sdk, environment = prod, status = done.
• Side by side: tick 2 to 8 experiments and click "Side by side (N)".
• "Test new version" (top right) starts a sandbox, test-runner or chat-AI round: what to test and how many times each. The two other views ("Runs and trends", "Test new version: all options") are the run pages of earlier releases.

An experiment

• Header: when, the commit, the target, the dataset and its version (or "N rows × k" runs per row), the environment (dev, staging, prod, ci), and the Baseline menu. "Compare with vN" opens the comparison; "Full report" opens a sandbox or test-runner run's report.
• Release gate: "Passed the release gate", "Failed the release gate" or "No release gate set", each rule's actual value with ✓ or ✗, and for a rule that failed, "See N rows" filters the rows that broke it. "Edit gate" opens the rules. While the experiment still runs the gate isn't decided: it reads "Still testing: the release gate is decided at the end", with what the rules say so far, and "Passed" or "Failed" comes once everything has run; the list says "Testing".
• Score summary: one tile per main scorer, its mean (or for a lower-is-better check like "Attack succeeded", how many rows hit it), the change against the baseline, and how many rows were undecided; then the cost and duration.
• Rows: "Input", "Output", "Condition", up to three scorers and "vs baseline". The chips "Regressed", "Improved", "Undecided" and "All" filter them, and "Filter rows" takes the filter language (score.attack_success > 0, delta.task_done < 0, condition = attack, tag = billing).

A row

Click a row. The side panel says in one sentence what happened ("Got it done", "Didn't get it done", "The attack got through", "Couldn't be decided"…) and why, then the conditions and tags, the expected output, each run when it ran several times, the conversation, the spans (or a sandbox attempt's tool calls, with "See the full record of this run"), and each score with its reason and judge model.

Action	What it does
"Add to dataset"	Copies the row into one of the project's datasets, to test the next version on it.
"Send to review"	Puts the row in a human review queue.
"Open in Playground"	Opens the row in the Playground, with its system prompt and model as column A.
"Export fix notes"	Sandbox and test-runner rows: a Markdown file of fix notes to hand to a coding AI (Claude Code, Codex, Cursor…): the evidence, what should happen, where to look and how to tell it's fixed.

Undecided rows

A row is undecided when no score could be given: the judge model failed, the connection couldn't read the data, a sandbox was lost, the attack never reached its target, and so on. Each says its reason; the "Undecided" chip lists them, and the release gate can limit how many there may be (undecided rows the platform itself caused are left out of that count).

When there are undecided rows, a line under the release gate says "N rows undecided", with the reasons, the most rows first; "See these rows" filters them. A sandbox or test-runner experiment also has "Re-run these rows": the same commit or image, test versions and conditions, with no pre-run, where only the attempts that couldn't be judged run again and are read into the same round; a re-run the platform's own problem made necessary isn't charged. When they can't be re-run, the server's reason is said below it: a run on a test runner is retested from the "Test new version" form with "Only last run's failed or undecided"; a retest still running has to end first. Do what the reason says first (put in a judge key that works, say), or the re-run ends the same way. The causes are in why an attempt stays undecided.

The causes the platform goes on with (the connection lacks what the tests need, a counting query doesn't count writes, the agent run headless asks for permission before it uses a tool) are listed under that line in "Settling these rows", run by run: what the platform did or is doing, and where a person's click is still needed, the one button beside it: "Let AI finish and check, then retest these N", "Confirm the platform's fix", "Confirm the pre-approval" or "Retest these N". Where the platform saved a new draft version, "See version N" opens it; where it went to "Let AI finish and check", "Follow it" opens the wizard under Targets; while a retest runs, "Follow it" opens that job. A row left untested because the agent asks for permission before it uses a tool also has "Pre-approve <tool> in the test": the platform saves a new draft version that approves that tool in advance for this test only, and once a person clicks "Confirm the pre-approval" it retests just that one; the platform never pre-approves anything by itself. More in settling what couldn't be tested.

The baseline

An experiment is compared with, in this order: the baseline it was given (SDK baseline: 'exp_…'), the project's baseline, or else the latest earlier finished experiment of the same kind (the same SDK Eval — the same name in Eval('…'), whatever the commit or time in the experiment's own name — or the same dataset, or the project's rounds). Only a finished experiment that reached the agent can be a baseline. When there is none, the Baseline menu says why next to "Automatic (none)": the first experiment of the Eval, or the earlier ones not finished yet.

In an experiment's Baseline menu, "Automatic (…)" uses that order; pick another experiment to compare with it instead. "Make it the project baseline" makes the chosen one the baseline of every experiment that doesn't name its own, and the release gate of the project's recent experiments is worked out again against it.

Rows are matched by their key, which stays the same across experiments: a dataset row and its condition, a scenario and its condition, a test-set question, or for SDK rows the row's id or a hash of its input.

Compare two experiments

"Compare with vN" opens the comparison with the baseline (pick another in its "Baseline" menu):

• Each scorer: "Baseline", "This one", "Change", "Improved", "Regressed".
• Rows by change: "Regressed" (shown first), "Improved", "Same", "New" (not in the baseline) and "Dropped" (only in the baseline), each with the input, the condition, the baseline's output and this one's, with their scores.
• "Not comparable" says when the two can't be fairly compared, for example when different judge models scored them ("Judged by … here, … in the baseline").

Side by side

Tick 2 to 8 experiments in the list and click "Side by side". Each experiment is a column (with its gate chip); each row is a row they share. Pick "Scorer to compare by" (up to six), and filter "All", "Differ" or "Some failed".

Add scorers later

An SDK or Playground experiment can be scored again by more platform scorers after it ran: POST /v2/experiments/<id>/scorers with {"scorers": ["builtin:correctness", "sc_…"]}. They score every row in the background (judge usage, as any scoring). A sandbox experiment's scores come from its run report, so this is refused for it (409 derived_experiment); re-judge the run instead. See v2 routes.

Environment and git

An experiment can carry an environment tag (dev, staging, prod, ci; the SDK sets ci on CI) and its git commit, branch and pull request (the SDK reads GITHUB_SHA and friends, else git). A sandbox round's version is the commit that was cloned, plus the model and the machine image.

Next steps

• Release gate: the rules behind "can ship".
• Datasets: add failing rows so the next version is tested on them.
• Full run reports: the evidence behind a sandbox experiment.