Sandbox runs

Start a run of the project's scenarios and test sets (in a sandbox, on a self-hosted runner, or against a chat AI), follow it, cancel it, and finish one that stopped early. Each round becomes one experiment.

Start a run

1. Click "Test new version" at the top right of any section (it opens the form in Experiments).
2. Under "What to test", pick "All", "Functional only", "Security only" or "Only last run's failed or undecided". Each choice shows how many tests it takes. A scenario the AI diagnosis found the platform wrote wrong last time (the platform's misjudgment), unchanged since, is left out of "Only last run's failed or undecided" (a test set is never left out for one wrong question): it would be the same test, left untested again. The page names it: confirm the platform's revised draft if there is one, or fix or remove the test.
3. Under "Times per test", pick 1×, 3× or 5×. More times show whether results are flaky, and each time calls the agent's model. Counts below what the pass criteria need are switched off. This applies to scenarios. A test set asks each question as many times as its own setting says.
4. Click "Start test · N". The page moves to the run's progress page.

The platform pre-runs first (below; at 1× with the normal condition only there's no separate pre-run, as the run itself is the pre-run), then runs the scenarios as one part and each test set as a part of its own, one after another. Together they're one round: one experiment, with one round report behind it.

A project runs one test at a time. Starting another while one is queued or running is refused until it ends or you cancel it. If a run lost contact with its test runner mid-run last time, tick the confirmation that you checked the test environment before you can start (see States).

More options

"More options: pick tests and conditions, or skip the pre-run" opens "Test new version: all options" (also under "Settings" → "Advanced"):

• Scenarios: every confirmed test is ticked; untick the ones to leave out. One run takes at most 200 scenarios.
• Conditions: the conditions the picked scenarios test; "Normal only" or "Select all". A condition the pass criteria use can't be removed.
• Repeats: 1 to 5 times per condition, for the scenarios.
• "Label (optional, shown in reports and comparisons)": up to 80 characters, such as "after fixing the payee account check". If you leave it empty in a project connected through a GitHub repository, the label is the first line of the tested commit's message. A run started by a push uses the pushed commit's message, and one started by a pull request uses the pull request's title.
• "Skip the pre-run and start the test".
• "Preflight checks" lists what's missing, each with "Fix". Whether the tests reach the agent goes by the latest pre-run or check, and by any formal test since that reached the agent: "The test at 10:42 reached the agent; …", so ticking "Skip the pre-run" no longer warns that the whole run may be wasted. When each picked scenario runs once in its normal condition only, this check says the run is what a pre-run would run, so there's no separate one, and the cost estimate no longer adds a pre-run. Parts of the test connection files that raise no error but won't run as written are listed here too, as warnings. So is a run that would pass the organization's "Longest run": in a sandbox project, when the scenarios, or one test set, would take longer than that by this project's history, the checks say so before the click (each test set and all the scenarios are each a run of their own, each held to the limit; from 24 October 2026). "Plan" counts the attempts and estimates the time from this project's history.

Click "Start test". Without pass criteria of your own, the button is "Start test with default pass criteria". "Retest: rerun a past run's scenarios and compare" is here too: it reruns a past run's scenario versions on the current agent, and the new experiment can be compared with that run's (compare).

The pre-run

Before the real run, the platform pre-runs: the normal condition of each functional test in the run, once (at most 12 tests), to see whether the tests really reach the agent. A pre-run gives a pre-run result for diagnosis, not a test report. It doesn't count as a run, stays out of the report list, version comparisons and pass statistics, and can't be a retest baseline. It checks that:

• the test reached the agent;
• the platform can see the agent's tool calls;
• attachments reached the agent;
• the agent uses a model to decide;
• the judge model works;
• the agent finished within its limits.

When it passes, the real run starts by itself. When it doesn't, the run waits and the page lists what failed and how to fix it.

A pre-run that just passed counts for the next run. A run goes without a pre-run of its own when the latest one passed for the same tests and nothing it checked has changed since (no test, test connection file or secret): in a sandbox, at the same commit (or image) within 6 hours; on a local test runner (since 4 November 2026), on the same test runner, not restarted since, with the same code, within 30 minutes. A test runner says its code each time it runs something and each time it asks for work: the commit and every uncommitted change, by content. If its code changes after the launch and before it takes the run, the run pre-runs first after all.

When only the platform's own model or judge model failed it (a rate limit, say; since 20 October 2026), that is the platform's problem, not the test connection's: the page says so, nothing is sent to the AI to fix, and the platform pre-runs again by itself 10 minutes later (at most 3 times), so the waiting run starts once the model is back. "Pre-run again once the platform is back" does it at once. The pre-run's attempt where the platform made a tool fail on purpose counts the same way when the platform's model then failed: whether the connection takes the error can't be told, and it isn't the test connection's problem (since 30 October 2026). An automatic run waits for these too (since 22 October 2026); if the last one fails the same way, it ends, saying it is the platform's problem and how many times the platform pre-ran it again, with a single mail. Such a pre-run's sandbox time isn't charged (see the platform's problem isn't charged).

A project connected to a GitHub repository is fixed by the platform first. When the reason is in the test connection, the platform goes on as "Let AI finish and check" does: it reads the repository, revises the test connection files from this pre-run's result and the sandbox log, and checks again. This pre-run is check 1, and there are at most 10 in all. Once one passes, the waiting run starts by itself. It works on behalf of whoever started the run. Its model calls count against the assistant's daily allowance, and each check is billed by sandbox time like a manual one. "What the platform did" on the page lists each step.

The same goes for a "Check once" or "Pre-run once" of your own that fails, on your behalf. When it has just given up and nothing has changed since (no new commit, connection file or secret), the same check isn't fixed again by itself; "Let AI fix it from this check" hands it over. When the failure is the agent's own (its code errors, it answers something else, it doesn't finish within its limits), the platform changes nothing and stops with the reason, because it only tests. A run gets one such fix.

Tests that check tool calls a replies-only connection can't see are rewritten. Sometimes the only reason is that the tests check tool calls (a call the agent must or must not make, a fault or an attack in a tool) while the connection sees replies only (attach: { mode: endpoint_only }). Then the platform leaves the connection as it is. Its model rewrites each such confirmed test to check the reply instead, and automatic approval reviews each rewrite as it does any draft; with automatic approval off, the rewrites wait for you. Each rewritten test lists what was taken out, and the confirmed version keeps running until its rewrite is confirmed. Once no test checks calls, the waiting run moves to the rewritten tests and is pre-run again. A test someone changed by hand is left for you. The rewrites are model calls counted against the assistant's daily allowance.

After a change that could fix it, the platform pre-runs again. While the run waits for its pre-run, you might change the connection files, a secret or the sandbox settings, or push a new commit to the bound branch. The platform then waits 30 seconds (a row of changes counts once) and pre-runs again on behalf of whoever started the run, up to 3 times per run. After a push, a run started by hand or by a schedule moves to the new commit.

What its tests need counts too. When a new version of one of the run's tests is confirmed (by you or by automatic approval), the waiting run moves to that version and pre-runs again. When a test file its tests use is uploaded (or the platform gets it itself), it pre-runs again. Tests and files the run doesn't use don't count. While a value the check needs is still missing, it doesn't pre-run, and the page names the value.

You can also:

• fix the test connection and click "I've fixed it, pre-run again" (where it was the platform's side, or the platform fixes the connection itself, the button is "Pre-run again": nothing for you to change first);
• copy the sentence under "Or send this to the dev AI" ("Copy sentence") and send it to the dev AI: it reads what's wrong, fixes the connection and pre-runs again until one passes (up to 5 times);
• click "Skip pre-run and start": it asks first, with why the pre-run failed, whether it failed the same way several times in a row, and that the real run will likely not reach the agent either and is still charged; it starts on "Start anyway"; or
• cancel the run.

A run waiting 24 hours with nothing happening ends (from 24 October 2026). When for 24 hours after its pre-run failed nothing pre-ran it again and the platform wasn't fixing the test connection, the run and the ones started with it, still waiting their turn, end, saying why, so they no longer hold the project or one of the organization's sandbox slots. Start it again once things are fixed.

Cancelling a pre-run cancels the run waiting for it. When an automatic run's pre-run fails, a project connected to a GitHub repository waits for the platform's fix and goes on if it works. Only if the fix fails does the run end, with the reason and a single mail. Any other project's run ends there, with the reason. A pre-run that only the platform's own model or judge model failed is the exception: the run waits for the platform to pre-run again, as above.

To pre-run without starting a run, use "Pre-run once" at the top of the "Test new version" form and click "Pre-run". In a sandbox, the virtual machine starts and prepares its environment first. The page says which stage the pre-run is at, for example "Pre-running: preparing the environment (cloning the repository, installing dependencies)", then "last progress": how long ago the sandbox last sent a log line, marked after 10 minutes with nothing new. A chat AI's questions go straight to its API, so a chat AI project has no pre-run.

When the run is the pre-run

When the scenarios part runs each test once, in its normal condition only, with no more than 12 tests, the real run is exactly what its pre-run would run. Then the platform doesn't pre-run separately: the run itself is the pre-run, and once it has run, the same checks as above tell whether the tests reached the agent. The other parts started with it (a security test set, say) wait until it says they did.

• They reached it: it's a real run like any other, with its report and experiment. A run in the next 6 hours at the same commit (or image), connection files and tests goes on it, as on a pre-run that passed, without pre-running again.
• They didn't: from there it is exactly a failed pre-run. The page gives the same reason (for example "Pre-run failed: the test didn't reach the agent"), the run goes back to "Waiting for pre-run", and what ran counts as its pre-run: no report, no re-judging or diagnosis, and it costs what a failed pre-run costs (when only the platform's own part failed it, its sandbox time isn't charged either). A repository project is fixed by the platform, the parts started with it keep waiting, and the platform's own pre-runs again, the 24 hours, and how an automatic run ends are all as above. Once it's fixed, the real run runs in full.
• If you cancel the run, it's cancelled like any other: what ran gets its report, and the parts started with it are cancelled too.

At more than 1× per test, with fault or attack conditions, or with more than 12 tests, the real run does more than a pre-run, so the platform still pre-runs first. A retest pre-runs first too. With "Skip the pre-run" ticked there's no pre-run at all.

Check once

A repository or image project also has "Check once" in its wizard under Targets (the latest shows there as "Connection check"): a pre-run in a sandbox, with each phase and the sandbox's logs shown live. When "Let AI finish and check" runs a check, it says which tests the check runs and why those, where it is now and how long ago it last moved. With no confirmed test yet, the platform checks with its own question: it asks the agent to introduce itself. That only shows the agent starts and replies, not that it gets the job done. Before the check, the platform looks at the values review.yaml and the tests read by name, and says which one is still missing. The details are in the connection guides: GitHub repository and your own container image.

What happens during a run

The progress page shows:

• the stages: queued, picked up, running, report archived;
• the ledger: one square per attempt, showing passed, failed or undecided once the report is in;
• telemetry: time elapsed, the attempt running, the heartbeat;
• the events, and the rules used: repeats, conditions and the pass criteria at the time;
• in a sandbox, the phases "Queued", "Booting", "Preparing", "Running", "Uploading", "Deleted", the log and the cost so far;
• a "Now: …" line with the stage it is at, for example "Now: preparing the environment (cloning the repository, installing dependencies)" or "Now: attempt 3 of 6". A sandbox run adds "last progress": how long ago the sandbox last sent a log line, marked after 10 minutes with nothing new; when it is marked while the sandbox is still preparing its environment, it goes on "if preparing prints nothing for 15 minutes, the platform stops it and says which step stalled" (see preparation that stalls).

Parts started together show as "N started together" and run in order, each starting when the one before ends. What happens inside a sandbox is in the GitHub guide's What happens during a run.

What the states mean

State	What it means
"Waiting for pre-run"	The pre-run goes first; the run starts when it passes.
"Waiting its turn"	A later part of a round; it starts when the part before ends.
"Queued"	Waiting to be picked up by the test runner, the sandbox or the platform executor. A sandbox run also waits here for a free slot: the organization caps how many sandboxes run at once.
"Claimed"	Picked up; about to start.
"Running"	Attempts are running.
"Done"	Finished; the report is ready.
"Failed"	It stopped before finishing; the page says why.
"Cancelled"	Someone cancelled it.
"Lost contact"	The test runner or sandbox stopped reporting and its lease ran out. The run isn't queued again by itself.

After a lost contact while running, what already ran may have had side effects: a claim submitted, a payment made, data changed. Check the test environment by hand (for a sandbox, reset the test backend), then start again. The next start asks you to tick that you've checked. If nothing had run yet, just start again.

For a sandbox run whose review.yaml uses only backends inside the sandbox (services, a tool proxy upstream on 127.0.0.1, the data reset and read addresses), whatever it changed went with the sandbox. The platform checks it itself: the job page says "The platform checked", the audit log records it as the platform's, and the next test (an automatic one too) starts straight away. Anything that might have reached outside (an outside address, an address from a project secret, a reset command, MCP tool servers, a connection that only sees replies) still needs a person to check. The rule is under after a lost run.

A run that stays "Queued" with a note that the test runner's reviewctl is too old starts once you click "Restart" under Targets → "Test runners".

Cancel a run

On the progress page, click "Cancel test" (with parts waiting after it: "Cancel (and the N waiting after it)").

• A run that hasn't started is cancelled at once.
• A running one stops after the attempt in progress. What already ran still gets a report; the rest counts as neither passed nor failed.
• A sandbox still preparing its environment stops preparing at once (the step in progress is stopped with everything it started), running nothing (on older sandbox images it stops once preparing is done).
• Writes already made in the test environment are neither rolled back nor replayed.

The page says when the cancel was requested. For a sandbox run it also gives a deadline: if the sandbox hasn't stopped by then, the platform deletes it (5 minutes after the request). Once it ends, the page says when, how long it took, and what was kept: the attempts that ran with their report (a pre-run's: its pre-run result). If no attempt finished, or attempts ran but uploaded no result, it says there's no report. The sandbox log is always kept.

Limits

A run also stops after the attempt in progress, with a report of what ran, when:

• it nears the organization's maximum run time;
• the month's sandbox hours or the prepaid balance run out;
• the platform pauses the organization.

How many sandboxes run at once, the maximum run time and their defaults are in Sandboxes and limits; spending in Billing and usage.

On a local test runner, each run also has a model-call limit (budget in its review.yaml). A current reviewctl gets a limit big enough for every planned attempt; the "Preflight checks" say when it isn't.

Preparation that stalls

While a sandbox prepares its environment (cloning the repository, pulling or building an image, installing dependencies, starting the services), a step that prints nothing for 15 minutes is stopped by the platform, and the job fails with "Preparing the environment stalled: <the step> printed nothing for 15 minutes (last line: …)". Preparation still going as the run's time limit comes near is stopped too: "Preparing the environment reached the time limit: not done after N minutes, still running <the step> (last line: …)". That time is billed as usual.

Once every setup command has exited, the log says "Dependencies installed" and the install is over. The platform then saves the folders in environment.cache as the dependency cache ("Saving the dependency cache: …"), and restores it at the start of the next preparation ("Restoring the dependency cache: …"). These two steps are the platform's own: each says how far it got every 30 seconds and takes at most 8 minutes. If the cache can't be saved or restored, or runs out of time, the log says so, the cache is skipped and the run goes on (installing as usual when it can't be restored); it never fails because of it. Should one of them stall all the same, the job says "Preparing the environment stalled: the dependencies were installed, and saving the dependency cache …" (or "restoring the dependency cache …"): that's the platform's problem, nothing in the test connection changes it, it isn't charged, and "Let AI finish and check" doesn't change the connection files for it.

When any other step stalls, waiting longer changes nothing; the test connection has to change:

• When installing dependencies stalls, most likely a package's install script (postinstall) is downloading or building something the agent doesn't use (a desktop app's electron-rebuild, say), or waiting on an address it can't reach. Install only what the agent needs (pnpm install --frozen-lockfile --filter <the agent's package>..., say), or skip the install scripts it doesn't need (--ignore-scripts, then pnpm rebuild <package> for one the agent does need). If the last line is already the installer's own summary ("added 5 packages in 17s", "Done in 43s") and the command still doesn't end, most likely something the install started is still running and holding its output: have the setup leave nothing running.
• Any other step (pulling or building an image, starting the services) is most likely waiting on a download or an address it can't reach: read the last lines of the sandbox log and change that step.

A check of "Let AI finish and check" that stalls counts as a check (up to 10): the AI gets the step and its last line, and changes the connection that way, installing only what the agent needs and leaving the agent's own settings as they are. The same stall twice running stops it to wait for you (see Let AI finish and check).

Resume a run that didn't finish

When a run on a local test runner stopped early because its model-call limit ran out or it was cancelled, its report and progress page offer "Raise the limit and finish the other N" (the limit ran out) or "Finish the other N" (it was cancelled). A run in a sandbox can't carry on where it stopped (the sandbox is deleted when its run ends): use a fill-in run below.

The test runner that ran it executes only the attempts it didn't: finished attempts are kept, not rerun, and there's no pre-run. It has to be online, and on a recent reviewctl (click "Restart" under Targets → "Test runners" otherwise). The new report says it finishes the previous run and how many results it reused.

Run the attempts a sandbox run didn't

When a sandbox run stops early (cancelled, near the time limit, the month's sandbox hours or the balance used up), its report and progress page offer "Run the N that didn't run". This costs money, since sandbox time is charged. One click starts a new sandbox that runs only those attempts:

• on the same commit (or image), class and architecture, with the same confirmed test versions, without a pre-run;
• a sandbox runs scenarios × conditions × repeats whole, so what didn't run is cut into a few runs (at most 5) that go one after another; N is the attempts they make together;
• the results are read into the round report. Each attempt that didn't run is filled by one of the fill-in run's attempts of the same scenario and condition, and the run is judged again by its own pass criteria. The fill-in run keeps its own report, marked "Fill-in: its results are read into the part it finishes" in the round;
• if the agent's code, prompt, tools or model changed meanwhile (a different version), the fill-in results are listed apart, not read in.

When it can't be done, the page says why: test sets can't be filled in yet; a scenario it used is archived or its version is gone; the project connects differently now (repository, image or commit); an earlier fill-in run hasn't ended; what didn't run can't be cut into whole runs. Start a new test then.

A sandbox lost part way is filled in by the platform. When a sandbox loses contact in the middle of a run, that's the platform's side: once the round ends, the platform fills in, once, the attempts the sandbox never got to, as the person who started the run. It goes the same way as clicking "Run the N that didn't run", and the audit log says "Platform ran the tests a lost sandbox never got to". A run is filled in by itself at most once, and a fill-in run's own run never is. Runs that stopped because they were cancelled, neared the time limit or ran out of credit or balance aren't filled in by themselves: you still click. Neither are lost runs whose test backend a person still has to check, nor any that can't make whole runs; these are refused as a click would be, and only logged.

This is the platform's problem, so the fill after a virtual machine disappeared mid-run isn't charged: neither its sandbox time, nor its agent's calls to the platform model, nor its judging, nor the lost sandbox's last stretch that left no result (see the platform's problem isn't charged). It goes ahead even when the organization's credit, balance or monthly spending cap is used up, held only to each run's own model and judge spending limits; it still doesn't when the organization is paused, its subscription has lapsed or the month's sandbox hours are used up. The job page says "Filled in by the platform: that test's sandbox was lost part way, so the platform runs the N attempts it didn't run itself; the results go into the round report. This was the platform's problem: this fill-in run isn't charged." The round report adds "Filled in by the platform (a sandbox was lost part way: the platform's problem, not charged)" after that part. A fill after a job "Lost contact" (its lease ran out) is billed as usual, and so is a fill you click. Switching off "Re-runs by the platform" in the project's Automations stops these fills.

Retest only what couldn't be tested

When a sandbox run has tests that couldn't be tested, "Retest only what couldn't be tested (N)" under the report's "Couldn't be tested" section, or "Re-run these rows" on its experiment's undecided rows, retests them in one go (see undecided rows and rerunning what couldn't be tested). The click doesn't start it yet: it first says how many attempts, which version of each test it runs (a version the platform fixed after that test says "it ran version N"), and whether it's charged; "Retest" starts it. It starts a new sandbox that runs again only the attempts that couldn't be judged:

• on the same commit (or image), with the same test versions and conditions, without a pre-run. Where the platform changed a test to settle it (the run's AI diagnosis's fix included) and that version is confirmed, the retest uses that version;
• the results are read into that run's round report: each undecided attempt (those the AI diagnosis found to be the platform's cause and set aside as "couldn't be tested (the platform's cause, not charged)" included) is replaced by one of the retest's attempts of the same scenario and condition, the latest try counting, and the run is judged again by its own pass criteria. The round report, its experiment and that part of the round read the same merged results. In the round report that part is "Retest of what couldn't be tested, N attempts", marked "Retest of what couldn't be tested: read into the part it retests", and the part it retests adds "N attempts that couldn't be tested retested, read in". The retested run's own report still shows its results as they were, and says at the top that what couldn't be tested there has been retested and that the round report counts the latest try;
• if the agent's code, prompt or model changed meanwhile, the results aren't read in; a change to the test connection alone doesn't stop that;
• sandbox time is billed. The job page says "Retest of what couldn't be tested: only the N attempts of that test that couldn't be tested run again; the results go into the round report, the latest try counting."
• when every attempt it retests was left untested because the platform's model failed (since 20 October 2026), or because the platform's own judge misread the reply, or because the AI diagnosis found the platform's harness, test or test environment at fault (since 25 October 2026), it isn't charged: the confirmation says so, the job page adds "This was the platform's problem: this retest isn't charged.", and the usage page lists it under "Retests of what the platform's model left untested".

When a test set's question couldn't be tested, its "what to do" says to test again from "Start a test" with "Only last run's failed or undecided" (a test set is retested that way, charged by usage), never "Retest only what couldn't be tested".

When it can't be done, the page says why: a run on a test runner, and a test set, are retested with "Only last run's failed or undecided" in the "Test new version" form; an archived scenario has to be restored under "Settings → Archived tests" first; a fill-in run or retest still going has to end first; a retest or a resumed run, a project now connected differently, test versions that are gone, or undecided attempts that can't be cut into whole runs need a new test.

What the platform handles itself

The platform does these by itself, and the page says what it did and why; whatever changes a setting can be switched back. Anything that spends more or could reach outside the sandbox is still a person's call.

• A sandbox that doesn't start: when the virtual machine hasn't reached the platform 12 minutes after starting, or the cloud couldn't create it, the platform starts another, free of charge. If that one doesn't start either, our engineers are told and the page says so; there's nothing for you to do.
• The wrong architecture: when a pre-run's log shows exec format error, a package missing for aarch64 and the like, the platform pre-runs again on the other architecture; if that passes, it switches the project's architecture, and "Settings" → "Sandbox" says so with "Switch back".
• A lost sandbox run: checked by the platform when every backend was inside the sandbox (above).
• Re-running undecided attempts: an attempt left undecided by a platform problem or a passing one is run once more within the same run, and the verdict counts the re-run; one caused by the platform's own problem isn't charged. See re-runs by the platform.
• Fill-in runs: when a sandbox is lost part way, the platform fills in what it never got to, once, free of charge (above); a run cancelled or stopped at a limit still waits for a person's click.
• Tests that couldn't be tested: each reason says what the platform is doing about it; when the organization has "Ask me before the AI changes things" off and "Re-runs by the platform" on (what the AI does by itself), a connection that lacks what the tests need is handed to "Let AI finish and check", and once its check passes only those tests are retested. See rerunning what couldn't be tested.
• Preparation that stalls: a step that prints nothing for 15 minutes is stopped, saying which step stalled (above).

Chat AI runs

A chat AI project starts runs from its "Start test" tab. Pick the chat AI; tick several to test them at once and compare side by side. Check the estimate of questions, judge model calls, cost and time. You can also tick "Compare with the last run", or rerun only the questions that didn't pass every time. The platform executor sends the questions. Judging cost per run has a limit: once it's reached, the rest isn't judged and the verdict can't be a pass. See Chat AI over HTTP.

Next steps

• Experiments: the run's scores, baseline and gate.
• Full run reports: read what the run found.
• Automations: run on a push, a pull request or a schedule.