Connect a GitHub repository

This is the recommended way to test an agent that calls tools and changes data. You install our GitHub App (read-only access to your code) and pick a repository and branch. For every run the platform starts a fresh virtual machine (a sandbox) in London. It clones the commit, installs dependencies, starts your test backend next to the agent and runs the tests. The virtual machine is deleted when the run ends.

Before you start

• A GitHub organization or account where you can install apps.
• An agent that can be started with a command (for example node dist/agent.js or uv run python -m my_agent) and that reaches its tools over HTTP or MCP.
• A test backend the agent's tools talk to, which can be started with Docker Compose, plus a way to reset it and read its data. Test environments and test credentials only: never point a run at production systems or real customer data.
• The keys your agent needs. They are stored as project secrets, below. A model key is optional: without OPENAI_API_KEY, the platform's model answers your agent.

1. Create a project and choose the connection

1. In the web app, click "New project", pick "Agent that calls tools", and name it.
2. Open Targets and choose "Connect a GitHub repository (recommended)" under the ways in.

2. Install the GitHub App

Click "Install the App on GitHub". On GitHub, choose Only select repositories and tick the repositories you want to test. GitHub sends you back to the wizard. If the platform can't take the installation, the wizard says why and what to do. For example, an installation already connected to another of your organizations has to be disconnected there first ("Disconnect" beside the account in step 1).

The App asks for the permissions below. It can only read your code; it writes only checks and its one comment on a pull request. GitHub requests an App's permissions as one set, so whoever installs it can't leave one out, and the Checks write permission is granted along with the rest.

Permission	Access	What it is for
Repository contents	Read	Clone the commit being tested into the sandbox
Metadata	Read	List your repositories and branches in the wizard
Pull requests	Read and write	Automatic runs when a pull request opens or updates; on projects with checks switched on, one comment on the pull request with the release gate's table, updated on every round
Checks	Write	Used only on projects with checks switched on: one check run per automatic run (the verdict and a link to the report) on the commit it tested

The platform doesn't push code, open pull requests or write to your repository's contents. What it writes, only on projects with checks switched on, is that check run and, for a pull request round, one comment on the pull request (found again and updated on every round, never a second one).

An installation made before the App asked for Pull requests: Read and write keeps Read until an owner of the GitHub organization accepts the App's permission update on GitHub. Until then a pull request round gets the check without the comment, and the check says what it takes to have the comment too.

An installation made before the App asked for Checks doesn't have that permission. An owner of the GitHub organization has to accept the App's permission update on GitHub before checks can be posted; until then, runs go on as before without a check.

When a sandbox clones, it gets a GitHub installation token that is valid for one hour and covers only the repositories you selected. The token is passed in a request header and isn't written into the checkout, and the agent's user can't read it.

3. Pick the repository and branch

Choose the account, the repository and the branch. The account is the one you just installed the App on (or added repositories to), else the one you last connected a project through; the repository is always yours to pick, and its default branch is filled in once you pick it. Before you save, the wizard shows the data boundary. The source code at the chosen commit is cloned into a sandbox in our cloud, and the sandbox is destroyed when the run ends. Only the run bundle (evidence, redacted traces and logs) stays, plus a dependency cache for 30 days. Tick "I understand" and click "Save binding". See the data boundary table for all connection methods.

4. The test connection file: .reviewctl/review.yaml

The platform reads the repository at the branch's latest commit and shows what it detected: the runtime, the lock file, any Compose files and whether the repository already has .reviewctl/review.yaml.

• If the repository has one, the sandbox uses it as it is.
• If not, the wizard offers a draft built from what it detected. Fill in the command that starts your agent and save it to the platform overlay. The overlay is the set of files the platform keeps for you and lays over the repository's .reviewctl/ folder in the sandbox. You can also keep a repository file and adjust a copy in the overlay. The overlay doesn't change your repository; to move the files into it, copy them and commit them yourself. A review.yaml that wouldn't load isn't saved, and the wizard says what is wrong and on which line, for example a missing start command, an unknown field, a key written out or a YAML slip.
• Helper files such as an adapter script (for example .reviewctl/harness.py) or test data can go in the overlay too. Under "Files in the overlay" in this step, click "Add a file", enter a path under .reviewctl/, then paste the content or load it from a local file. In the sandbox they land in .reviewctl/ along with review.yaml. Each file is text only, at most 256 KB.

Who wrote it

The overlay's file list has a "Written by" column: "You" (uploaded, saved or put back by a person here), "Your developer machine's AI" (saved by a coding AI connected to the platform) or "The platform" (written by "Let AI finish and check" and its fixes). Changing one key in a file (a value the platform writes into runner_env before a check, say), or saving it unchanged, leaves "Written by" as it was. For files saved before the column existed, the platform filled it in from the audit log; where it can't tell, it shows "—".

The platform's AI doesn't change files you wrote by itself. When an AI diagnosis says connection files need changing and some of them are yours, it isn't handed to "Let AI finish and check" automatically: the diagnosis says why, and leaves "Let AI fix the connection from this diagnosis" for you to click; a click fixes it as before. A diagnosis calls these files "the connection file you wrote", never "the platform's". They are still the test connection, not your agent's code.

Mistakes that raise no error

Some things read fine and fail no check, but won't run as written. The platform lists them when the overlay is saved, under "Test connection file" in this step, and in the "Preflight checks" of "Test new version: all options": "These parts of the connection files raise no error, but won't run as written: …". They are warnings; nothing is refused:

• The path of the reset command, the data reader's file or query catalog, scenarios.dir or a version file written from the repository root. They are all read from .reviewctl/, so [node, .reviewctl/seed.mjs] looks for .reviewctl/.reviewctl/seed.mjs. The warning names the line and how to fix it.
• A file in agent.version's tool_config_files or prompt_files that exists nowhere: the platform skips it, and the version comparison can't see it change.
• A runner_env variable that never reaches the agent: to hand it over, add it to agent.env_passthrough (see runner_env).

Two more are said elsewhere. Two routes joined by a comma on one line are refused on save, and the error names the line and asks for one route per line. That backend.reset runs as the test runner's user, so the agent can't read files it makes outside the repository's copy, is said in the run log (see review.yaml reference).

Let AI finish and check

If you'd rather not write review.yaml yourself, click "Let AI finish and check" at the top of this step. That click is the only thing you do. The platform does the rest, and the step shows each part as it happens:

1. Read the repository. It reads the head of the bound branch through the GitHub App, read-only: the tree, and up to 40 text files (150 KB in all) that say how the agent starts, such as the README, the Dockerfile and Compose files, dependency manifests, .env.example, the entry points and the modules they import, the text files that code opens by name (a policy or prompt file it loads at run time), and a few scripts and tests. It never reads .env, a file that looks like it holds a secret (keys, certificates, credential files) or a binary, and it hides anything shaped like a key in what it does read.
2. Write the connection files. The platform's model writes review.yaml. When your agent has no "one task in, one reply out" entry, for example a webhook bot whose replies go out later, it also writes an adapter such as .reviewctl/harness.py that calls the function that writes the reply. When the repository shows the agent calling its tools over HTTP or MCP, it routes those calls through the tool proxy. When the repository has a test backend that can be reset and read, it connects that too. Otherwise the connection sees replies only, and the step says so plainly ("This connection sees replies only…", "Changed data isn't checked…"). When the agent asks its user before it acts (for permission, or to confirm something), the adapter passes the question to the platform, which answers as the test says; it never approves for the user (see When the agent asks its user). While writing, it can try the agent in a trial sandbox, run as a check runs an attempt: the data reset first, as the test runner, in the same copy of the repository the agent then uses; the tool calls through the platform's tool proxy; and "yes" to any question the agent asks its user. The files are checked as your own save would be, then saved to the platform overlay in your name, "Written by" "The platform"; the audit log marks them AI-written.
3. Write a test, if needed. A check runs a confirmed test. If the project has none, the model drafts one or two small ones and automatic approval reads them. They are in the agent's own language (English for an English agent, as for the default security tests), and their dates are written the way a user says them ("yesterday"), never a fixed day that later falls outside the agent's own rules. If approval turns them down, the model rewrites them from the reasons, up to 2 times. If they are still turned down, it stops and asks you to confirm one under Datasets → "All tests (detailed list)". If someone turned automatic approval off for the project, the platform doesn't confirm a test in their place; it stops and says so.
4. Check once, up to 10 times. It runs "Check once" as you. When a check fails because of the connection, the model reads the result and the sandbox log (the lines that look like errors with the lines around them, then the end of the log). It also reads up to 5 repository files the error points to that it hadn't read yet, never a file that holds secrets, and why the earlier checks of this round failed and what was changed. Then it revises the files and checks again, at most 10 checks per click. A check that only the platform's own model or judge model failed (a rate limit, say; since 20 October 2026) isn't the connection's: it doesn't count against the 10, the model isn't asked to change anything, and the platform checks again by itself 5 and then 15 minutes later.
5. The result. "The connection works" with the next step ("Test new version"), or exactly what you have to do: add a secret it names, confirm a test, or why this repository can't be tested this way and which way in fits it better.

In the cases below the platform would start it by itself. By default it first asks on the project's Targets page: "The platform would spend about $X and about N min more fixing the test connection. Fix it?", $X and N what such a round usually costs and takes for this project. "Fix it" starts exactly what it would have fixed (the diagnosis, what the drafts need, the failed check); "Not now" does nothing and charges nothing, and when you want it fixed later, "Let AI finish and check" carries what those drafts need. Clicking "Let AI finish and check" while it asks is the same as "Fix it". A run you started that waits for its pre-run waits for your answer; one started by a push, a pull request, a schedule or the API has nobody at the page, so it ends as before and the question stays on Targets. An owner can switch "Ask me before fixing the test connection" off under "Policy" to have the platform fix it without asking (what the AI does by itself):

• A pre-run or a check fails (see Check once).
• The platform's AI diagnoses a run and finds the test connection at fault. It fixes from the diagnosis, unless files you wrote are among those to change: see Who wrote it.
• Automatic approval finds test drafts that are right but need a change to the test connection to run. It fixes what the review asks and writes no test of its own in step 3; once a check passes, AI reviews those drafts again.
• A run has tests that couldn't be tested because the connection lacks what they need (it can't read the data or see the tool calls), and the organization's owner switched "Ask me before the AI changes things" off with "Re-runs by the platform" on (what the AI does by itself). Once a check passes, only those tests are retested; see settling what couldn't be tested.

Under "What happened" it says who it was started for and why, and which run or drafts it is fixing for. A round you started by clicking "Let AI fix the connection from this diagnosis", or by clicking "Fix it" on the platform's question, says so ("The platform asked whether to fix it; … clicked "Fix it" at …"); only a round the platform started without asking says "The platform started this round itself, and it is billed to your organization".

While a question is open, a thing in "What this round was to fix" that passed in a later test says so, and isn't handed to the AI on "Fix it". A project has one question at a time. When the platform finds something else to fix before you answer, the newer question takes the older one in (the older card says it went into the newer one, and "Fix it" on it starts the newer round); while a round is already fixing the project, the platform asks nothing more, and the diagnosis keeps its "Let AI fix the connection from this diagnosis" for when that round is done.

When it is handed something to fix (what "Let the platform read the repo" listed under "The platform fixes these", a diagnosis, the change some drafts need), the model gets each thing with why, how and when it is done. If it hands back the connection files exactly as they are, the platform sends them back once; if nothing changes again, it stops there, saying which things weren't fixed and why the AI didn't change them (its own words; when it gave none, the last thing it said, or how many files it read and tries it made before it stopped), without running a check or charging anything more. "Details" shows what the platform sent back to the AI, as what it was asked to do, without the AI's form-filling instructions. Under "What this round was to fix" each thing then says "changed .reviewctl/harness.py" or "not done: .reviewctl/harness.py, which it names, wasn't changed". A check that passes after such a round says what it tested: each confirmed test's normal condition. After a diagnosis its checks also run the fault and attack conditions the diagnosis found failing, once each (a condition of the same name in another test runs too), so such a round's checks run a few more attempts. Where the connection the platform wrote still exits with its own error in one of them, the check doesn't pass ("the check ran …, and the connection the platform wrote still exited with its own error there"), and the AI goes on from that check's log and checks again; where none does, the pass says "Of what the diagnosis found, … ran in this check and the connection held". One the check can't run (the test no longer has that condition, say) is said not to have run — to know whether it is fixed, retest what failed from "Test new version" — and the card then says "The tests reach the agent, but … wasn't tested by this check" instead of "The connection works". One that ran but whose fault or attack never happened (the agent didn't call the tool it was on), and a thing the platform can't tie to a test and condition, aren't tested either: under "What this round was to fix" that thing says "this check didn't test it" and why. The question's "about $X and about N min" comes from this project's own repairs once it has had two or more, the AI's trial sandbox included, with the extra conditions its checks run. While a round the platform started itself is running, it says what such a repair usually costs and takes (the same figures the question gives), with "Stop this round"; "Have the platform fix these", "Have the platform try again" and "Let AI fix the connection from this diagnosis" say them too.

While the platform's AI writes the connection files, the last line under "What happened" follows it: "The platform's AI is writing: so far N model calls, read N files, changed N connection files…", updated at every step, with "last moved" and the model-call count moving with it, so a long write shows it is going; before the model's first answer it says "waiting for the model's first step (a step can take minutes, longer when it writes a long file)". While a step is still waiting for the platform's AI it says "step N is waiting for the platform's AI (N ahead, N min so far)" ("next in line" when it goes next); when the platform's model is rate limiting or didn't answer properly and the platform tries again after a wait, it says so, about when the next try is, and how many tries so far. Only once its turn comes does it say "step N, no output from the model yet (N min so far…)", counted from its turn, not from the time it waited. reviewctl connect --ai in the terminal and the developer AI's get_connect_assist show the same line.

Fixing the test connection changes the wiring only, never what is tested: the repository's own seed data, sample tickets and records, and fixtures stay as they are (malicious instructions or injected attack text in them included), never swapped for "clean" data. Such a thing in what "Let the platform read the repo" lists ("provide clean seed data without the malicious instruction", say) goes under what is likely wrong with your product, not to the platform. A round handed something to fix, or started by the platform, that ends without changing a single connection file isn't charged: the card says "Nothing changed, not charged", and what its platform AI (and its trial sandbox) cost doesn't count to your organization; the usage page lists it under "not charged".

The platform's AI being busy isn't your problem. Your organization runs up to 4 tasks on the platform's AI at once (connections, plans, AI approval, diagnoses…), and the model service may be rate limiting. When it isn't its turn, it waits in line until it is, for up to 2 hours, and the step says its place in line, about how long, and until when; you don't need to come back and click. First come, first served: "Let AI finish and check" and "Let the platform read the repo" take turns in the order they were clicked, 4 at once in your organization, so the first clicked are done first and a later click never goes ahead. While one waits, it says how many tasks are ahead, about how long (from how long each lately took) and until when, a time that doesn't move. Once its turn comes it says "Its turn came: the platform's AI is writing (minute N)", not a place in line; each step still waits for the organization's other AI calls, so some slowness is normal, and while a step waits it adds "this step is waiting for the platform's AI (N ahead)". Platform AI calls someone is waiting on ("Let AI finish and check", "Let AI fill in and test", the assistant, the starter questions drafted while a chat AI is set up) go ahead of background work such as diagnoses, AI approval and default security tests: when one of your organization's turns frees up, they get it first. Background work always keeps at least one turn, so it never waits forever. While it runs a check it gives its turn to the next task, and if the check sends it back to the AI it goes ahead of the tasks clicked after it. While it waits, its trial sandbox goes to another test or check of your organization waiting for a slot. Only after 2 hours without its turn does it stop, saying it's the platform's problem; clicking again puts it back in line.

Other problems on the platform's own side don't stop it either. The model may give no usable answer, the check's sandbox may not come up or be cancelled by the platform, or the platform may restart several times. In those cases the platform goes on by itself 5 minutes later, or 15 minutes later the second time, at most twice, and the step says when. If the second retry fails too, it stops and says it already tried again. A check the platform cancelled says why; one stopped at a cost limit isn't retried (it would meet the same limit). When another test or check is running in the project, the check waits for it to end (looking every 5 minutes; this isn't a retry). Clicking "Start test" while it works never waits for it: the test starts, the trial sandbox goes to it, the AI goes on writing, and the check comes after that test ends. A suspended organization or a used-up allowance isn't retried. A check whose preparation stalls (a step printing nothing for 15 minutes) isn't the platform's side: it counts as a check, and the model changes the step it stalled at; see preparation that stalls.

If the round the platform runs for a failed pre-run used up its 10 checks or 5 rewrites, and you then change the code (push a commit), a connection file or a secret, the platform gets one last round: when the waiting run's next pre-run fails too, it fixes it once more.

When the same problem comes back twice running (the same step, the same error), it stops and waits for you: nothing more is changed or checked, and nothing more is charged. It says whose problem it is and what the round has cost so far, with a "Try again" button beside it. Once stopped, the platform starts no round of its own (not for a failed pre-run, a diagnosis or AI approval of drafts) until you click "Try again", change a connection file or a secret, or push a commit. Every earlier round stays: under the button, "N earlier rounds" shows how each started and ended and what it cost.

What it doesn't do:

• It never changes your repository and never opens a pull request: it writes to the platform overlay only, and you can view, edit or delete every file it wrote there.
• It never sets a secret. When the agent needs a real third-party key the platform can't give, it stops before checking and names it, what it's for and where to get one (not OPENAI_API_KEY: without one, the platform's model answers the agent). Add it under "Secrets", then click the button again: it goes straight on to checking with the files it wrote. When the repository's docs or code name a way to run without that key (another search service that needs none, say), it says so too, with "Use the way without a key" beside it: click it and the platform writes that setting into agent.env in review.yaml, no longer asks for the key, and goes on checking. The agent is then tested that way, so results may differ from the original service. The platform only offers what the repository itself names, and changes nothing until you click; a test plan that chose the way without a key is connected the same way. A key still missing is also pointed out above "Start test" in the "Test new version" form, with what goes wrong without it.
• It doesn't fix your agent. A check that fails because of the agent's own behaviour stops there and says so: the platform only tests.
• It doesn't change the settings of the agent under test. In review.yaml, the variables and values already in agent.env and runner_env, the names already in agent.env_passthrough, and agent.model belong to the agent you're testing (turn off its environment scrubbing, say, and it's no longer the same agent). When the AI removes or changes one, the platform puts it back, saves the rest, and "What happened" says so, for example "The AI wanted to change MAVIS_BASH_ENV_SANITIZE in agent.env; the platform kept it: it's the agent under test's own setting, not the test connection". It may add new variables.
• It doesn't loosen anything. Settings a person decides are locked: it can't turn on attach.process_egress, change attach.enforce_forbidden_writes, use attach.apply: in_place, add a cloud section, choose another judge model, raise the per-attempt limits, set proxy or preload variables (HTTP_PROXY, NODE_OPTIONS, …), point an address outside the sandbox, or add install commands other than a package manager's. Where it wrote one of these, the platform puts it back and lists what it changed. Text in your repository is read as data only: instructions written there for an AI are ignored.

The model's calls count against the assistant's daily allowance, which is held before each call. Each check is billed by sandbox time, like a manual one. If no model is configured on the platform, the button is off and says why. You can still edit everything by hand, before, during or after. Only one runs at a time per project, and "Cancel" stops it along with its check in progress.

A repository layout that works:

your-repo/
├── .reviewctl/
│   ├── review.yaml            # how to start and test the agent
│   └── scenarios/             # scenarios and the data queries (state_queries.yaml)
├── docker-compose.test.yml    # the test backend
├── package.json / pyproject.toml / go.mod …
└── src/ …

An example review.yaml:

schema_version: 1
# the platform replaces this id with your project's own when it runs in the cloud
project: { id: prj_01J8ZK3Q9MXYZ0000000000000, name: expense-agent, domain: expense }

environment:                     # optional: how to install dependencies
  setup: [pnpm install --frozen-lockfile]

services:                        # optional: the test backend, started next to the agent
  compose: docker-compose.test.yml

agent:
  driver: command
  command: [node, dist/agent.js, --message, "{{task}}"]
  env_passthrough: [OPENAI_API_KEY]   # values come from project secrets

attach:                          # the tool proxy: the agent's tool calls go through the platform
  mode: http_proxy
  base_url_env: EXPENSE_TOOLS_BASE
  upstream_base_url: http://127.0.0.1:47591/tools
  routes:
    claim_list: { method: POST, path: /claim_list }
    claim_submit: { method: POST, path: /claim_submit }

backend:
  mode: own
  reset: { type: http, url: http://127.0.0.1:47591/reset, state_ref_path: $.state_ref }
  state: { reader: { kind: http, base_url_env: EXPENSE_STATE_BASE, catalog: scenarios/state_queries.yaml } }

runner_env:                      # optional: values that aren't secret, for the test runner itself
  EXPENSE_STATE_BASE: http://127.0.0.1:47591

judge: { model: "openai:gpt-4.1-mini" }
scenarios: { dir: scenarios }
budget: { per_attempt: { minutes: 2, model_calls: 30 } }

The sandbox runs the agent command from the repository root. Services are started with Docker Compose on the same virtual machine, and the agent reaches them on 127.0.0.1, so in your own Compose file, publish the ports it uses on 127.0.0.1 (for example "127.0.0.1:47591:47591"). If the repository has a Dockerfile or a devcontainer.json, the wizard says "The sandbox will build the repository's Dockerfile" (or its devcontainer.json) and drafts an environment.container section instead of install commands. Untick it to install on the virtual machine instead. Every section and field is in the review.yaml reference.

5. Choose a class

The class sets the size of the sandbox; Standard (2 vCPUs, 8 GB memory, 30 GB disk) suits most agents. See sandbox classes.

Recommended class. The platform recommends a class from the test connection and the logs of the latest sandbox runs. With 4 or more containers beside the agent (compose services, plus the agent's own when the sandbox builds the repository's container), it recommends Large, and with 8 or more, Extra large. When a log says the memory ran out (Killed, out of memory, exit code 137), the disk filled up (No space left on device) or the setup ran out of time, it recommends a class bigger than that run's. The recommended class is shown in this step (and in "Settings" → "Sandbox") beside the class the project runs on now, with why and which run's log said it; the class box keeps showing your current class, "Switch to Large" applies the recommendation at once, and "Keep "Standard"" (your current class) puts it away. Once the test connection says what it installs, only that counts: a very large package in a dependency file the connection doesn't install is no reason. A bigger class costs more per hour, so the platform never changes the class by itself, and never recommends a smaller one.

Run location. In the sandbox settings, a project's run location is Auto (the default) or London; both run the sandbox in London. Sweden (EU) is no longer offered since 24 October 2026 (see run location). Where your data is kept: data processing.

6. Add project secrets

The "Secrets" section first lists the names the platform found in review.yaml and your confirmed tests (agent.env_passthrough, the address variables of the state reader and the data reset, the tests' identity variables), each with who gives it now: "Added (project secret)", "In review.yaml's runner_env", "Provided by the platform: nothing to add", "Not a secret: the platform can write it into runner_env" or "Missing: add it before the check". The platform sets the tool proxy's address (attach.base_url_env) in the sandbox, so there's nothing to add for it. The first missing name is already filled into the form, and clicking another name fills that one in; add the value and click "Add". Values are sealed as they arrive. They reach only the sandbox running your job and are masked in its logs. A judge model the platform provides needs no key of yours. See project secrets.

When a secret you added isn't used by the test connection file, this section says so, because the agent only gets the variables listed in agent.env_passthrough. "Add to the connection file" adds them to agent.env_passthrough in the overlay's review.yaml. If the overlay has no review.yaml yet, it copies the repository's there first. Step 4's editor then shows the file as it is now, so you don't have to go back and edit it by hand.

A value that isn't secret, such as the state server's address in the example above or a test account of a test backend your repository starts, goes under runner_env in review.yaml instead of being a secret. A secret of the same name still wins. When the platform finds such a value in the repository's .env.example, a port a Compose file publishes on 127.0.0.1, or the test backend's own address, "Write into runner_env" writes it into the overlay's review.yaml. The repository's copy stays as it is, and if the overlay has no review.yaml yet, the platform copies it there first. "Check once" writes it in by itself when the overlay has review.yaml, and says so on the page. For a name that looks like a secret, the platform never takes a value from the repository. A repository whose test backend brings its own test accounts may need no secrets at all. See secrets or runner_env.

Your agent's model key is optional too. Without OPENAI_API_KEY, the section says the platform answers the agent's model calls through the sandbox and bills them by usage. If the agent asks for one of the platform's models, that model answers; otherwise the project's model does (models). To use your own key, add an OPENAI_API_KEY. See your agent's model.

7. Write tests

Open Datasets → "All tests (detailed list)", and under "Let AI draft your tests" click "Let the platform read the repo" (recommended for a repository project). The platform's model then reads the repository, read-only (the README, the entry code, the tool definitions, the prompts and business rules, the seed data and fixtures), and writes functional tests and the capability list using the names in review.yaml. For the test files those tests need, the platform copies the ones in the repository by their path, and AI writes samples of any TXT, CSV, JSON or XML file the repository doesn't have. PDFs and images are for you to upload (under "Attachments" on the test's row in the dataset). Automatic approval reviews the drafts as usual, and the platform revises the ones it turns down. Once the first functional test is confirmed, the platform also writes a default set of security tests. You can also let the dev AI read the code, write attack tests under "Generate an attack dataset", create a "New test set", or "Write a scenario" yourself. See Scenarios and test sets. Confirm the tests you want to keep.

8. Check once

Back in the wizard under Targets, click "Check once". It runs one of your confirmed tests. With none confirmed yet, the platform checks with its own question: it asks the agent to introduce itself. That only shows the agent starts and replies, not that it gets the job done. If the sandbox has no identity variable, the platform adds a synthetic test identity, AGENT_TEST_USER, to the overlay's runner_env and says so.

Before the check, the platform looks at the values review.yaml and the tests read by name. If a secret the check needs is missing, the check doesn't start, and the page names the value and where it's used; if you know the agent does without it, click "Check once anyway". A test's identity variable that isn't there yet is pointed out beside the test. Without it in the sandbox, the pre-run turns that test down.

The platform starts a sandbox, clones the repository, installs dependencies, starts the services and runs that test once, to see that the test really reaches your agent. You can watch each phase and the logs live: queued, booting, preparing, pre-run, uploading, deleted. A check gives a pre-run result for diagnosis, not a test report, and doesn't count as a run. You can also skip it, since "Start test" pre-runs first by itself.

When a check or a pre-run runs two attempts or more, the last one answers the agent's first tool call with a timeout on purpose (the tool isn't run), to see whether the test connection hands that error to the agent. A connection file the platform wrote that can't take it and exits with its own error fails "Does the connection hand a tool's error to the agent", and the platform fixes it: otherwise every fault and attack a run injects would end the connection midway, and how the agent copes couldn't be tested. Where what is tested as the agent is a harness the platform wrote (standing in for it), that harness exiting with its own error after the error reached it fails the check the same way, and the platform fixes it; it is never said to be "the agent's own program". That attempt's reply isn't read for the other checks, nor counted in their "N of N attempts", which say "Besides: 1 attempt carried the platform's deliberate tool error".

The wizard ends here. Once a check passes, it points to the project's Automations, where push and pull request runs, the schedule and API tokens are: see Automatic runs below. The latest check stays under Targets → "Connection check".

If it fails, the page shows what went wrong and the sandbox's logs. For a repository project, the platform's AI then fixes the test connection from the check's result and log and checks again, the same way "Let AI finish and check" does. This check counts as check 1, there are at most 10 in all, and the page shows each step. When the AI can't fix it, it says what you need to do.

If it has just given up and nothing has changed since (no new commit, and nobody changed the connection files or the secrets), the platform doesn't fix the same check again by itself, because it would most likely end the same way. "Let AI fix it from this check" on the page hands that check over in one click.

Typical fixes are a missing project secret, a port not published on 127.0.0.1, or a wrong agent command. When the repository's Compose file reads .env through env_file, the sandbox can't start the services, because a .env is rarely committed. Leave services out if the test doesn't need them, or write them as named services. A connection file the platform drafts leaves services.compose out in that case and says why. An exec format error or a package that can't be found for aarch64 means the architecture is wrong; a package manager skipping an optional package built for another system (bun's skipping …-darwin-arm64, npm's SKIPPING OPTIONAL DEPENDENCY) is normal and doesn't count. The platform then checks once more on the other architecture by itself. If that passes, it switches the project's architecture and says so under "Settings" → "Sandbox", where you can switch back. More in troubleshooting.

9. Start a test

Click "Test new version" (top right of any section), choose what to test and how many times per test, then click "Start test". It pre-runs once first, and the real run starts when the pre-run passes; at 1× with the normal condition only, the real run is what the pre-run would run, so there's no separate pre-run and the run itself is the pre-run. You can cancel at any time. The run stops at the end of the current attempt and still produces a partial report.

10. Read the experiment

The run becomes an experiment under Experiments, with the release gate at the top, a tile per scorer and one row per test and condition. Its "Full report" shows the run's own verdict: "Passed", "Not passed" or "Couldn't be judged". It passes only when every test passed every time; when nothing failed but results are missing, it's "Couldn't be judged". Each test is shown as passed, failed or couldn't be tested. For each problem you get the conversation and the data checks behind it, and how to fix it; "Export fix notes" on an experiment row saves them as a file for your coding AI. When the agent's tool calls go through the tool proxy (attach in step 4), each tool call with its arguments and results is there too. With attach: { mode: endpoint_only } the platform sees only replies and data, so the report says "Tool calls not visible" and an attack judged only by tool calls is undecided. See what the report shows. The version of a cloud run is the commit that was cloned, plus the model and the machine image.

What happens during a run

1. The platform creates the job and a one-time bootstrap token (valid 10 minutes), then creates a virtual machine from its machine image in the sandbox network.
2. On the virtual machine, the test runner exchanges the bootstrap token for a session token that works only for this job; the bootstrap token is then void.
3. The test runner fetches the overlay files and the project secrets, clones the commit, restores the dependency cache, runs setup and starts the services. With no OPENAI_API_KEY among the secrets, it also starts the local relay to the platform's model.
4. Each attempt runs the agent as a separate, unprivileged agent user in its own resource-limited scope and working directory. Logs stream back to the page.
5. The test runner uploads the run bundle and reports completion; the virtual machine shuts down and the platform deletes it.

If the virtual machine hasn't reached the platform 12 minutes after it was created (step 2 never happened, and the bootstrap token has expired), the platform deletes it and starts another by itself. It's the same run, it isn't charged, and the page shows the reason and the first machine's console output. The run ends only if the second machine doesn't reach the platform either. The platform then sends the job id and both machines' boot logs to our engineers, and the page says "The platform's engineers have been told". There's nothing for you to do. The same applies when the cloud can't create the virtual machine: the platform tries once more a minute later.

If a virtual machine disappears mid-run, the run is marked incomplete, and the attempts already run are judged as usual. Once the round ends, the platform fills in the attempts it never got to, once, by itself; the results join the round report, and the audit log says "Platform ran the tests a lost sandbox never got to". This was the platform's problem: that fill-in run isn't charged, nor the lost sandbox's last stretch that left no result (see the platform's problem isn't charged). A lost run whose backends may be outside the sandbox and still need a person to check them isn't filled in: your agent may already have changed data in the test backend, so the page asks you to reset the test backend, then fill in or start again. Switching off "Re-runs by the platform" in the project's Automations also stops these fills.

Limits and allow-lists

Your organization's policy limits cloud runs. By default, 2 sandboxes run at once, further runs wait in the queue, and a run can last 120 minutes. Sandboxes reach the internet on port 443 through one fixed address in London, which you can add to your test backend's allow-list. See Sandboxes and limits.

Automatic runs

In the project's Automations (the wizard links there at its end once a check passes), switch on runs for every push to the default branch and for pull requests (not from forks), and optionally a GitHub check with the verdict on each commit tested. Schedule and API tokens on the same page run tests on a schedule or from your own CI. For how automatic runs work, whom they run as and what they post on GitHub, see Automatic runs.