Local test runner

Keep the agent on your own machine. A small program called the test runner runs there. It takes tests from the platform, runs them against your agent locally and uploads the results. It only makes outbound HTTPS connections, and the platform never connects to your machine. Choose this when your agent depends on systems inside your network, or when code and data must stay in-house.

Before you start

• A dedicated test machine (macOS or Linux) that can run your agent and reach its test backend. The agent runs there as an ordinary process, without container isolation, so don't use a machine that holds production credentials or data.
• Node.js 22.5 or later.
• Outbound HTTPS (port 443) to the platform's address. No inbound ports.
• The agent's repository on that machine, with its own dependencies installed (its Python virtual environment, npm install and the like), so the agent runs there by itself. The test runner only runs it; it installs nothing.
• The agent's model key: optional. With no OPENAI_API_KEY (and no OPENAI_BASE_URL) on that machine, the platform's models answer the agent, billed by usage as in a cloud sandbox (from 29 October 2026): reviewctl opens a relay on the machine's 127.0.0.1, gives the agent its address and a placeholder key, and passes the calls on to the platform with the test runner's own token; nothing to create or copy. An agent on the OpenAI SDK (which reads those two variables) works as it is. To use your own key, set OPENAI_API_KEY in that machine's environment or the repository's .env (values are never uploaded). An agent that defaults to another provider's model (Anthropic, Gemini and the like) uses that provider's key when the machine has it; without it, reviewctl connect --ai first switches the agent to the platform's model through the agent's own settings (MODEL=openai/gemini-4-argon in agent.env, say), billed by usage, and asks you for that key only when the agent can't be switched (it only uses that provider's SDK, or its model is fixed in the code).
• The platform's gateway still works too (for the agent's everyday use outside tests, say): on that machine set OPENAI_BASE_URL=<platform address>/gw/openai/v1 and a project token as OPENAI_API_KEY (create one under Automations → "API tokens" with "Allow uploading logs and experiments" ticked, which lets it call models), with a model <platform address>/gw/openai/v1/models lists. The agent's calls go through the AI Gateway to the platform's models, billed by usage, and show in the project's logs. The review.yaml the AI writes puts OPENAI_BASE_URL in agent.env_passthrough, and the platform doesn't take that address out as a public one.
• Nothing to set up for the judge model: the platform provides it, billed to the organization by usage as the platform's judge is in a sandbox (see the last paragraphs of "Install the CLI"). The review.yaml the AI writes names the project's model as the judge. Only when review.yaml names a judge model of your own and that machine has its key is your own used; the platform's project token (pj_…) is no such key.
• One project, one way in: a project already connected through a GitHub repository or its own image is tested in the cloud sandbox. To test on your own machine, create a new project and choose "Local test runner" under Targets. Run the command below on such a project and the platform refuses it before it registers the test runner or uses the pairing code.

The quickest way: let AI finish and check

Rather than write review.yaml and an adapter yourself, you can run one command in your agent's repository. The platform's model reads the repository and writes the test connection files under .reviewctl/. The command then pre-runs a check on your machine; when the check fails, the model revises from the result and checks again, up to 10 checks.

1. Install reviewctl (step 1 below) and sign in once: reviewctl login --api https://<platform address> (approve it in the browser).
2. On the web, open Targets and choose "Local test runner" (the self-hosted runner). Under "Let AI finish and check: one command" you get the command with your project's id filled in. If this machine isn't a test runner of the project yet, click "Create pairing code" first, and the code is added to the command (valid for 10 minutes, works once).
3. In your agent's repository root, run:

reviewctl connect --ai --project <project id> --code ABCD-EFGH    # a machine that already is a test runner needs no --code

Someone else looks after the machine that runs the agent? Open "Instructions for your developer" below the pairing code button, create a pairing code, and send all of it to your developer: it's the steps above, with the pairing code in the command. They sign in with their own account in your organization; without one, the organization's owner gives them an invitation link under "Organization → Members".

What it reads and sends. The command reads the repository on your machine by the same rules the platform uses for a GitHub repository. It reads the README, dependency manifests, container and Compose files, .env.example, the entry points and the modules they import, the text files that code opens by name (a policy or prompt file it loads at run time), modules that look like the agent, and a few scripts and tests. That's at most 40 files and 150 KB in all; a file over 24 KB is cut to its beginning (at a character's boundary, never through one), and a file over 200 KB isn't read. It also sends the names (not the contents) of up to 1,500 paths. In a Git repository it looks only at the files Git tracks or doesn't ignore.

It never reads .env or other files that may hold secrets (it sends only their names), binaries, dependency and build folders such as node_modules, links out of the repository, or reviewctl's own run records. Anything that looks like a key is removed on your machine before sending, and again when the platform receives it. When an older reviewctl sends more than the size caps, the platform fits it to them and says, in the first line of "What happened", which files it read the beginning of and which it didn't read; an upload with a file it never reads (a secret file, a binary, a path out of the repository) is refused as a whole. Before sending, the command lists each file and its size and asks you (Enter means no; --yes skips the question). The uploaded repository text is kept only until that "Let AI finish and check" ends.

Writing the files. The command writes the review.yaml and adapter the AI wrote into the repository's .reviewctl/, and nowhere else. If the platform sends any other path, the command refuses the whole round and writes nothing. A file that's already there is backed up to .reviewctl/.backup/<time>/ first. Before writing, the command lists each file as added, changed or unchanged, shows the command the check will run, and asks you once (Enter means yes). .reviewctl/.gitignore ignores .review/ (the checks' run records) and .backup/. Look the files over, and commit them yourself if they look right.

Checks. Each check is a pre-run on this machine. The command acts as the test runner and runs one test with .reviewctl/review.yaml, by the same rules as reviewctl runner start. A local test runner doesn't install dependencies, build containers or start services, so have what your agent needs installed and running on this machine (or your network) first. The platform takes out any environment or services section the AI writes. When the program the check runs isn't in the repository yet (.venv/bin/python, say), when it runs a bare python3 while the repository's dependencies are in its own virtual environment (.venv or venv), or when the repository has dependencies but no node_modules, the command says so as soon as it writes the files. Each attempt runs in a temporary copy of the repository; see "Running in a temporary copy of the repository" below. When a check fails, the command prints the error under the check's line (keys removed) and sends the lines of its log that look like errors, after removing secret values and anything that looks like a key on your machine, together with what it found missing above; the AI revises from them.

Secrets. Only names are sent, never values. On a local test runner, the agent's third-party services use this machine's keys; its model calls use this machine's OPENAI_API_KEY when there is one, the platform's models otherwise; the judge model is the platform's (your own when this machine has its key). The AI never counts OPENAI_API_KEY as a missing secret. When the AI says the agent needs a secret, the command looks for it in this machine's environment and in the repository's .env, which it loads. If any is missing, it names them, again at the end, and the result on the web page says so too. Once they're set, run the same command again: it goes on checking from the files it wrote last time, without asking the model again.

The test identity. The synthetic test identity the AI writes into review.yaml's runner_env (AGENT_TEST_USER: test-user-1, not a real account) counts as an identity variable this test runner allows as soon as the files are written, so the tests the AI writes next use it, with no --allow-auth-ref.

One already running. A project has one "Let AI finish and check" at a time. When the last command was stopped, the command says which machine it was on and when it last answered, and asks whether to cancel it and start again (--yes never cancels one for you). You can also click "Cancel" in the "Local test runner" wizard under Targets on the web; left alone, the platform stops it once reviewctl on that machine hasn't answered for 10 minutes.

How it ends. When a check passes, it exits with 0; then keep the test runner running with reviewctl runner start --config .reviewctl/review.yaml. When it stops, it says why and exits with 1. The reasons are: a secret is missing, a test needs confirming (the project has AI auto-approval off), the repository can't be tested this way, the check failed because of the agent itself, 10 checks failed, or the assistant's daily allowance is used up. If you say no, press Ctrl-C or it runs too long, it's cancelled and exits with 2: at the first Ctrl-C (or when the terminal closes) the command tells the platform at once, and the platform's AI stops before its next model call, while the check running on this machine winds down; press again to quit at once, also with 2. When the model provider turns down the agent's key (a project token used as an OpenAI key, say), the AI leaves the connection files alone and stops, saying to set a valid key, or to remove OPENAI_API_KEY from this machine so the platform's models answer, with exit code 1; once that's set, run the same command again and it goes on checking from the files written this time. The web page shows each step as it happens, like the GitHub wizard's "Let AI finish and check". While the platform's AI writes, the command says every so often where it is: while a step is still waiting for the platform's AI it says "step N is waiting for the platform's AI (N ahead…)", and only once its turn comes "no output from the model yet" or how much it has written (see Let AI finish and check).

Costs. The model calls count against the assistant's daily allowance. The checks run on your machine, so no sandbox time is charged; a judge model the platform provides is billed as judge model usage. The platform locks the same settings as for a GitHub repository (the judge model, the per-attempt limits, proxy variables). On a local test runner, addresses can only be on this machine or your internal network, never a public address, except the platform's own AI Gateway (<platform address>/gw/…). For the command and its options, see CLI.

To do it step by step yourself, follow the steps below.

1. Install the CLI

The test runner is part of the reviewctl command-line tool. The platform serves its install package itself, with a version number, at https://<platform address>/downloads/. First see which version is the latest, and its checksum:

curl -fsS https://<platform address>/downloads/latest.json
# {"version": "0.1.20260930120509", "file": "reviewctl-0.1.20260930120509.tgz", "sha256": "…", "size": …, "node": ">=22.5", "built_at": "…"}

Download that file and the checksum published next to it, check it, then install it:

curl -fsSO https://<platform address>/downloads/reviewctl-<version>.tgz
curl -fsSO https://<platform address>/downloads/reviewctl-<version>.tgz.sha256
shasum -a 256 -c reviewctl-<version>.tgz.sha256    # on Linux: sha256sum -c reviewctl-<version>.tgz.sha256
npm install -g ./reviewctl-<version>.tgz
reviewctl --version                                 # e.g. 0.1.20260930120509+2e690dd

You can also skip the check and install straight from the address: npm install -g https://<platform address>/downloads/reviewctl-<version>.tgz. Either way, npm fetches the CLI's open-source dependencies from its registry.

EACCES (permission denied)? With the system's own Node (Ubuntu's apt install nodejs, say), the global folder needs root. No sudo needed: install it into your own folder and add that to PATH:

npm install -g --prefix ~/.local https://<platform address>/downloads/reviewctl-<version>.tgz
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && . ~/.bashrc    # for zsh, ~/.zshrc
reviewctl --version

The CLI and the test runner speak English unless the system language is Chinese; set REVIEWCTL_LANG=en or zh to choose. Every command and option is in Command line.

Each release of the platform comes with a new version (its last number is when it was built). When the platform has a newer version, this test runner's row under Targets → "Test runners" says "Version x.y.z is available" and gives the one-line upgrade command; reviewctl connect --ai and reviewctl runner start say so in the terminal too when they start. Restart the test runner after upgrading.

Have "Restart" update it too

A reviewctl installed with npm install -g can be updated by the platform: start it on the test runner with reviewctl runner start --config review.yaml --self-update. Each test runner opts in for itself; without the flag nothing changes. From then on, when it is behind the platform's version, its row's button reads "Update and restart", and once the test runner is idle it:

1. downloads the package from the platform it is registered with (only a file under /downloads/ at that same address);
2. checks the downloaded file's SHA-256 against the checksum the platform publishes with the package, handed to it with this restart;
3. installs it with npm install -g <the file just checked> only when they match (no shell; npm fetches the open-source dependencies from its registry as usual), then restarts.

If they don't match, the download fails, or the reviewctl on that machine wasn't installed with npm install -g, nothing is installed. It restarts on the version it has and writes the reason in the test runner's log, and its row on the web says the last "Update and restart" didn't install, with the command to upgrade by hand. The checksum comes from the same platform as the package, so it guards against a damaged or swapped download, not against the platform itself.

The judge model. A local test runner uses the platform's judge model too (from 22 October 2026): the test runner calls it through the platform's metered judge proxy with its own runner token, and the platform's model key never leaves the platform. When review.yaml names a model the platform provides, the project's model judges in its place, with the second judge and embedding model as in a sandbox. It's billed to the organization as judge model usage, with the same per-job cap as a sandbox. The replies to judge and the tests' points then go to the platform's judge model. If you'd rather not, set the key for the judge model review.yaml names on that machine (OPENAI_API_KEY for openai:gpt-4.1-mini, say): the test runner then uses yours, and the replies don't go through the platform. The test runner's log says at the start whose judge it uses. When review.yaml names a model the platform doesn't provide and the machine has no key for it, there's no judge model.

The agent's own model calls the same way (from 29 October 2026): with no OPENAI_API_KEY on that machine, the platform's models answer the agent through the platform's metered model proxy, with the test runner's own token and a sandbox's spending caps; the test runner's log says at the start who answers the agent this time. With OPENAI_API_KEY set, your own is used.

2. Write review.yaml

1. In the web app, click "New project", pick "Agent that calls tools", and name it. Open Targets and choose "Local test runner". The page shows the first lines of review.yaml, including your project's id.
2. In your agent's project directory, generate a starting configuration (--project takes the project's id from the web page: registering the test runner needs project.id in review.yaml to be that id), then fill in how to start the agent, its tools, and how to reset and read the test data:

reviewctl init --project <project id> --domain <your business domain>    # e.g. prj_01J8…, expense or generic
reviewctl doctor                                                          # checks reset, tool observation, fault injection and data reading

Every section and field is in the review.yaml reference. reviewctl loads a .env file next to review.yaml. The agent under test receives only the variables review.yaml passes to it (agent.env and env_passthrough) plus basics such as PATH and HOME, never the rest of the machine's environment. Secret values never go into review.yaml or into reports.

3. Register the test runner

In the "Local test runner" wizard under Targets, click "Create pairing code". The code is valid for 10 minutes and works once. "Create a new pairing code" voids the one on the page at once: registering with it is refused, saying it was voided. On the test machine:

reviewctl runner register --api https://<platform address> --code ABCD-EFGH --config review.yaml

The test runner's credential is stored in ~/.reviewctl/runners/<project id>.json, readable only by your user. You can revoke it on the web page at any time.

Not doing the setup yourself? Use the one command of "Let AI finish and check" above, or put it into a sentence for your coding AI (the "dev AI"), which then connects to the platform, fixes the test connection and checks again by itself. A dev AI on "MCP over HTTP (no reviewctl needed)" needs no reviewctl itself, but a local project's pre-runs and tests run on the test runner, which still needs reviewctl installed and reviewctl runner start running. When the dev machine is the test runner, the coding AI connects with reviewctl dev connect … --runner, and its MCP server starts this test runner when it starts; see Using the platform from a coding AI.

4. Keep it running

reviewctl runner start --config review.yaml

Run it under systemd or launchd so it comes back after a restart. While it runs, it asks the platform for work every few seconds and reports a heartbeat, and the page shows it as online. Press Ctrl-C once to stop after the current attempt and upload a partial report, or twice to stop at once.

Each attempt is judged as soon as it ends, while the next ones run (since 5 November 2026), so the report comes soon after the last attempt instead of after judging them all. Up to three attempts are judged at once. Each attempt's judge calls still go one after another, and it is judged exactly as before. While your agent waits on the platform's model, the judge asks one thing at a time. If the judge model is rate limited, unavailable or too slow while several calls are out, the run goes back to one at a time for a while; the second judge model isn't asked in its place for that. Once it has gone one at a time for at least a minute and 5 calls in a row were answered, it judges three at once again; each time it has to go back to one at a time, it waits twice as long before trying three again (at most 10 minutes). Close to a spending limit, the judge counts each call it has out at the most it may cost, and sends one more beside them only while those fit under the limit (the new call's own most not counted; since 5 November 2026, so room for one call at its most lets two go at once), one at a time when even the one out doesn't fit; a run never goes past its limit by more than one call, as when it judged one at a time. Once its calls out then are back and a few more are answered, it tries one more at once, so it judges three at once again if the limit is raised or more room is left (before 5 November 2026, it stayed at one at a time to the end of the run). The report's JSON says when and why the judge went one at a time (usage.judge.concurrency). An attempt the platform may run once more is judged once its try that counts has run; a test with a simulated user is still judged after the last attempt.

Running in a temporary copy of the repository

The agent under test may edit files and commit code (aider does), and its tools (a filesystem or Git MCP server, say) may write the repository. So the test runner, the checks of connect --ai included, copies the repository into the system's temporary folder for each attempt: .git is copied whole, so the agent's commits stay in the copy; .venv and node_modules are linked to yours; uncommitted files such as .env are in the copy too. The data reset, the data reads and the stdio tool servers the platform's tool proxy starts run in the copy as well. The next attempt gets a fresh copy, a job's copies are deleted once it's done, and your repository stays as it is. The first time it copies, the log says where the copy is and how big.

• A repository that is a git worktree or a submodule (its .git is a file pointing elsewhere) gets no .git in the copy.
• What is copied is the folder the agent runs in: the repository's root when review.yaml is in its .reviewctl/ (as "Let AI finish and check" writes it), or a folder that is a Git repository's root. A review.yaml in some other folder of a repository isn't copied (it may use paths above its folder): every attempt runs right there, and the log says so.
• A repository over 500 MB (.git included, .venv and node_modules not) isn't copied: every attempt runs in your repository itself, and the log says so. Give the test runner a clone of its own then.
• The linked .venv and node_modules are still yours: packages the agent installs into them change them.

What the platform can and cannot do through a test runner:

• It can choose which confirmed tests to run, under which conditions, and how many times.
• It cannot send commands. The agent's command and its environment come only from the review.yaml on your machine, and the test runner rejects any job field it doesn't know.

5. Write tests, pre-run, start a test

1. Under Datasets → "All tests (detailed list)", write tests with "Let AI draft your tests", "New test set" or "Write a scenario", and confirm the ones you want to keep (scenarios and test sets).
2. Click "Test new version", then "Start test". It first pre-runs each functional test of the run once to see that the tests really reach your agent, and the real run starts when the pre-run passes. At 1× with the normal condition only, the real run is what the pre-run would run, so there's no separate pre-run: the run itself is the pre-run. Cancelling the pre-run also cancels the run waiting for it. To start without a pre-run, tick "Skip the pre-run and start the test" when you start. A retest (rerun an earlier run's tests and compare) uses the default pass criteria until you set your own, like a normal run.
3. The round becomes an experiment with its release gate; its "Full report" shows the verdict, the evidence for each problem and the fix notes. Sometimes only the judge model can tell whether the agent said it was done, and the judge model couldn't be used (review.yaml names a model the platform doesn't provide and the test runner has no key for it, or every call failed). Those tests are then "Couldn't be tested", and the verdict is never a pass.

When a pre-run fails because the entry program errored out, the pre-run's "Details" give what the test runner reported, as it was (the last lines on stderr, or why the program never started, such as spawn .venv/bin/python ENOENT), keys removed; a dev AI reading the pre-run gets it too.

You can also run tests from the command line, for example in CI:

reviewctl review --wait                           # exit code 0 passed, 1 blocked, 2 incomplete, 3 error
reviewctl retest --run <baseline run id> --wait   # retest a new version against an earlier run and compare
reviewctl login --api https://<platform address>  # once: sign in with a device code
reviewctl push --run <run id>                      # upload a local run to the platform

See Command line for the options.

What enters our cloud

Only run bundles (evidence, redacted traces and logs) and your test files enter our cloud. Source code, test data and credentials stay on your machine, with one exception. With reviewctl connect --ai, and only once you agree, a selection of source files picked by the rules above goes to the platform's model, with anything that looks like a key removed, and is kept until that "Let AI finish and check" ends. The error lines of a check's log go the same way, keys removed first. Run bundles record environment variable names, never their values. With a local test runner, the judge model is the platform's by default: the replies to judge and the tests' points go through the platform's judge proxy to the judge model; with the judge model's key on that machine, your machine calls it with your own key instead, not through the platform. A run bundle may unpack to at most 512 MB, 10,000 files and 64 MB per file; a bigger one is refused and nothing of it is kept.