Using the platform from a coding AI

The coding AI on a developer's machine (Claude Code, Cursor, Codex and the like) can connect straight to a project and go through the whole loop with MCP tools: the test connection, writing tests, review, pre-runs, formal tests, reports, and testing again after a fix. Nobody copies text between the web page and the coding AI. How far it may go is up to the permissions you tick for the connection on the web.

The platform calls a coding AI connected this way the "dev AI". It is your own coding tool on its own model account; the platform never runs a coding AI on your code.

Make a connection

1. Open the project's "Settings → Dev AI".
2. Under "It may:", tick what this connection may do (see the table below).
3. With "Start formal tests" ticked, fill in "Spend at most $" beside it, or tick "No daily limit" (see the daily cap). On a project on a local test runner the default is $5 a day; on a repository or image project (a cloud sandbox) it is no daily limit.
4. Under "How the AI connects:", choose "reviewctl on the dev machine" or "MCP over HTTP (no reviewctl needed)".
5. Under "Language the AI gets:", choose 中文 or English (see Language).
6. Click "Get connect command".

Only a person in the browser can make a connection or change its permissions. A connection acts as the person who made it and lasts 30 days; it stops working if that person leaves the organization.

Permissions

Permission	Default	Ticked, it may
Write test drafts and upload files	On	Hand in test drafts, upload the test files they're missing, and have the platform's AI review the drafts again
Change the test connection files (.reviewctl)	On	Save the test connection files the platform holds, checked as when a person saves them under "Targets"
Pre-run	On	Have the platform pre-run once on the test runner or in a sandbox
Start formal tests	On: up to $5 a day on a local test runner, no daily limit in a cloud sandbox	Start formal tests, held to the daily cap
Confirm tests	Off	Confirm drafts, checked as when a person confirms them on the web. Off: the platform's AI still reviews the drafts, and the rest wait for a person on the web
Have the platform's AI finish the connection	On	Start "Let AI finish and check"
Change the pass criteria	Off	Save the project's pass criteria

Reading needs no permission: a connection can read tests, jobs, reports, diagnoses and the test connection files. It can cancel only the formal tests and pre-runs it started itself.

A step not ticked isn't done, and the coding AI is told: "This connection isn't allowed to "…", so nothing was done. Ask a developer to tick it under "Edit permissions" on this connection in "Settings → Dev AI" on the web". The refused attempt is listed on the connection's row too.

Each connection has a row in "Settings → Dev AI": the machine's name for one connected with reviewctl, "Remote MCP" for one over HTTP, then what it may do, what it has used today and what it did last. "Edit permissions" opens the same boxes; a change applies from its next call and goes into the audit log. After 5 failed submissions in a row, or 5 pre-runs in a row that don't pass, the connection pauses until someone clicks "Resume". While it's paused, the coding AI's get_status lists the problems of the last submission that failed (or why the last pre-run didn't pass) and links this page. "Disconnect" closes it (it asks first).

A connection made before this update may write drafts, upload files and pre-run, nothing more. To let it do more, click "Edit permissions" on its row.

Language

Each connection has its own language, Chinese or English: "Language the AI gets:" when it's made, the page's language unless you pick the other. Its row says which ("The AI gets English"), and "Edit permissions" changes it, from the connection's next call, written in the audit log like a permission change. A connection made before languages gets Chinese.

What follows the connection's language is everything the platform says to its AI: the server's instructions and each tool's title, description and arguments; every result and next step; refusals and errors; the prompt from get_prompt (in English it asks for the tests' own text in English too); pre-run, job and run results; get_report, get_diagnosis and get_fixes; and the sentence the page gives you to send it. What a test, a person or the agent under test wrote (test titles, the agent's replies) stays as it was written. The platform AI's diagnosis writes each fix note in both languages, and an English connection (or the web in English) reads the English one, with the condition in English too ("under normal conditions"); where the platform's own rules reword a fix note (the lead that says to change only your own code, its own note in a fix's place, a file's name put right), they reword the English too. A diagnosis made before this, or one whose AI gave no English even when asked, stays in Chinese. A pre-run's result in get_job and wait_job is in the connection's language too. The reason a draft waits for a person ("The revision from the platform AI's diagnosis removes or loosens …", with the AI's own reason in it whole) and the reasons in a report's pass criteria follow the connection's language as well, and such a draft is listed as one a person confirms (needs_person in list_tests). A security check's name in get_diagnosis is its English one ("System prompt extraction"), as in get_fixes. A test written in English has the platform's own words in its rules file in English too: the header above it, who confirmed it when automatic approval did ("AI auto-approval (on by default)"; the web says it in its reader's language), and the judge's instructions built from its goal. A test saved before keeps its judge's instructions until it changes. A default security set written in English has the platform's own words in it in English too: the header, the checks' names, what the judge reads the agent does and who is signed in, and the examples in the judge's instructions; a set written before stays as it was, and an English connection reads the kinds of attack by their English names anyway ("question r17 (System prompt extraction)" in get_report). The notes connect --ai adds to the starter tests it writes for an English agent ("Assumptions made while drafting") are in English too, and so are the platform's own checks noted on an English test (a test file still missing, a rule whose arguments the test runner didn't report). submit_answer says what each test runs and checks (tests) in the connection's language, and get_experiment_rows says why a score is undecided in it. What reviewctl prints in your terminal follows REVIEWCTL_LANG or the system's language, as before.

How the coding AI's side learns it: over MCP over HTTP, from the token's connection on each call. With reviewctl dev mcp, reviewctl dev connect saves the language in the connection's file under ~/.reviewctl/dev-links/; every answer from the platform says the connection's language as it is now, and reviewctl dev mcp says that answer's own words in it and writes it back into that file (reviewctl dev status writes it back too), so after a change on the web the very next answer is all in the new language. The tools' descriptions follow from the next listing; the server's instructions are read when it starts, so restart the MCP server (or the coding AI's session) to have them in the new language.

Connect with reviewctl (stdio)

The dev machine needs reviewctl first, see Command line: install.

The page gives a sentence to copy to the coding AI. It runs, in the repository:

reviewctl dev connect <code> --api https://<platform address> --register <claude|cursor|codex> --yes

The pairing code works once, within 10 minutes. "Get a new code" voids the one on the page at once: its row goes, and connecting with it is refused, saying it was voided. The command connects to the platform, saves the connection on this machine under ~/.reviewctl/dev-links/, one per repository (one machine can be connected to several projects at once, and each command uses the connection of the repository it runs in), and registers the MCP server reviewctl-dev (that is, reviewctl dev mcp) with the coding tool:

--register	How it registers
claude	claude mcp add when the claude command is on this machine; otherwise the repository's .mcp.json
cursor	The repository's .cursor/mcp.json
codex	codex mcp add when the codex command is on this machine; otherwise it only prints the lines to add to ~/.codex/config.toml, and never edits config in your home folder
none	Nothing

An identical reviewctl-dev already there is left alone. A different one is replaced only if you answer y at a terminal; elsewhere (or with --yes) it's kept, and the command says how to change it. A config file it can't read is never touched. Without --register, it asks at a terminal which tool; elsewhere it only prints how to register by hand.

Once registered, open the coding tool in this repository again to get the tools. Until then, the coding AI can use the same steps as reviewctl dev commands (see Command line).

If --register didn't work, register by hand:

claude mcp add reviewctl-dev -- reviewctl dev mcp     # Claude Code
codex mcp add reviewctl-dev -- reviewctl dev mcp      # Codex

Cursor: in the repository's .cursor/mcp.json, write

{ "mcpServers": { "reviewctl-dev": { "command": "reviewctl", "args": ["dev", "mcp"] } } }

To disconnect: run reviewctl dev disconnect in the repository. Its connection is closed on the platform and the one saved on this machine for it is deleted; other repositories' connections stay. The ~/.reviewctl/dev-link.json an older reviewctl saved still works: a repository with no connection of its own uses it, and disconnecting in such a repository ends it. Remove the registered MCP server yourself: claude mcp remove reviewctl-dev (Codex: codex mcp remove reviewctl-dev), or delete reviewctl-dev from .mcp.json or .cursor/mcp.json.

Start the local test runner with it

For a project on a local test runner, pre-runs and formal tests run on the test runner, which has to be online (get_status says whether it is, and what to do if not). When the dev machine is the test runner, use --runner:

reviewctl dev connect <code> --api https://<platform address> --register claude --runner

This registers reviewctl dev mcp --runner. Once the MCP server is up, it starts this project's test runner in the background with reviewctl runner start, and stops it when the server exits. The log is ~/.reviewctl/logs/runner-<project id>.log. For it to start, this machine must already be registered as the project's test runner (see Local test runner), the project in review.yaml must match, and no test runner may be online on the platform already. It uses review.yaml in the current folder, else .reviewctl/review.yaml, or the one --config names. get_status says whether it started, and why not.

For a project on a local test runner, the sentence the page gives and the commands to register by hand already have --runner; run without it, reviewctl dev connect says so.

Connect with MCP over HTTP

To skip installing reviewctl on the dev machine, choose "MCP over HTTP (no reviewctl needed)". The connection is made as soon as you click "Get connect command", with no pairing code. Register it in the coding tool with the lines the page gives for it. Claude Code (run in the repository):

claude mcp add --transport http preworld https://<platform address>/v1/dev/mcp --header "Authorization: Bearer <token>"

Cursor and other tools that read mcp.json:

{ "mcpServers": { "preworld": { "url": "https://<platform address>/v1/dev/mcp", "headers": { "Authorization": "Bearer <token>" } } } }

Codex: see Codex below. Other coding tools take the same address and header the way they register a remote MCP server.

The token starts with dl_ and is shown only this once: once you leave or refresh the page it's gone, and if you lose it, disconnect the connection and make another. Don't commit it to the repository. A call with no token, or with the token of a connection that was disconnected or has expired, is refused with 401 saying which, and that the way back is a new connection's token in the coding tool (PREWORLD_TOKEN, or the Authorization header), not reviewctl dev connect. The 401 is in the connection's language (a disconnected or expired one still has it); for a token the platform never made, in the language the request's Accept-Language asks for, and in both when it asks for neither. Once it's registered, tell the coding AI the sentence on the page (in Chinese, which your AI reads fine): "用 preworld 的 MCP 工具连上测试平台，从 get_status 开始，按每一步返回的「下一步」做，最后告诉我结果。" (Connect to the test platform with preworld's MCP tools, start from get_status, follow the next step each one returns, and tell me the result.)

Until the coding AI first calls the platform with that token, the connection's row reads "Waiting to connect"; "Connected" only once it has.

MCP over HTTP has the same tools as reviewctl dev mcp except upload_file: the platform can't read files on your machine. Upload the test files drafts are missing on the web, or use reviewctl; the prompt and the next steps over HTTP say so, and never send the AI to upload_file.

On a local test runner's project, "no reviewctl needed" is about the dev AI itself. Such a project's pre-runs and formal tests run on the test runner, which still needs reviewctl installed and reviewctl runner start running (see Local test runner); with the test runner offline, the coding AI stops at "start the test runner first" as soon as it connects. Over MCP over HTTP the platform doesn't know which machine the coding AI is on, so get_status says to ask whoever runs your team's test runner to start it on that machine, and that a pairing code for a new one is made with "Create pairing code" under "Targets" → "Local test runner".

On a project whose connection sees replies only (attach.mode: endpoint_only in review.yaml), the test runner reports no tools. A scenario still writes one in tools: the name of the function or entry the agent replies to the user with, which the platform doesn't check calls against. Such a scenario writes no must_call, never, fault, attack or data_checks, and says with goal_text and reply_must_say what the reply must contain.

Codex

The Codex CLI, its IDE extension and the ChatGPT desktop app all read ~/.codex/config.toml. The page gives the lines to add there. They read the token from the environment variable PREWORLD_TOKEN, so the file holds no token:

[mcp_servers.preworld]
url = "https://<platform address>/v1/dev/mcp"
bearer_token_env_var = "PREWORLD_TOKEN"

codex mcp add preworld --url https://<platform address>/v1/dev/mcp --bearer-token-env-var PREWORLD_TOKEN writes the same lines. Then set the token where Codex starts, and start Codex again:

export PREWORLD_TOKEN=<token>

To keep it, put that line in your shell profile (~/.zshrc, ~/.bashrc), never in a file in the repository.

The ChatGPT desktop app doesn't read shell profiles. On macOS, either run launchctl setenv PREWORLD_TOKEN <token> and quit and reopen the app (it lasts until you log out), or keep the token in the Keychain and have Codex read it each time with http_headers_helper, a command that prints the request headers as a JSON object:

security add-generic-password -s preworld-review -a "$USER" -w    # asks for the token

#!/bin/sh
# ~/.codex/preworld-headers.sh (chmod +x)
token=$(security find-generic-password -s preworld-review -w) || exit 1
printf '{"Authorization":"Bearer %s"}\n' "$token"

[mcp_servers.preworld]
url = "https://<platform address>/v1/dev/mcp"
http_headers_helper = "/Users/<you>/.codex/preworld-headers.sh"

The Preworld Review plugin for Codex brings this MCP server for https://review.nice2.io, reading the token from PREWORLD_TOKEN as above, and a skill that walks Codex through the loop: what to check first, how to read the report, diagnosis and fixes against the code, what it must never do (loosen the pass criteria, confirm tests without the permission, commit the token, run any of the agent's code on your machine, log into the test runner), and what to tell you at the end. It isn't in a public plugin directory yet; with a copy of the repository it's in (a Codex marketplace named preworld):

codex plugin marketplace add <path to the copy>
codex plugin add preworld-review@preworld

After that, ask Codex "Test this agent with Preworld Review". A [mcp_servers.preworld] in config.toml takes the place of the plugin's server: that is how to point the plugin at another platform address, or use the Keychain helper.

Codex without a person at the keyboard (codex exec), as in CI: its approval policy is never by default, so every preworld tool that isn't read-only is refused ("MCP tool call requires approval, but approval policy is never"). --approve-for-me routes approvals through Codex's automatic review, but can't be combined with -s (--sandbox). With a sandbox mode of your own, set the approval policy and the reviewer instead:

codex exec -s workspace-write -c approval_policy="on-request" -c approvals_reviewer="auto_review" "Test this agent with Preworld Review" < /dev/null

Started in the background, codex exec waits for standard input ("Reading additional input from stdin...") unless it's given < /dev/null.

Test after every change (AGENTS.md)

Codex, and other coding tools that read AGENTS.md, follow what it says in the repository. To have the coding AI test the agent before it calls a change done, paste this into the repository's AGENTS.md:

## Testing the agent

This repository's AI agent is tested with Preworld Review, through the `preworld` MCP tools.

- After changing the agent (its code, prompts, tools or model settings), run the Preworld Review loop before saying the change is done: start from `get_status`, pre-run with `try_run`, start a formal test with `run_tests` and wait for it with `wait_job`, then read `get_report`, `get_diagnosis` and `get_fixes`.
- Check each finding against the code before changing anything. Never loosen the pass criteria or change a test's expectations to make it pass.
- Before a formal test, say what it will cost; stop and tell me if the connection is paused, out of its daily cap, or not allowed a step.
- At the end, tell me the verdict, the report link, and what you changed.

The loop and the tools

There are 40 tools (39 over MCP over HTTP, which lacks upload_file). Each one's result says what to do next, and a step the connection isn't allowed says who should switch it on, and where.

Each tool has a title and MCP annotations, so a coding tool can tell what needs your approval: the reads (get_status, list_tests, get_report, wait_job and the like, and the removed plan steps, which only say so) are readOnlyHint (get_tasks isn't one: it takes the open tasks); one that may change or end what is there (submit_answer, finish_task, put_wiring_file, start_connect_assist, review_drafts, confirm_test, cancel_job, set_gate) is destructiveHint; try_run, run_tests and start_connect_assist run the agent under test (openWorldHint).

Which code ran. try_run, get_setup, get_job, wait_job and get_report say which code each pre-run and formal test ran, in their text and in their structured content (code). On a local test runner: the commit, whether there were uncommitted changes, and each changed or new file with the first 12 characters of its sha256, to compare with your copy (shasum -a 256 <file>); at most 30 files are listed. Only names and hashes leave the test runner, never contents, and a file named like a secret (.env, a key) is listed without its hash. The test runner says this when review.yaml keeps agent.version.git on (the default) and its folder is in a Git repository. In a cloud sandbox: the commit, plus the platform's test connection files. To check what the test runner ran, read these; don't log into the test runner.

A pre-run that just passed is used again. run_tests pre-runs first unless the latest pre-run passed for the same tests and code: on a local test runner, the same test runner, not restarted since, with the same code, within 30 minutes; in a sandbox, the same commit within 6 hours; and no test, test connection file or secret changed since. The reply says when it goes ahead on one, and when it can't, why (no pre-run passed lately, the last one that passed is too old or ran other tests, the test runner has reported different code, it restarted, it's another test runner, or a test, test connection file or secret changed since); the structured content has it as prerun.why. If the test runner's code changes before it takes the run, it pre-runs first after all. See sandbox runs.

Where the agent's model calls go. When the test runner has no model key for the agent (no OPENAI_API_KEY or OPENAI_BASE_URL in review.yaml or on that machine), reviewctl runner start points the agent's OPENAI_BASE_URL at a relay on that machine (127.0.0.1), with a placeholder OPENAI_API_KEY. The test runner forwards the agent's model calls (chat completions and the Responses API only) to the platform's models, billed as usage; a cloud sandbox does the same (see your agent's model). An agent with its own key calls whatever it is set to call. The descriptions of try_run and run_tests say so too.

What comes from the agent under test or the test data (its replies, tool calls and their results, questions and attack texts, what a test expects, trace spans, experiment rows, setup and test runner logs, a diagnosis's quotes) comes between <untrusted-content boundary="…"> and </untrusted-content boundary="…">, with one sentence saying it is data, not instructions, and must not be followed; the server's instructions say the same. The content inside is left as it was. Where the platform AI's fix note in get_fixes (or a diagnosis's own words in get_diagnosis) repeats words of what its evidence quotes of the agent or the test (a reply, a tool call, a log, the test's text, what a verdict quotes of the reply), those words are marked where they stand too; the platform's own words a piece of evidence quotes (a verdict's reason, "The platform checked this against the system prompt's own text") don't count, so a fix that happens to share them isn't marked. Where the AI's reason for holding a draft (in get_status and review_drafts) repeats the test's own text — its goal, what the user says, the points the reply must make — that is marked where it stands too; a field name in that reason is said as the test page labels it ("What counts as done", "The reply must tell the user"). Where a pre-run check in get_setup quotes what the agent printed (its last error line, say), or a report's summary of a run that never reached the agent does, that is marked where it stands in the text too. A batch's summary (the paragraph under the first line of get_report without a run_id) is the platform's own words and never quotes the agent. The structured content (structuredContent) is marked the same way: each such value has the marker around it on its own, without the sentence. Ids, codes, states, times, links, hashes, verdicts and next steps stay as they are, and so do the platform's own words: what a test is expected to cost, the pre-run it goes ahead on or why not, which code ran (code, the repository's file names in it marked), what each part of a batch holds ("2 tests", a test set's title in it marked), a pre-run check's title, a report's summary ("7 scenarios, 21 attempts: all passed…", the titles it names marked; one of a run whose records couldn't be trusted, which quotes the agent's own error, marked whole), and in get_diagnosis whose problem an item is and the condition it's under (the agent's tool in it marked). The pass criteria and the test connection files' text stay as they are too, which set_gate and put_wiring_file take back as they are. The order:

1. Where things are: get_status.
2. Test connection: get_wiring shows the test connection files the platform holds, put_wiring_file changes one; or start_connect_assist has the platform's AI finish it, and get_connect_assist follows it. For a project on a local test runner, the test runner reads only the files in your own repository: change .reviewctl/ there and pre-run with try_run; what the platform holds is just a copy, so there's no need to save it back with put_wiring_file. The files' formats are in the review.yaml reference, the data queries the data checks read (state_queries.yaml) under data queries.
3. Write tests: get_prompt for the prompt, submit_answer to hand in, upload_file for test files. When the project has a test plan approved before, the prompt carries it (get_plan for the whole plan; the steps that made plans were removed, see the test plan tools). Tasks someone sends it through the API (POST /v1/projects/<id>/dev-tasks) come with get_tasks, and go back with finish_task; the web sends none any more.
4. Review: review_drafts has the platform's AI review, get_test shows the rules to check against the code, confirm_test confirms (with the "Confirm tests" permission). A draft a rule holds for a person (one whose fix from the platform AI's diagnosis takes out or loosens a check, say) is never read by the AI: review_drafts and get_status give its reasons and say a person confirms it on the web, and it isn't counted among the drafts "the platform's AI is reviewing", nor in the N of review_drafts' first line, "Asked the platform's AI to review the drafts again (N)". Its started is true only when a draft went to the platform's AI; auto_approve says whether the project's AI review is on.
5. Pre-run: try_run; when it doesn't pass, get_setup shows why. While the pre-run is still going, get_setup and try_run say where it runs (the test runner or a cloud sandbox), in their first line and their next step, how long it has run, and how long this project's last pre-runs there took; in a sandbox with none yet, the platform's estimate (starting the sandbox, then each test once).
6. Formal test: run_tests starts it, wait_job waits for it.
7. Results: get_report, get_diagnosis, get_fixes. Without a run_id, all three are about the latest formal test: the runs launched together (the functional tests and each test set) are read as one. A pre-run's run_id works with get_report only: a pre-run gets no AI diagnosis and no fix notes, so get_diagnosis and get_fixes say it's a pre-run, and its problems and fixes are in get_setup. A formal test still running or being judged says which job to wait for with wait_job. The diagnosis's fix notes say how to check a fix on the platform, never by running the agent under test on your machine: a dev AI pre-runs with try_run, then starts a formal test with run_tests; a person pre-runs, then starts a formal test on the web. A fix note the diagnosing model wrote otherwise is sent back to it once to be written again.
8. Check against the code, fix, and go back to step 5.

Tool	Does	Permission	Command
get_status	Drafts, test files still missing, whether the test runner is online, what this connection may do, what next, and the link to these docs on the platform's own host (the server's instructions give it too)		reviewctl dev status
get_prompt	The prompt the platform wrote for this repository: which tests, files and todos to write, in what format		reviewctl dev prompt
submit_answer	Hands in the JSON; lists the problems if it doesn't pass, and the connection pauses after 5 in a row	Write test drafts and upload files	reviewctl dev submit <file>
upload_file	Uploads a test file a draft is missing (only in reviewctl dev mcp)	Write test drafts and upload files	reviewctl dev upload <path>
get_tasks, finish_task	Tasks sent to it through the API; done, or handed back		reviewctl dev tasks, reviewctl dev done <id>
get_wiring	The test connection files the platform holds: text, hash, who saved each last		reviewctl dev wiring
put_wiring_file	Saves one file under .reviewctl/, checked by the web's save rules	Change the test connection files	reviewctl dev push <file>
start_connect_assist, get_connect_assist	Has the platform's AI finish the connection and check it; how far it got	Have the platform's AI finish the connection (to start it)	reviewctl dev assist, reviewctl dev assist --status
list_tests	Tests and drafts: id, conditions, AI review, last result, and the pass criteria. A scenario with both a normal and an attack condition is listed once (groups has functional and security), so the count is get_status's		reviewctl dev tests
get_test	One test's rules (YAML)		reviewctl dev test <id>
review_drafts	Has the platform's AI review the drafts again; says which wait for a person, and where	Write test drafts and upload files	reviewctl dev review
confirm_test	Confirms a draft	Confirm tests	reviewctl dev confirm <id>
try_run	Pre-runs once (one attempt, no report) and waits; the connection pauses after 5 in a row that don't pass	Pre-run	reviewctl dev retry
get_setup	The latest pre-run: can the tests reach the agent, and which code the test runner ran; if not, the problems and fixes		reviewctl dev setup
run_tests	Starts a formal test (as "Start test": a pre-run first, unless one that just passed ran the same code and tests; then the test). The scenarios and each test set are a job each, queued together: returns every job id, what it's expected to cost and the most it may cost. conditions with faults alone (no normal condition, no attack) needs pass criteria that name them: while nobody has set any, the default ones judge only each test's normal condition and attacks, so such a run is refused, and the reply says why and that leaving conditions out runs every condition	Start formal tests	reviewctl dev run
get_job, wait_job	Where a job is; wait until it ends (it returns once that job ends, naming the jobs launched with it that are still running). Without a job_id, the last formal test this connection started; with none yet, it says so and names the connection's own pre-run, whose result is in get_setup (wait for it with its job_id); a pre-run started after that test and still going is named beside it, and so is a part launched with it that stopped at a spending limit (one that stopped before it began is said not to have started, never to have finished). Each attempt's state is in words ("passed", "undecided", "not run"); the structured content keeps the codes		reviewctl dev job, reviewctl dev job --wait
cancel_job	Cancels a job this connection started		reviewctl dev cancel <id>
list_runs	Latest formal tests: verdict, problem count, report link		reviewctl dev runs
get_report	A formal test's conclusion: the first screen's words, which code it ran, whether the platform swapped the model, what failed or couldn't be decided and why (a test set question by question). The first line counts tests: one that couldn't be tested under several conditions counts once, with the conditions after it ("9 tests (under 15 conditions) couldn't be tested"); why each couldn't be decided is in the platform's words, without internal codes		reviewctl dev report
get_diagnosis	The platform AI's diagnosis: whether the cause is the agent, the test connection or the test itself		reviewctl dev diagnosis
get_fixes	A run's fix notes (the same as "Export fix notes" on an experiment row gives)		reviewctl dev fixes
set_gate	Saves the project's pass criteria	Change the pass criteria	reviewctl dev gate <file>

The test plan tools

The test plan step was removed (see test plans): the platform no longer makes plans, and the web has no page to approve one. These tools are in MCP only; there are no reviewctl dev commands for them.

Tool	Does	Permission
get_plan	The test plan approved for the project before: its version, the plan in full, and what next (get_prompt for the prompt that carries it). With no plan approved, it says the step was removed and how to write tests instead
start_plan, answer_plan, next_plan_round, write_plan, edit_plan	Removed: they start and change nothing, and only answer the error plan_removed, saying the step was removed and to write tests with get_prompt and submit_answer instead

The experiment, log and release gate tools

The coding AI can read the project's experiments, its live logs (traces) and its release gate too, to find causes against the code; and it can draft dataset rows and scorers. All on this connection's own project only: another project's experiments, traces and datasets aren't there, even in the same organization. Reading needs no permission; drafting needs "Write test drafts and upload files", and what it drafts waits for a person to check on the web. It can't change the release gate, run an experiment or score anything (nothing here spends). These tools are in MCP only.

Tool	Does	Permission
list_experiments	The project's experiments, newest first: source, status, whether the release gate passed, each scorer's mean, cost
get_experiment	One experiment and its release gate: each rule, whether it held, the actual value, the rows that broke it
get_experiment_rows	An experiment's rows: input, expected, output, each scorer's value and reason; against a baseline, the change. The filter language picks the failing ones, e.g. score.builtin:task_done < 0.5
list_traces	The project's traces, newest first: the start of input and output, status, online scores, models; filter e.g. status = error, issue = <issue id>
get_trace	Every span of one trace in call order and depth: tool and model calls' input and output, where it failed, online scores with reasons
list_datasets	The project's datasets: its own, and those standing for its tests (scenarios, test sets)
add_rows	Adds rows to one of the project's own datasets, marked as not looked at yet, for a person to keep or delete	Write test drafts and upload files
create_scorer_draft	Creates a new project scorer, marked as drafted by the dev AI; new ones only, used nowhere until a person tries it under Scorers and adds it	Write test drafts and upload files
get_gate	The project's release gate rules and the latest finished experiment's result on them; read only: a person changes it under "Settings → Release gate" on the web

For a project on a local test runner, try_run and run_tests both need the test runner online. When it isn't, the platform starts nothing, and says to run reviewctl runner start on that machine, or to use --runner.

The daily cap

A connection with "Start formal tests" ticked spends at most "Spend at most $" a day (UTC): $5 by default on a project on a local test runner, $1,000 at most.

No daily limit

Tick "No daily limit" beside "Spend at most $", when you make the connection or under "Edit permissions" on its row, and the connection has no daily cap of its own. It is the default for a new connection on a repository or image project (a cloud sandbox). Connections made before keep the cap they have.

• The organization's own limits are then the only brake, as for a test started on the web: the period's credit and the balance, the spending cap, the prepaid balance, and the organization policy.
• run_tests checks the test against what the organization has left, by the same rule as below: in a sandbox, its sandboxes at the organization's longest run and what its models are expected to cost. If it doesn't fit, nothing starts, and the reply says what it would need, what the organization has left, and that an owner tops up or raises the spending cap on the billing page.
• While it runs, the platform's models stop where the organization's credit runs out, as on the web; what is left untested isn't charged. An organization with no spending cap at all has only each run's own limits.
• get_status says "Formal tests: this connection has no daily limit", the run_tests reply says so beside what was used today, and the connection's row reads "about $X used today / no daily limit".
• The audit log records the change with the rest of the connection's permissions. If someone sets a cap while a test runs and the test stops there, the pages say it had no daily limit when the test started.

The rest of this section is about a connection with a daily cap.

• Each run_tests first works out what the test costs, by the same rule as the start page, and holds that from the connection's cap for today as it starts. If it doesn't fit, nothing starts, and the reply says the estimate, the ceiling, how much was used today and what's left, and what to do (test less, test tomorrow, or have someone raise the daily cap).
• A project on a local test runner has no sandbox cost: what counts is the platform's judge model, and the agent's model when the platform's model answers the agent. A test there holds what it's expected to cost: "Expected to cost about $X (usually $low–$high, from this project's last few runs)", as the start page says it, the pre-run included. While it runs, the platform's models stop at what is left of the connection's cap for today, as at a run's own limit: the judge model judges no more, and the replies after that can't be judged (not tested); the platform model answering the agent answers no more, and the run stops after the attempt in progress with a partial report. So the daily cap is still a cap on what is actually spent. The reply still says "up to about $…": each run's limits on the judge model and the agent's model, not lowered to fit the daily cap.
• A security test that hasn't run in this project yet is estimated with the agent's cost from this project's functional tests and the judge model's from the organization's latest security tests (judging a security question costs more than a functional test).
• With nothing to go by yet (no test has finished on a test runner in this project or organization), there's no estimate: the test holds what is left of today as it starts, and stops there the same way while it runs.
• When the agent's model counts: reviewctl on the test runner passes the agent's calls to the platform's models only when that machine has no OPENAI_API_KEY of its own (and no OPENAI_BASE_URL), and tells the platform so. When none of the project's test runners does, the agent's model isn't in the ceiling; an agent that calls no model (a script, say) cost nothing for its model in the runs so far, so the estimate leaves it out too.
• In a project that runs in a cloud sandbox, the judge model and the platform model answering the agent also hold what they're expected to cost and stop at what is left of today, as on a test runner. Only the sandbox, billed by the second, doesn't stop at the daily cap while it runs, so each sandbox (one for each run, and one for a pre-run if it has one) is held besides for the organization's longest run. Two tests run once each and expected to cost about $0.08, plus up to two standard sandboxes at 2 hours each, fit the default $5 a day (before, a test held its "up to about $…", over $13, and was always refused). While it runs, the models stop at the daily cap less what other tests hold and these sandboxes at their longest run, so the daily cap holds even if the sandboxes run that long. When it doesn't fit, the reply says what it would hold and the least the daily cap would have to be.
• The connection's row on the web lists each formal test it started: on a test runner what it's expected to cost and its ceiling; in a sandbox also what it held as it started ("expected to cost about $0.08, holding $0.43 as it starts (up to about $13.35)"). A formal test that stopped at the daily cap while it ran gets a line of its own as it ends, with how many attempts that left untested (or, where only the judge stopped, unjudged) and what the cap was when the test started if someone lowered it. A refusal on the row reads in the page's language, whichever language the connection reads.
• What's used today counts the formal tests this connection started today: one still running, which may still spend, counts the larger of what it holds and what it was billed so far (in a sandbox, each sandbox at its longest run until it is deleted and billed); one that has ended counts what it was billed, its pre-run with it. get_status says it too, with what it counts. While a formal test runs, get_status (running_today) and the connection's row also say how much of it is that test, what it held as it started, and what it was billed so far. With the daily cap unchanged (or raised), a running test doesn't stop at what it holds, only once what it has billed (each sandbox counted at its longest run) and today's other formal tests reach the cap, so what's used today can read above the cap while the test goes on. If someone sets or lowers the daily cap after it started to below what's used today, it stops at its next platform model call (see the next item); running_today then has stopping, and both say the test is stopping. list_experiments gives each experiment's whole cost, the platform AI's diagnosis of its runs included (diagnosis_usd), which this cap doesn't count.
• When someone sets or lowers the daily cap under "Edit permissions" while a test runs, the platform's models check their next call against the new cap: the judge model and the platform model answering the agent alike, on a connection that had no daily limit too (before 5 November 2026, up to 30 seconds later there). Where the new cap is below what the test holds plus today's other formal tests (since 5 November 2026), the next call of the agent's model or of the judge is refused whatever the test has billed, and the test stops after the attempt in progress, stopped at this connection's daily cap: the row's "reached the daily cap" line, the report and the tools say what the cap was when the test started, or that it had no daily limit then. Before, it went on until what it had billed (each sandbox at its longest run) reached the new cap. Where the new cap isn't below that, and still leaves room (in a sandbox, after what was billed and each sandbox at its longest run), the test goes on until that is spent, then stops after the attempt in progress; what passes the cap is at most the calls in flight. Raising the cap only lets the test go on. A part launched with it that hasn't started yet (a test set after the scenarios, say) is checked the same way before its sandbox is started: with nothing left, no sandbox is started for it and it ends there, stopped at the cap, and get_job, wait_job and get_report say so.
• A pre-run on its own (try_run) and "Have the platform's AI finish the connection" are billed as usual and don't count toward this cap: a pre-run by sandbox time and model usage, the finishing by the assistant's daily allowance.
• A refusal at the daily cap links the settings page where it's raised ("Settings → Dev AI"). When a formal test runs out of the daily cap while it runs, wait_job, get_report, get_fixes and get_status link it too, and say what the cap was when the test started if someone lowered it while it ran. get_report counts what couldn't be tested as the web's first screen does, tests and attacks apart, and an attempt the cap stopped counts only as that, never also as "the platform's model didn't answer": the run's check "Did the platform's model answer" and the batch's summary say the run reached a spending limit (the platform's brake), and quote nothing the agent handed back then.
• Which limit stopped a formal test (this connection's daily cap, the organization's balance, or the run's own) is decided as it ends, with the cap as it was then, and kept on the job: raising (or lowering) the daily cap afterwards, as the pages suggest, changes nothing the round report, the report page, the progress page or get_report, wait_job, get_fixes and get_status say. One that ended before this is read from the connection's row, or from the cap as it was when it ended. An attempt a limit stopped before it was judged doesn't count toward the "the platform can't see the agent's tool calls" check either: such a run opens with the limit, never with fixing the connection.
• The organization's credit, balance and the limits in its policy still apply. When a tool is refused because the organization's credit and balance, its spending cap or its prepaid balance are used up, the reply says an owner of the organization has to fix it, and links the billing page (/org?tab=billing).

Next steps

• Command line: every reviewctl dev command.
• API and tokens: the dl_ token and /v1/dev/*.
• Test plans: how a plan approved before is used; making one was removed.