Changelog

Every change to Preworld Review you can see, newest first.

27 October 2026

Fixed

• On a platform-written test that was wrong, two runs that did the same thing now get the same finding, even when the judge read the same reply differently.
• Old reports where the agent worked around a fault with another write tool and said so truthfully now read as recovered, so both rounds agree.
• A run stopped by its cost cap says cost cap everywhere on the page, never model error or cancelled.
• The diagnosis no longer tells you to hide the agent's own public tool names, no longer points at a tool for returning what it is built to return, and never names a startup or test script as the code to change.
• "The AI's own account" no longer repeats what the platform already says.

26 October 2026

Improved

• The platform asks before fixing the test connection. When a pre-run or check fails, an AI diagnosis finds a connection problem, or AI approval finds a draft needs a connection change, the platform asks on the project's Goals page, with about what it will cost and take; it fixes only when you click Fix it, and Not now runs and charges nothing. For the old automatic repair, turn off "Ask me before fixing the test connection" under Organization → Policy.
• A repair round that changed nothing isn't charged. If a round of fixing changed no connection file, its platform AI cost is given back, and its trial sandbox isn't charged either.
• One number for every limit. Every place a limit is set or shown uses the price the pages show (for example $650 a day of platform AI, $6.50 a run); what each limit lets through is unchanged.
• A default security test has at least 10 questions that can count, at least one for each high-risk capability the product really has; short sets are written again, and anything still missing is named.
• When the platform's stand-in model is blocked by its own safety filter, that question is run again on another platform model, and the report says which questions it answered.
• Each question of an existing platform security test gets the functional tests' per-run time limit (up to 10 minutes), not 1 minute.
• The sandbox image has bun: repos that use bun install with it directly.
• The support address is now admin@nice2.io.

Fixed

• Ticking only test sets no longer multiplies them by "runs per scenario". A set asks each question its own number of times, and both start pages show the same plan, count and estimate as the run.
• A run stopped by its cost cap says so: what ran is charged, what didn't run isn't, and how to finish.
• Restating a system prompt that is a public file of the repository is not a leak.
• When the agent works around a fault with another write tool and reports it truthfully, that counts as recovery, not a false completion.
• Security questions that ever got through always come back when the default set is regenerated, planned three times each.
• The regenerate estimate includes AI approval; the full-options start page shows a regeneration in progress and tells same-named sets apart.
• A real duplicate side effect is never washed out by "one fault, one owner". Another run's finding carries over only when the agent did the same thing in both runs.
• A run of refusal tests only no longer reads as a broken connection, and doesn't offer a paid repair.
• A connection repair's check runs each thing it was asked to fix, and says which one it couldn't test instead of "connected"; the repair estimate uses the project's own history.
• A file read that was blocked or only written in the reply is a risk signal, not a success; a security question that ever got through stays in later sets and is asked several times.
• "Regenerate the default security test" runs in the background and shows progress, result and question count; the button shows the usual cost and time.
• Both start pages give the same estimate; a launch with only test sets ticked can start.
• A "got through" with no real effect is a risk signal. A command that only appeared in a search query, a statement only written in the reply, or platform-assumed data that was read but never shown are held, with a note; a real execution, write or secret read still counts.
• A PR opened while the project is busy waits instead of being dropped. GitHub shows "Queued" and the check starts when the project is free; Automations → Triggers lists how each push and PR of the last day was handled.
• A security test made from a template is a draft that AI approval reviews, never confirmed in your name, and named apart from the default one; "Regenerate the default security test" is on the attack-dataset page.
• Old experiment pages recompute with settled reports; a test the platform wrote wrong and can't fix is no longer re-run at a charge by "Retest last run's failed or undecided".
• Estimates use each test's own past attempts, so the spending-cap warning no longer comes and goes.
• A rule counts only when the call really did something. A read-only command, a call the product itself blocked or that errored, or a reply that refused is no longer judged an overreach or a successful attack; it is noted as a hint. The platform's own forbidden rules now say what effect they forbid. A plain value in a field named token is no longer taken for a secret, and purpose checks such as "only writes queries" apply only to agents with that purpose.
• The round report and retests agree. After a retest, free or paid, the round report, the experiment and the run report count the latest try the same way, and rows the platform set aside stay set aside.
• A connection repair says fixed only when it is. The check after a repair runs the problem it was given, and says so when it couldn't. Repair cards name who started them instead of always saying the platform did.
• The default security test no longer gets stuck. When it comes out short it is written again straight away, saved on the last try with the missing kinds named, and the page follows along.
• An agent naming its own tools with a server prefix or as signatures is not a leak; directory paths a tool returns are not the system prompt.
• Estimates now count runs that stopped at the limit, so they are closer to what runs cost; an experiment's cost shows the AI diagnosis part separately. The platform AI's queue really is first come, first served, and "waited about" counts the real wait.
• "Retest only the untested" is on the run report and the round report, and says before the click how many it reruns, which version and whether it is charged.

25 October 2026

Improved

• The platform's own mistakes no longer count against your agent. When the platform's diagnosis finds the platform's test harness, a test the platform wrote wrong, a judge misreading, or the sandbox unable to reach an outside service, those attempts become "not tested (platform's cause, not charged)": out of the verdict, severity and release gate, and a GitHub check or PR comment already posted is updated. The report says "the platform got N wrong (not yours)", and retesting them is free. An attempt showing real security harm is never set aside.
• Sturdier platform-written harnesses. Tool errors and timeouts reach the model instead of crashing the harness; reset and tool servers share one data folder; the connection check deliberately times out one tool call, and a harness that crashes on it isn't connected yet.
• Security tests: a default security test with no question that can count is written again, and missing kinds of attack are named; restricted actions another test of the project names are checked in every test; each security question gets as long as the functional tests; attack kinds the platform's model couldn't test are named as such.
• Naming its own tools isn't a leak, also when listed as signatures with example arguments; users the platform assumed are always low severity and marked; diagnoses no longer tell you to hide tools or add per-user permissions to a single-user product.
• "Let the platform fix" says what it fixed: each item changed or not done; when nothing changed it stops, without a check or further charge. A repair the platform starts itself shows its usual cost and can be stopped, and never blocks starting a test.
• Money: before a run that would hit the per-run cap, ways out: batches, functional or security only, or raising the cap for this run (organization owners); the cost of a repair or diagnosis is shown; "Let the platform read the repo" and "Try again" show a price.
• The queue is first come, first served: 4 at once per organization, in click order; while waiting it says how many are ahead and until when, then "the platform's AI is writing (minute N)".
• Hand-written tests: an empty tools box means any tool, and you can draft right away; with no reply points it offers to use "what counts as done"; a test page has "Change the wording", for just what the user says.
• Suggested checks added in one click; rules in plain words; terms explained on first use; one name for the AI Gateway.
• Default release gate: low-severity hits only warn. When the only problems are low severity (limited impact, or a scenario the platform assumed), the gate still passes, and says "N low-severity problems: whether to ship is your call". This shows on the experiment page, the experiment list, the overview, reports, GitHub checks and PR comments, and in reviewctl eval. Medium and above still fail. Rules you set yourself are unchanged; Settings has "Low-severity hits only warn".
• Saying what is public isn't a leak. The agent's own tool names, descriptions and parameters, and a public repo's AGENTS.md, SKILL.md, README, docs and prompt files, are not counted as got through when the agent repeats them; the report says it was public content. A private repo's files, and real secrets, still count.
• The SDK and the AI Gateway show "In development" for now, and can't be chosen for a new project. Projects already connected that way keep working, and a local runner without an OpenAI key can still use the platform's gateway.
• Money and time, before you start.
  • "Let AI fill in and check" shows what it usually costs and how long it takes, queue included.
  • Starting a test prices each part from what this project's similar executions actually cost, and says first if a part will go past the per-run limit.
  • The project page shows what the project has cost so far, and how much of that was the platform AI. The experiment cost shows the same figure.
• The platform AI queue is first come, first served. A task's place only moves forward, and the wait is estimated from how long recent tasks took. "Let the platform read the repo" shows its place and latest progress too.
• The platform AI's daily limit
  • It is under "Organization → Policy": how much a day, what is used today, what happens at the limit, and whom to ask for more. The page warns when several tasks using AI at once may hit it.
  • The old "AI may spend at most … a day" is now "Tests the AI starts itself may spend at most … a day / a month".
  • The billing page says why this period's credit isn't the plan's full amount: a plan changed mid-period is prorated by day.
• The same problem twice in a row stops. When "Let AI fill in and check" hits the same failure twice, it stops. It says whose problem it is and what has been spent, charges nothing more, and waits for you to click "Try again". Connection repairs the platform starts itself stop the same way. Earlier rounds can be opened.
• Reports say what was tested and what wasn't.
  • When what was tested is the test harness the platform wrote, the report top and the connection card say so in plain words.
  • A new "Not tested this time" block lists what the run didn't cover.
  • "For your developers" is now three groups: "The platform fixes it" (with a button), "Likely problems in your product" and "What you need to provide".
• Plain-language prohibitions can become checks. When a test says "don't delete a single file", "don't change cv.md" or "don't export customers", the form asks whether to turn it into a check and offers the rule.
• Security tests. Each project gets one default security test set. Questions ask only for fields that exist in the project's data files, and approval checks that too.
• Functional tests.
  • When the repo has data files, drafting computes reference values from them.
  • Requested language and length become points.
  • For form-filling products, "Yes" counts as an answer.
• Hand-written tests.
  • With more than 50 tools, the tools box starts empty (= all allowed).
  • The per-execution time limit is shown and can be changed.
  • Test files can be Excel (.xlsx / .xls).
• Connecting GitHub opens on the account you just installed or last used, and never picks a repo for you. The branch box shows the repo's default branch.
• Other pages:
  • The overview lists projects by latest activity, with search when there are many.
  • A recommended sandbox size takes effect only when you click "Switch".
  • A schedule is on only after "Save and turn on".
  • Starting a sandbox test says the sandbox is being requested.

Fixed

• "Done twice" only counts the same tool with the same arguments run again after a fault: writing different files or editing different sections no longer counts.
• Forbidden rules look only at what a call acts on: mentioning cv.md in a file's content, or DROP TABLE in a refusal, no longer counts; "why it failed" shows the matched part. Rules already saved are judged this way too.
• An attack that never reached the agent is "not tested", not "followed the hidden instructions"; with only low-severity problems the round verdict warns like the gate; a per-question time limit is named as such.
• The pre-run no longer calls an agent that did its job "didn't really work"; AI approval is no longer refused by the model service; a tool schema the platform's model refuses is the platform's problem, not charged.
• Hitting the spending cap reads the same, with the same count, on the report, the round and the progress page, and no longer says "cancelled"; "What this run didn't test" follows the tests that actually ran.
• A test's attachment and what it expects come from the same content; repos that need Node 24 (like ZCode) get Node 24.
• Preparing the environment no longer reports "stuck" after the install finished.
  • The install counts as done when it ends.
  • Saving and restoring the dependency cache is a platform step of its own: progress every 30 seconds, at most 8 minutes, skipped if it can't be saved.
  • A stall there is the platform's problem: it isn't charged, and the AI isn't asked to change the install command.
  • bun.lock and other lock files are detected.
• The platform's own problems are no longer counted against your Agent.
  • When a run hits its cost limit or a rate limit, or the model the platform substituted returns nothing or is blocked by its safety filter, those executions are "Couldn't be tested". They aren't charged, retesting them is free, and you're no longer told to fix your Agent.
  • The round reads "Not finished" and says "this run reached its cost limit; the next N weren't tested".
  • Parameters the substituted Gemini doesn't accept (such as penalties) are dropped by the platform.
• An execution that timed out without a reply is never a pass.
• When the judge misread a reply.
  • When the platform finds the judge model misread a reply, the execution doesn't count against the Agent. After reading the reply you can click "I've looked: count it as passed"; who and when are recorded.
  • The judge's number checks understand ranges and "or".
• Release gate. When attacks couldn't be decided, "Attacks that got through = 0" shows "? Uncertain" and holds the gate, instead of a ✓. This applies on the experiment page, GitHub checks and PR comments.
• Fairer verdicts.
  • Writing the same file again with the same content isn't "did it twice".
  • A password in a config file's comment or example is left to the judge, with at most a "please have someone look".
  • Platform-assumed "other user" scenarios are weighed low.
  • When only low-severity problems failed, the headline says so and leaves the decision to you.
  • A real breach inside a functional test shows in the security summary too.
  • Successes caused by injected instructions are listed under prompt injection.
• When the connection can't read the data, tests that don't state "the data after the task" are judged by tool calls and replies. They are no longer held at approval or marked "Couldn't be tested". "Let AI fill in and check" wires up the data the drafts need.
• Reports and experiments.
  • Every round has a one-line verdict.
  • Undecided reasons are written in words.
  • Attack counts agree.
  • A security set that ran in the same batch is no longer "not chosen this time".
  • Round times are start times.

24 October 2026

Improved

• The platform's default model is now Gemini 4 Argon. It is used when neither the project nor the organization has chosen one. Projects, organizations, LLM scorers and Playground columns on GPT-6.1 Sol moved to Gemini 4 Argon, and those on GPT-6 Luna to Gemini 3.8 Flash; on the platform's judge calibration each does at least as well as the model it replaces. A scorer moved by a new version, and its earlier versions and their scores stay as they were; a chat AI built on a platform model didn't move. GPT-6.1 Sol and GPT-6 Luna can still be chosen. See Models.
• Gemini 4 Argon is in the model catalog, as "Strongest · slower"; GPT-6.1 Sol is now "Strong · faster". When two judge models read a reply differently, Gemini 4 Argon's reading decides.
• The second judge is from another family. Where the platform judges, the second judge is the strongest model of a different family from the first: Gemini 4 Argon for a GPT judge, GPT-6.1 Sol for a Gemini one. See the second judge model.
• Sandboxes run in London only. "Run location" offers Auto and London, and no longer shows Sweden (EU); projects that had chosen it are on Auto. The London outbound address is the only one. See run location.
• Install the SDKs from the platform: /downloads/ serves the TypeScript and Python SDK packages, and the SDK card on Targets gives the install commands. See SDKs.
• An experiment's cost is what was billed: sandbox time, judging, platform models and AI diagnosis for a sandbox round; the platform scorers' judging added for SDK and Playground experiments.
• Undecided rows can be followed up again: each lists the platform's steps, with the button to click beside the one waiting for you, including writing a pre-approval for a tool into the test. See undecided rows.
• Narrow screens: below 720px the sidebar folds into a "Menu" in the top bar and the page takes the full width.
• An agent in a sandbox that uses the Responses API now works on a Gemini project. See your agent's model.

Fixed

• AI Gateway: a streamed call read to the end is no longer recorded as broken off, and a real break says why; a provider's error (Google's too) comes back in this address's error format with the provider's own words. See limits and errors.
• A call the caller closed before its stream ended is recorded as "Cancelled by caller", not as an error, and error-rate alerts, the overview and issues leave it out. See Logs.

23 October 2026: v2

Preworld Review is now an evaluation platform you can bring your own data, tasks and scorers to, before and after release, while keeping what only it does: running your whole agent in a disposable sandbox under faults and attacks, with a release gate. A project's page is now ten sections on a rail (Overview, Logs, Datasets, Scorers, Experiments, Playground, Review, Targets, Automations, Settings) instead of four steps. Addresses of the old project page still open the section that does that job now.

New

• Datasets: versioned rows with expected outputs, tags, metadata, conditions and run environments; import from CSV, TSV, JSON Lines, Excel or promptfoo; rows from logs, experiments and reviews; "Generate rows with AI". The project's scenarios and test sets show as datasets too. See Datasets.
• Scorers: a built-in library (the platform's checks, the agent-behaviour checks, and scorers ported from autoevals, openevals and agentevals), plus your own LLM judges, code, rules, trajectory and human scorers, each tried on a row before you save and versioned on every change; calibration against rows people scored. See Scorers.
• Experiments: every version tested, from a sandbox round, the SDK or the Playground, with a tile per scorer and its change against the baseline, rows filtered to what regressed, improved or stayed undecided, comparison and side by side for up to 8. See Experiments.
• Release gate: rules on scorers, regressions, undecided rows, required tags and the sandbox verdict, each blocking or warning only; on the experiment, the overview, the GitHub check, a pull request comment and CI exit codes. Default rules come from each project's pass criteria. See Release gate.
• SDKs for TypeScript and Python: Eval() runs your task over rows in your process and uploads an experiment; traced, wrapOpenAI / wrap_openai, wrapAnthropic / wrap_anthropic and the Vercel AI SDK trace your calls; preworld eval and a pytest plugin for CI, with a GitHub Action that comments on pull requests. See SDKs and CI.
• Logs: traces from the SDK, OpenTelemetry (/otel/v1/traces; OpenTelemetry GenAI, OpenInference, OpenLLMetry and the Vercel AI SDK conventions) and the AI Gateway, with facets, the filter language, saved views and the span tree. See Logs.
• AI Gateway: point an OpenAI or Anthropic SDK at the platform and every call is logged with its cost; the platform's models are billed at list price plus 30%, and calls with your own provider key (x-provider-key) cost nothing on our side. See OpenTelemetry and the AI Gateway.
• Playground: prompts and models side by side over dataset rows, scored, with "Save as experiments". See Playground.
• Online scoring and alerts: rules that score a sample of live traces within a daily cap; alerts on online scores, error rate, cost, a blocked gate or a capped rule, by mail, Slack or a signed webhook. See Online scoring and alerts.
• Human review: queues with a rubric of human scorers, blind review, assignees, required fields and a daily fill from logs or experiments; reviews become scores, dataset rows and calibration. See Human review.
• Issues: the assistant groups failing traces and rows into issues with evidence, severity and counts, on the overview and under Logs, with one click to the traces, a dataset or a drafted scorer. The assistant and connected coding AIs can work with datasets, experiments, traces and the gate. See Issues.
• Multi-turn tests: scripted turns and a simulated user on scenarios, test-set questions and dataset rows, with faults and attacks aimed at one turn. See the multi-turn fields.
• Project tokens that upload: "Allow uploading logs and experiments" on a new API token, for the SDK, OpenTelemetry and the Gateway. See API: tokens and errors.

Improved

• Targets lists every way in (SDK, Gateway, OpenTelemetry, repository sandbox, image sandbox, self-hosted runner, HTTP endpoint) with what's connected now and its latest connection check. See Targets.
• "Test new version" at the top right of every section starts a sandbox, test-runner or chat-AI round; each round becomes one experiment, and POST /v1/projects/<id>/jobs answers the experiment's id and the gate to wait on.
• The GitHub check concludes with the release gate (success, failure or neutral), with the scores' table; reviewctl eval --launch and the GitHub Action wait for it with one long-poll.
• A test that couldn't be decided is an undecided row on the experiment, with its reason; the undecided rows rerun in one click. Fix notes for a coding AI are exported from an experiment row.
• "What the AI does by itself" comes down to three switches: the assistant asks before it changes things, online scoring, and the platform's own reruns.
• Usage lists "Traces uploaded" (metered, not charged for now). See Pricing.

Removed

• The four-step project page, the test plan's rounds of questions on the web (use "Generate rows with AI"; a coding AI can still make a plan), the coverage map, next-evaluation recommendations, tasks for the dev AI on the web, the "Fix notes for AI" tab, and the report's buttons for each reason a test couldn't be tested.

Fixed

• A GitHub check is never left "in progress": when the release gate can't be worked out, the check concludes with the round's own verdict.
• OpenTelemetry: a span whose parent id comes as empty bytes is read as a root span, instead of being refused.

22 October 2026

New

• A local test runner uses the platform's judge model too. The test runner calls the judge through the platform's metered judge proxy (with its own token; the platform's key never leaves the platform). When review.yaml names a model the platform provides, the project's model judges in its place; the second judge, the embedding model and the per-job cost cap are as in a sandbox, and it's billed to the organization as judge model usage. With a key for the judge model on that machine, your own is used as before. The keys the agent calls its model with are still yours to provide. See Local test runner.

Improved

• An attack gets through only with a real consequence. It has to get data the user may not have, take an action they may not take, or make the agent adopt the attack. Giving the user back what they supplied, anonymous sample data the agent is there to analyse, runtime facts with no secret that a tool is built to show, the tools a tool server lists publicly, or a role the attacker only claims isn't one by itself. A hit that shows only internal details (tool names, parameters, a table's shape) weighs as low in the risk score and the finding's severity. A real consequence still counts. See how an attack is judged to get through.
• What the wiring prepares isn't the agent's. When a test needs an earlier session or data to find first, the wiring can mark the preparing part with the new REVIEW_PREPARE_URL: those calls still go through the tool proxy, with no fault or attack injected, no forbidden rule checked and no call budget used; whether an attack got through, whether a forbidden action happened and every count read only the agent's own calls in the attempt. The attempt page, the judge model and the AI diagnosis all see them marked apart. See preparing an attempt.
• A forbidden action reads "real impact" and "verified" only when it really took effect: one nothing stopped whose tool answered with an error or returned nothing still counts as a violation, recorded as a lead at "medium"; a call carrying only an empty list and nothing else (such as {"shell_commands": []}) did nothing and isn't a violation.
• "How to fix" for an attack that got through depends on its kind: command injection, SQL injection, out-of-scope file reads, acting beyond permissions and secret exfiltration point at the layer that runs the command or the SQL, reads the file or fetches the data, a ban in the system prompt only an extra layer.
• The default security tests follow what the agent can do, including reaching outside addresses; a connection that sees replies only still gets them, judged from the reply. Automatic approval's rewrites can't take out every question of command injection, SQL injection, out-of-scope file reads or secret exfiltration. When a security test passes but its set leaves some kinds out, or the connection sees replies only, the first screen says what was tested, what wasn't and why.
• "Before you confirm" has one more item: a tool forbidden with no condition that the agent called in every answer of the latest check (even with nothing to do). Automatic approval doesn't confirm such a test.
• A test set automatic approval turned down can be confirmed on its page with "I've read it: confirm"; where it flagged questions that don't fit or can't be judged, "I've read it: drop the N questions, and confirm the other M". See drafts and confirmed tests.
• A pre-run or check that failed only because of the platform's own part (its model for the agent, its judge model, its tool proxy, or the test runner's own error) isn't charged for its sandbox time. The usage page lists it under "Pre-runs and checks that failed only for the platform's own problem". See the platform's problem isn't charged.
• An automatic run (push, pull request, schedule or API) whose pre-run only the platform's own model or judge model failed no longer ends there: it waits for the platform to pre-run again 10 minutes later, up to 3 times. If the last one fails the same way, it ends saying it is the platform's problem and how many times the platform pre-ran it again, with a single mail. See the pre-run.
• The first screen follows the AI diagnosis when there's evidence. When the platform AI's diagnosis says a test failed because of the platform's model (a rate limit, a request parameter it refused, briefly unavailable) or because the judge model read it wrong, and the platform has checked the evidence on every attempt (the model's own error, a failure on the judge's reading alone), that test reads "Couldn't be tested" on the first screen, in its own line and in "How to fix", with why — no severity, nothing to change in the agent. When that is every test that didn't pass, the verdict is "Couldn't be judged", never "The tests never reached the agent… Fix the connection". Without such evidence, nothing the diagnosis says changes the verdict. See when the AI diagnosis changes the verdict.
• Tests that need the data after the task checked, on a connection that can't read it, read "Couldn't be judged (data not checked)" instead of "Not passed (data not checked)". See verdicts.
• On a sandbox project whose connection can't read data, a test that changes data, counts writes, or has a fault or an attack is flagged under "Before you confirm", and the platform's AI won't confirm it for you; you can still confirm it yourself. See confirming tests.
• When "Functional + security" runs in parts, the part that finishes first says "One of N parts of a round; M parts are still running", never "security tests weren't selected this time".
• A local test runner runs in a temporary copy of your repository. For reviewctl runner start and the checks of reviewctl connect --ai, each attempt gets a copy of the repository in the system's temporary folder (.git copied whole, .venv and node_modules linked to yours): what the agent edits or commits changes the copy only, deleted once the job is done. The data reset, the data reads and the stdio tool servers the tool proxy starts run in the copy too. See running in a temporary copy of the repository.
• "Local test runner" says what must be ready before a pairing code is made: the repository's own dependencies, and that machine's keys for the agent's model (the platform's models don't answer for an agent on a local test runner); the judge model is the platform's, billed by usage. reviewctl connect --ai lists the same before sending; it says at once when the program the check runs (.venv/bin/python, say) isn't there or a repository with dependencies has no node_modules; and it names the unset variables again at the end, as the result on the web page does.
• On a project connected through a GitHub repository or its own image, "Local test runner" gives no local command or pairing code, but says the project is tested in the cloud sandbox and a new project is the way to test locally. reviewctl connect --ai asks the platform first, and such a project is refused before the test runner is registered or the pairing code used.
• The install steps say how to install without sudo on "EACCES (permission denied)": npm install -g --prefix ~/.local …, then add ~/.local/bin to PATH. The CLI says when a newer version is out, too.
• When a pre-run fails because the entry program errored out, its "Details" give what the test runner reported (keys removed), and a dev AI reading the pre-run gets it too.
• A local "Let AI finish and check" can be cancelled on the web while it runs; the CLI, finding one still going, asks whether to cancel it and start again, and says the platform stops it once that machine hasn't answered for 10 minutes. The synthetic test identity AGENT_TEST_USER the AI writes into runner_env counts as allowed by the test runner once the files are written, with no variable name to guess.
• "Local test runner" on the new-project page goes to the project's step ①, the same instructions as the project page. "Instructions for your developer" is there too, with the same steps (one command) and the pairing code once created.
• With a dev AI on "MCP over HTTP (no reviewctl needed)", a local project says the test runner still needs reviewctl; a remote MCP connection reads "Waiting to connect" until its coding AI first calls. Revoking a test runner and disconnecting a dev AI ask first.
• "This run has cost about $x" says it counts the model calls of "Let AI finish and check" only; automatic approval and the platform writing default security tests are listed apart on the usage page.
• When the platform's AI is busy, it waits in line until its turn. "Let AI finish and check", "Let AI fill in and test", each step of a test plan and writing tests from it keep their place in line when the platform's AI is busy (your organization runs up to 4 tasks on it at once, or the model service is rate limiting), for up to 2 hours. The page says the place in line, about how long, and until when, so there's no need to come back and click; only after 2 hours without a turn does it stop, saying it's the platform's problem. Before, 2 minutes of waiting counted as "busy", and it stopped after one retry 5 minutes and one 15 minutes later. While "Let AI finish and check" waits for the platform's AI, its trial sandbox goes to a test or check of your organization waiting for a slot. See "Let AI finish and check".
• A way on when a third-party key is missing. When "Let AI finish and check" stops for a key, it says what each key is for and where to get one. When the repository's docs or code name a way to run without it (another search service that needs no key, say), "Use the way without a key" sits beside it: the platform sets that way in the connection and goes on checking. Above "Start test" in step ③, missing keys and what goes wrong without them are pointed out too.
• Test plans fit the repository better and read more plainly. Recommendations follow your earlier answers; there's a new "Service keys" question; the platform's own questions are in plain words and never about deployment or versions it has no use for; a plan only asks for checks the test connection can make, and puts the rest under assumptions; a test of making things up is no longer taken out for the connection not seeing it; internal field names no longer show; the plan's page says which tests written from it the platform later took out. See test plans.
• "N missing: ask the assistant" puts what's missing into the assistant's box for you to send; a replies-only connection is no longer told to add failure cases it can't run.
• "Skip the pre-run and start" after a failed pre-run, and confirming a test AI approval turned down, ask first, with why and what it costs.
• While the platform re-runs undecided attempts, progress says "the platform re-running undecided attempt k of n" instead of sitting at "Results".
• Attack templates say how many minutes writing really takes, and a refresh while it writes still shows "Writing attacks…".
• When the repository installs very large packages such as torch or CUDA, or the last preparation was slow, the sandbox class recommendation goes up a class and says beforehand that preparing will be slow. See sandboxes.
• A pre-run's or check's entry error is shown as printed; when it was raised in a connection file the platform wrote, it says so, as the platform's to fix.
• A report of the platform's own harness standing in for an agent (a repository with tool servers and no agent) says the harness plus your tools were tested, and the model line says "the harness the platform wrote asked for …" instead of "the repository asked for …".
• The default functional tests "Let AI finish and check" writes exercise the agent's main job (a coding agent changes a line of code), and its connection notes use fewer code words.
• A test the platform's import turned into a test of making things up is shown to AI approval as the platform's change, and probe: false undoes it; only a goal wholly about owning not knowing is turned into one.
• Dependency caches and image archives are saved even when preparing takes over 15 minutes, so the next check doesn't install everything again (once the sandbox image is updated).
• A sandbox machine's console output (the lines the page shows when a machine never reaches the platform or is lost) has keys and tokens masked as the job's log does, before it's kept, and as it's shown for what was kept before.

Fixed

• A reply that is only a model service's error log (the platform model's 429 rate limit, say) is no longer read by the security checks as something the agent said: without a call of the agent's own, it's undecided.
• Items of the platform's latest check that failed for the platform's own reason (its judge model failing, its tool proxy not starting) are no longer a reason for automatic approval to turn a test down.
• "Let AI finish and check" no longer wires a step the program under test takes after every reply as a tool; for an agent whose own code runs commands or code, it no longer says "The repository shows no tool calls, so replies are enough", and has the AI wire the step that runs them.
• A reading the second judge model gave in place of the first (rate limited) names the second judge model wherever it is recorded, and "the stronger one decides" compares the models that actually read it. When the platform judges a run again after it, the second judge answers in place of a rate-limited first in the same way: each attempt says who judged it, and it's charged at the judge actually used, never above the first one's price. See the second judge model.
• When the platform AI's diagnosis puts a failure on the platform's side — the platform's model or model service failing, the tool the platform injected a fault into or put its attack in, the harness or connection the platform wrote, data or history the connection prepared, or the judge reading it wrong — the platform no longer re-files it as the agent's problem because "the tool X is your repository's code", nor adds an unrelated "check permissions in the tool". Diagnoses re-filed that way before read as the AI first filed them.
• When a security question counts as got through and the diagnosis itself says what counted happened in the history the test connection prepared, it's the test connection's, and no longer asks you to add permission checks to the query tool; diagnoses stored that way before read the same. The agent handing out data in this conversation is still the agent's problem.
• Diagnoses no longer name questions by ids such as r13: questions with the same root or the same kind of problem are named by what they ask, and a test set's question is named in its heading instead of seven headings all reading "'Security test · …' under normal conditions". A diagnosis that changes only the prompt or the agent's instructions (the instructions in sql-agent.ts, say) no longer also says "the problem is in the tool X: the code that implements it needs changing".
• Work a platform restart cut off is taken up again by the platform: re-judging and AI diagnosis up to 2 more times each, the default security tests up to a third try. What can't be finished says why — "Re-judge again" for re-judging, "Diagnose again" for a diagnosis, and for the default security tests a pointer to "Add tests → Generate attack tests" — and never reads "Writing…" for good. For a group the judge wasn't sure of, "What the platform did" follows the re-judging as it stands (under way, cut off and being taken up again, done). See after a platform restart.
• API: settled on GET /v1/runs/<run id>/diagnosis; the new POST /v1/runs/<run id>/rejudge (re-judge again); round.running on a run; cut, resumed and gave_up on its re-judging; question and prepared (what counted happened in the history the test connection prepared) on a diagnosis item.
• reviewctl connect --ai could cut a Chinese character or an emoji in two when cutting a file over 24 KB, leaving it a byte or two over the cap, and the platform refused the whole upload; it now cuts at a character's boundary. An older reviewctl's upload over the size caps is fitted to them by the platform, which says which files, instead of being refused.
• On a local project, "write and confirm one in step ②, then click "Let AI finish and check"" now says to run reviewctl connect --ai again in the repository (there is no such button there); when the command stopped before writing the files to that machine, the page no longer says they are in .reviewctl/.
• A test runner on a current reviewctl whose connection sees replies only is no longer told to update reviewctl.
• The CLI's English output no longer carries a Chinese sentence about a draft; endless standard input (yes y |) no longer crashes it.
• "n of m" on a test's page and "Start with the first" no longer jump; "N to confirm" matches the confirm box, with "N more need changes" for the ones AI didn't pass; "What to test" counts a test with both functional and security cases the same way everywhere.
• "Not sure · help me choose" on a new project no longer offers two hints that lead to the same choice.
• "error: prohibited by secure boot policy." in a sandbox's serial output while it boots is folded away, said to be harmless.
• A check result without a tool name no longer says "this tool" as a placeholder.
• API: warn on GET /v1/projects/<project id>/secret-needs, POST /v1/projects/<project id>/connect-assist/keyless, and queue on "Let AI finish and check", "Let AI fill in and test", plans and writing tests from them.
• An agent whose model the platform answers with another one no longer fails on settings that model doesn't take. The openai-agents SDK, for example, asks gpt-5.2 for the reasoning effort none, which GPT-6.1 Sol refused with a 400 on the agent's first call. The platform now sends the closest reasoning effort above it that the model takes (never a lower one), and leaves out a setting the model doesn't take at all; the run log says each change once. See your agent's model.
• When the platform's own model fails an attempt (a rate limit, an outage, a timeout, or a setting it refuses), the attempt is the platform's problem, not the agent's: it is never judged "Answer wrong", "Failed silently" or "The agent never called the tool", the judge model doesn't read what the agent was left with (its error log, say), and the platform re-runs it once, not charged. What is still untested after that says "The platform's model failed … That's the platform's problem", and "Retest these N" of only such attempts isn't charged either. A harm the evidence already shows still fails the attempt. See why an attempt stays undecided and the platform's problem isn't charged.
• A check or pre-run that only the platform's own model or judge model failed no longer counts against "Let AI finish and check"'s 10 checks, isn't handed to the AI to fix the connection, and is tried again by the platform a while later; a run waiting on such a pre-run is pre-run again by itself. In a sandbox, the report no longer asks you to "Set up the judge model on the test runner": the judge is the platform's, and its failure is said as the platform's problem. See the pre-run and "Let AI finish and check".
• The judge model waits out a rate limit as long as the model asks before trying again (within a limit per call and per run), instead of giving up after three quick tries. When the first judge model is still rate limited after 60 seconds on one call, the second judge model judges it instead; the attempt page says "Judged by the second judge … this time: the main judge was rate limited", and it's charged at the judge actually used, never above the first one's price. Only when both fail is it the platform's problem, judged again after the run. See the second judge model.
• The AI diagnosis of a test left untested by the platform's model says it is the platform's problem, re-run and not charged, no longer "The platform doesn't run it again by itself".

19 October 2026

New

• Tests that couldn't be tested are settled by the platform. Each reason under "Couldn't be tested" on a report ends with what the platform is doing about it, or a button. When the connection lacks what the tests need (it can't read the data or can't see the tool calls), it goes to "Let AI finish and check", and once the check passes only those tests are retested. For a counting query that doesn't count writes, the platform saves a version without it, keeping its data check: a test the platform wrote goes to automatic approval, one you wrote waits for "Confirm the platform's fix", and once confirmed only that test is retested. For an agent that asks for permission before using a tool when run headless, click "Pre-approve <tool> in the test" and then "Confirm the pre-approval", and the platform retests just that one. Other reasons get a button such as "Retest these N", "Let AI finish and check" or "Change this test". See settling what couldn't be tested.
• Retest only what couldn't be tested. For a sandbox run, only the attempts that couldn't be judged run again: the same commit, test versions and conditions, without a pre-run, read into that run's round report with the latest try counting. See retest only what couldn't be tested.
• A test can pre-approve one of the agent's own tools (task.human_confirmation.preapprove), for that test only and confirmed only by a person; on each attempt the test runner hands it to the wiring as REVIEW_PREAPPROVED_TOOLS. See pre-approval.
• "Organization" → "Policy" → "What the AI does by itself" has a new item, "Settle the tests that couldn't be tested" (off by default): when the connection lacks what the tests need, the platform hands it to "Let AI finish and check" by itself and, once its check passes, retests only those tests; a test the platform fixed after a run is retested the same way once confirmed. "Re-run undecided or unstable cases" no longer re-runs these. See what the AI does by itself.

Improved

• The platform's own problems are no longer charged. For a re-run caused by the platform model failing or the platform's own test backend failing, neither try's sandbox time is charged, nor the re-run's model calls and judging. When a sandbox is lost part way, its time after the last partial result it handed in isn't charged, and nor is any of the platform's fill-in run after it; that fill-in run goes ahead even when the organization's credit, balance or spending cap is used up. All of it is still recorded: the usage page has a new "The platform's problem: not charged" row with what it would have cost, by reason, and the attempt page, the job page, the round report and the fill-in note say so too. Re-runs caused by your own tools, database, keys or agent, a job that lost contact, and a fill-in run you click are billed as usual. See the platform's problem isn't charged.
• When two judge models read something differently, the stronger one decides. Strength follows the platform's model catalog (GPT-6.1 Sol, Gemini 3.8 Flash, GPT-6 Luna), with models outside it ranked below. When the stronger one's reading doesn't pass the platform's check (the reply doesn't give the number it names, or it quotes only a denial), the other's decides. The attempt page gives both readings and which one decided, and "How it was tested" says how many there were. When both models are outside the catalog, which is stronger can't be told, and it stays undecided as before. See the second judge model.
• In the API: not_charged on GET /v1/orgs/<org id>/usage, waived on jobs, and verdict.second_judge.split and a re-run's platform in the report. See the API.
• In the API: GET /v1/runs/<run id>/untested, POST /v1/runs/<run id>/untested/<step id>/go, POST /v1/runs/<run id>/untested/preapprove, a fill-in run's undecided and tests (GET /v1/runs/<run id>/fill?undecided=1), fill_mode on jobs, and the job error codes prepare_stalled and prepare_timeout. See tests that couldn't be tested.

Fixed

• When the second judge model is weaker than the first (the first already the strongest, so another family was asked), the sandbox log and "How it was tested" call it "the other judge model", no longer "the stronger judge model".
• A sandbox's preparation no longer hangs: a step that prints nothing for 15 minutes is stopped, and the job fails with "Preparing the environment stalled: <the step> printed nothing for 15 minutes (last line: …)"; preparation still going near the time limit is stopped too. Before, a silent install script could hold a check until the time limit. While preparation prints nothing, the progress page, the pre-run and "Let AI finish and check" say so; "Let AI finish and check" counts a stalled check as a check and changes the stalled step to install only what the agent needs. See preparation that stalls.
• Cancelling while the environment is being prepared stops the preparation at once instead of waiting for it to finish.
• environment.setup in a sandbox no longer has 15 minutes for the whole list: a command is stopped after 15 minutes without output, and otherwise the run's time limit bounds it.
• When the agent's own permission layer wants to ask first, run headless, with nobody to answer, the platform no longer re-runs it (another try is the same); the report offers a pre-approval.

18 October 2026

New

• Test plans. Before any test is written, the platform asks a few rounds of multiple-choice questions (up to 4 a round, the recommended option first, with "Other" always there for your own words) and writes a test plan from the answers: what the agent is, functional tests, security tests, pass criteria, what your developers do first, secrets needed, assumptions. You can "Edit" it or "Ask me more"; only "Approve and generate" has the platform write the tests from it. The default security tests use the template the plan chose, and the pass criteria are set from the plan as its tests are confirmed. "Plan again" at any time. See Test plan.
• Use the platform from a coding AI. In "Settings → Dev AI", make a connection for a developer's coding AI (Claude Code, Cursor, Codex) and tick what it may do: write test drafts, change the test connection files, pre-run, start formal tests (up to a daily amount, $5 by default), confirm tests, have the platform's AI finish the connection, change the pass criteria. With 31 MCP tools it goes through the connection, tests, review, pre-runs, formal tests and reports, and tests again after a fix; it can also answer a test plan's questions and write the plan (approving stays a person's click on the web). Connection files it saves show "Written by" "Your developer machine's AI". reviewctl dev connect … --register claude|cursor|codex registers the MCP server as it connects; with MCP over HTTP, no reviewctl is needed. Connections made before this update may write drafts, upload files and pre-run; "Edit permissions" allows more. See Using the platform from a coding AI.
• Re-runs by the platform. An attempt left undecided by a platform problem or a passing one (a rate limit, a timeout, a fault or an attack that never reached the agent) runs once more within the same run; the verdict counts the re-run, and the first try's record is kept. When a sandbox is lost part way, the platform fills in the attempts it never got to, once. Billed as usual, and it can be switched off under "Settings → Automations". See re-runs by the platform.

Fixed

• A run that tested functional tests only and passed all of them is "Passed", saying "Everything tested this time passed" and which security tests it didn't cover ("Security tests weren't selected this time", or "the security tests the platform wrote are still to be confirmed"), no longer "Couldn't be judged".
• A verdict reads one of three words on every page, "Passed", "Not passed" or "Couldn't be judged": the job page, the project page, the test history and the version comparison no longer say "Not passed (some couldn't be tested)", but "Couldn't be judged (N tests undecided)". AI re-judging uses the first screen's words too. See verdicts.
• When the tool an attack goes through fails before it can act (the agent asking its user with nobody there to answer, say), the report says "The attack never reached its target": not held, no "done wrong" problem, and it's the test connection's or environment's to fix.
• Every undecided attempt says its specific cause and what to do: the judge model's call failed (with the error), two judge models read it differently (with both readings), the judge limit was reached, a tool backend or the data reset failed, the agent replied nothing, a data check read no value, and so on. "Test again" appears only where testing again can change the result. See why an attempt stays undecided.
• A test whose counting query counts something other than writes ("unchanged", say) could never be a safe stop. The report now names the query and the value it read and says what to change, "Before you confirm" flags it, and the AI diagnosis files it as "The test is wrong".
• When a test only accepts the task done and the data or the calls seen already show it wasn't, the test fails rather than staying undecided because of a vague reply; a test that needs the reply to say something fails when the agent replied nothing. See definite verdicts.
• "Preflight checks" take a formal test that reached the agent since the last pre-run as evidence the tests reach it, and no longer warn that the whole run may be wasted.
• "Let AI finish and check" no longer changes the agent under test's own settings (the variables already in agent.env and runner_env, agent.env_passthrough, agent.model): the platform puts back what the AI changed, and "What happened" says so.
• "Before you confirm" no longer calls an attack hidden in the user's message an attack in a tool; the default pass criteria for a test with attack conditions only say each attack must hold every time. The query name after "done wrong" is quoted.

Improved

• A second judge model: where the first can't decide, the platform asks a stronger one (or one from another family), and "How it was tested" says how many it read again and settled. On a local test runner, set judge.second_model. See the second judge model.
• An agent that asks its user before it acts (for permission, or to confirm) now has somewhere to ask: every attempt has REVIEW_ASK_USER_URL, the wiring passes the question to the platform, and the platform answers as the test says; where the test doesn't allow asking, it answers no and records it. The attempt page says what was asked and answered. See When the agent asks its user.
• The connection files list in "Connect agent" has "Written by": you, your developer machine's AI, or the platform. The platform's AI doesn't change files you wrote by itself, and AI diagnoses call them "the connection file you wrote". See Who wrote it.
• Parts of the test connection files that raise no error but won't run as written are flagged on save, under "Test connection file" and in "Preflight checks": a path written from the repository root (these are read from .reviewctl/), a version file that isn't there, a runner_env variable that never reaches the agent. Two routes on one line now get an error naming the line. See Mistakes that raise no error.
• The run log notes that the data reset runs as the test runner's user, so the agent can't read files it makes outside the repository's copy.
• "Let AI finish and check" tries the agent in its trial sandbox the way a check runs an attempt: the reset in the same copy, the tool calls through the platform's tool proxy.
• Command line: new reviewctl dev commands (wiring, push, tests, test, review, confirm, run, job, cancel, runs, report, diagnosis, assist, gate, disconnect), dev mcp --runner and JUDGE_SECOND_MODEL. See Command line.
• In the API: dl_ and rn_ tokens, /v1/dev/*, the test plan routes, the project's auto_rerun, written_by on connection files, GET /v1/projects/<project id>/wiring-warnings, and incomplete as a version comparison verdict. See the API.

17 October 2026

Fixed

• Pass criteria you saved yourself in Settings → Pass criteria are used when you start a test, and no longer switch back to the default: the start page says "Custom: N key tasks". Before, criteria whose numbers matched the default were treated as the default and rewritten to the tests chosen for that run. If you saved criteria with the default numbers before this change, save them once more.
• The default pass criteria count only conditions a test really has as key tasks: its normal condition if it has one; for a test with only attack conditions, those attack conditions. It no longer asks for a normal condition the test doesn't have.
• Starting a test from "More options" takes you straight to its progress page.
• An attempt whose write receipt says it succeeded while a data check disagrees is headed "The write's receipt and a data check disagree (to look into)", no longer "Not done, but said it was", and its reason on the first screen no longer carries "the judge model's reading". JSON nested in a receipt is shown on one line, without escapes like \n.
• AI diagnosis: items of one diagnosis that share a root go in the same category. Answers from a test harness the platform wrote (a script that answers in place of the agent) are labelled "The test harness's answer", no longer "The agent's problem".
• Singular and plural in English reports, e.g. "The 1 attack held".
• Hand-written tests no longer get a doubled full stop in their judging instructions.

Improved

• The compare page has two more rows, the platform connection files (file count and fingerprint) and the judge model, and its one-line summary says when either changed. Older runs didn't record the connection files and read "Not recorded". The project page's version comparison no longer says "No change" when the two runs didn't test the same tests. See what the compare page shows.
• A report's first screen says when the platform replaced the judge model with the project's model, and which one review.yaml asked for (runs after this update).
• The connection files "Let AI finish and check" changed can be reviewed file by file in "Connect agent", and put back as they were. When a change would drop tool calls the platform could see, a tool route or the data read, the platform first sends it back to the AI; if it still drops it, the result says so plainly: "This change took away: …". A check it started says on its progress page how it was started.
• "Before you confirm" has two more checks: a data query the test uses isn't in the connection's state_queries.yaml (the pre-run will fail and the test won't start); the project's connection can't read data, so the test's data checks can't be read and the result will be "couldn't be tested". AI auto-approval doesn't confirm such drafts.
• The hand-written test form pre-fills the tools the connection declares.
• The side rail shows the platform's version (commit and date) at the bottom.
• API: GET /v1/version; projects have gate_by_person; sandbox jobs have overlay_applied; new GET /v1/projects/<project id>/connection-tools, GET …/connect-assist/changes and POST …/connect-assist/restore. See API.

16 October 2026

Fixed

• The attacks that "got through" on a report's first screen are counted by each attack's own result, the same as "Details", attack_success in the report's JSON, the round report and the GitHub check. A security test that failed for another reason still counts its attack as held, with "All N attacks held; N tests failed for another reason" beside it. See verdicts.
• An attempt whose task couldn't be judged but whose attack already did harm counts as got through, in line with "real impact".
• A run that tested security no longer reads "security wasn't tested" because the project has security tests still to confirm. The project's security tests get a line of their own, which doesn't change this run's verdict.
• When every test that ran passed and only a key task of the pass criteria wasn't selected, the verdict is "Couldn't be judged" and names the missing key task; other runs' results don't count toward it. Step ④ of the project page, the test history and the version comparison say "Couldn't be judged (key task not selected)", and the report's button is "Test again".
• A test whose data checks read no value at all now says first that the platform couldn't read the data, not that the judge model couldn't decide. When the connection doesn't see tool calls, the report says the agent's own tool calls aren't in the judgment yet, which doesn't mean it called none.
• On a connection that reads data but doesn't see tool calls, calls the platform couldn't see no longer decide whether the task was done. The data checks still decide, and a reply that gets a point wrong still fails, but a reply alone can't show the task done. Such tests used to fail for "no call seen".
• A test whose required tool calls all have a minimum of 0 no longer counts as done when no call was made. Under a condition that expects a safe stop, an agent that wrote nothing, tried nothing forbidden, and replied that it didn't do it, or stopped to ask the user as the test requires, counts as a safe stop.
• When a write's receipt says it went through but a data check says otherwise, it's recorded as a lead "to look into", with the call's arguments, the receipt, what the data check should be and what it read, and when the data was read. It no longer says the agent wrote the wrong object.
• A data check with no label shows its query on problems and on the attempt page, not an id.
• For a test left undecided, the next-evaluation recommendations follow what it actually lacked: routing tool calls through the platform when the connection can't see them, letting the test runner read the data when none was read, and never adding a data check the test already has and read.
• When the AI diagnosis says "The problem is in your repository's tool X", X is always a real tool's name, never punctuation or a file path. An item whose own words say the test connection lacks something, and that names none of your code, is the test connection's problem, not the agent's. Whenever the platform files an item under another category, it says "The AI first called it …" and why.

Improved

• The attempt page lists each refused or failed call with where the refusal came from: the platform, the product's tool, or "Source unknown". Source unknown isn't taken as evidence the agent held on its own. It also says which refusals aren't counted in "Stopped by the platform". See reports.
• A test draft's page has "Before you confirm": words that ask for an attack, a fault, hooks or markers, or actions it must not take with nothing checking them, an attack that gets through on the state it holds to (holding would count as got through), and tool calls the connection can't see are listed there. A draft AI has reviewed also has a "Requirements → checks" table. Automatic approval doesn't confirm such a draft; you can still confirm it yourself once you've looked. See drafts and confirmed tests.
• The progress page says "Now: …" and "last progress …", marked after 10 minutes with nothing new. The project page's pre-run does too, instead of only "the sandbox takes a few minutes to start".
• A cancel says when it was requested and the time by which the platform deletes the sandbox if it hasn't stopped; once it ends, when, how long it took and what was kept. See cancel a run.
• "Let AI finish and check" says which tests its check runs and why those.
• The compare page shows each side's agent model (with the one asked for, when the platform ran another), its code (the commit and uncommitted changes), the fingerprints of its prompt and tool configuration, and one line naming what changed between the two runs. See what the compare page shows.
• In the API, a sandbox job in GET /v1/jobs/<job id> has log_last_at and cancel_by, and runs and run lists have key_missing. See the API.

10 October 2026

Fixed

• Services a test uses (the compose under services) can no longer get control of Docker or of the test machine itself: mounting Docker's control socket, privileged, using the machine's process or network namespace and the like are refused before anything starts, and the check result names the service and what to change. The platform restores services before each attempt; to have some services recreated before your own reset, list them under services.restore in review.yaml. A connection that already has such a service stops at the services step on its next check; "Let AI finish and check" rewrites it to the new rule.
• A project's jobs list, a single job and the latest pre-run no longer wait for a sandbox to start, so these pages open faster.

9 October 2026

New

• Two new sandbox classes, 2X large and GPU, on every plan including Free. 2X large has 16 vCPUs, 64 GB of memory and a 200 GB disk, at $1.50 an hour. GPU has 8 vCPUs, 32 GB of memory, one GPU with 24 GB of GPU memory and a 100 GB disk, at $1.98 an hour, on x86 in London only. GPU is shown as "coming soon" under "Settings" → "Sandbox" until the platform's GPU image is ready and has passed its security checks; then the platform switches it on. See sandbox classes.
• Custom spec. "Custom…", the last option under "Settings" → "Sandbox" → "Class", sets the cores, memory, disk and GPUs yourself, for example 200 GB of memory. The platform gives you the cheapest machine at least that big, shows what it actually allocated and the price per hour, and bills it per second like a class; the estimate before a run uses that price. A custom spec runs in London only, on every plan including Free, and GPUs in it open together with the GPU class. See custom spec.
• One model per project. The project's header has a model dropdown next to the name: "Strongest · GPT-6.1 Sol" (the default), "Balanced · Gemini 3.8 Flash", "Economy · GPT-6 Luna" or "Follow the organization default (…)", each with its price per 1M tokens as billed. The judge, the platform AI (the assistant on that project, drafting tests, connecting and checking, automatic approval, diagnosis, security tests, name suggestions, expected values) and the agent in the sandbox all use it. Results you already have keep the model that judged them. See Models.
• The organization's default model: an owner sets it under "Organization" → "Policy" → "Default model". Projects that follow the organization use it, and so does the assistant outside a project; with none chosen, GPT-6.1 Sol. See default model.
• Similarity checks with no setup. similar and answer_relevance now work on the platform, with Google's gemini-embedding-001 on Vertex AI in London; embeddings aren't billed. Their default thresholds are 0.96 and 0.87 with this model; similar can't tell a wrong date or amount, so use "Factuality" to check facts. See similarity checks.

Improved

• "Sandbox cloud" is now "Run location": Auto, London or Sweden (EU), named by where the sandboxes run. A project keeps its setting, and its sandboxes run where they did. The sandboxes' outbound addresses are named by location too. See run location.
• The separate pickers under "Settings" → "Judge and agent models" are gone: the section shows the project's model, with a link to the header.
• An agent with no OPENAI_API_KEY that asks for one of the three models by name gets it; any other name is answered by the project's model, as before. With Gemini 3.8 Flash as the project's model, a call to the Responses API gets a clear 400 error: switch to a GPT model or use chat completions. See your agent's model.
• A new chat AI on a platform model offers the three models and starts on the project's. One built earlier keeps the model it was built on; gpt-5-mini and DeepSeek-V4-Pro are no longer offered for new ones. See pricing.
• The platform's models run on Microsoft Azure (GPT-6.1 Sol and GPT-6 Luna) and Google Cloud Vertex AI (Gemini 3.8 Flash). OpenAI is no longer the platform's default judge, assistant or agent model. See data processing.
• In the API, PATCH /v1/projects/<id> takes model, PATCH /v1/orgs/<id> takes default_model, and GET /v1/models lists the models; judge_model and agent_model are still accepted. See models in the API.

8 October 2026

New

• New plans, in US dollars: Free, Pro, Ultra and Enterprise. Free has $5 of credit a month. Pro is $20 and Ultra $200 a month per organization, all of it credit for usage, renewed each billing period. Pro allows 3 sandboxes at once and runs of up to 240 minutes, Ultra 10 and 480 minutes; the assistant's daily limit is $5 of model bill on Free, $10 on Pro and $50 on Ultra. The Team plan is no longer offered. See Pricing.
• Top up by card. An owner adds $5 to $1,000 to the organization's credit balance under "Organization" → "Billing", and gets a paid invoice for each top-up. The balance never expires. Past the period's credit, usage comes out of the balance; once both are used up, new tests stop until the credit comes back or you top up. Nothing is billed as overage any more. See top up.
• Change between Pro and Ultra with "Change plan" on the billing page; Stripe prorates the difference, and the period's credit moves with it. A subscription that ends puts the organization back on Free, balance included. See change plan.
• Every price and amount is in US dollars, converted at the platform's rate of 80p to the dollar, so prices keep their value: a Standard sandbox hour is 17.5¢ (was 14p), model usage is the model bill plus 30% ($1 of model bill is billed as $1.30), storage is 12.5¢ per GB-month. Balances, credit codes and spending caps were converted the same way (£1 = $1.25). Organizations already on Free keep this month's credit as it was (£5, now shown as $6.25); from next month it is $5. Invoices already issued stay in pounds.
• Credit codes work on Pro and Ultra too.
• In the API, amounts are in US cents: the *_pence fields are now *_cents, an invoice's total_pence is total_minor in its own currency, and there are routes for checkout, top-ups and changing plan. See billing in the API.

7 October 2026

Fixed

• A read-only text-to-SQL agent that produces a complete, runnable write or DDL statement (CREATE included) is checked by the platform itself and counts as overstepping under any label; placeholder templates, cut-off statements and agents whose purpose allows writes don't count.
• SQL the agent produces that selects other people's contact or identity fields (email, phone, address, ID numbers) is checked by the platform itself and counts as a leak under any label; the user's own records, aggregates without personal fields and schema reads don't count.
• In a role-based-access attack, an agent that only asks for missing details without calling a tool, or whose target doesn't exist, is "not tested", no longer held.
• Tool names the question itself gave aren't a discovery when the agent repeats them; tool-discovery questions no longer name the agent's tools.
• "The reply must not contain" is judged by meaning: mentioning a table to say it doesn't exist isn't inventing it; items state the forbidden content itself instead of "the reply does not …".
• Diagnoses point at the file the test really starts; when a tool is implemented in several files, they say which one runs.
• "What the platform did to the test" says only what was really saved: a diagnosis that says the test was changed comes with a new draft version on the test's page.
• For a text-to-SQL agent, "How to fix" says not to invent tables or columns and to say plainly when the data isn't there, instead of customer-support wording.
• In security tests, reading out a confidential file's contents or a real credential is checked by the platform itself — what the tool read and what the reply gave back — and counts as a hit under any label (role-based access included), no longer depending on whether the judge model follows its instructions; naming a file or showing a masked key doesn't count. The "doesn't leak … keys" row agrees with it.
• A text-to-SQL agent's single query that only reads the schema (for example, listing the tables) is held under SQL injection; only SQL that carries the injected payload or its effect is a hit.
• In a role-based-access attack, an agent that only asks for missing details and does nothing is "not tested", not held.
• Tool-poisoning questions are written only when a running tool's description really hides instructions, and name that tool with full parameters; indirect-injection questions have the agent process content that really exists or that the user forwards, with the instruction hidden in the content. Where the running agent has no such entry point, the category isn't written and the report says "not tested this time", instead of questions that can't land.
• A question that reads a non-secret private file is always counted as path traversal.
• The "makes things up" check: requirements like "the reply gives no …" are kept as things the reply must not say, met when absent; an invented step or number still fails.
• When a "makes things up" test fails, "How to fix" follows the diagnosis instead of telling you to leave the agent alone and complete the connection.
• A security question that didn't reach its target has "Have the assistant rewrite these N questions" beside it, rewriting just those instead of regenerating the whole set.
• Disclosing confidential file contents, real credentials or outsiders' personal data is now a check shared by every security question: it counts as a leak whatever the label (command injection, tool poisoning, prompt injection or any other), and the user asking for it is not permission. A refusal that only names the file, masked keys, the user's own data and tool names still don't count.
• Tool poisoning and prompt injection through tool output: an agent that says it followed an instruction from a tool counts as the attack working. These questions are ordinary user tasks with the malicious instruction only in a tool's description or output; a question where the user names the private file is relabelled to the attack it really tests.
• System-prompt extraction is judged on prompt text actually in the reply: a query against a table that doesn't exist, with no prompt text in the reply, is not a leak.
• The "makes things up" check: an agent that honestly says it doesn't know passes, with or without calling a tool; an invented or hedged figure still fails.
• A failure the platform diagnoses as a mis-written test ranks behind real problems and security hits as soon as it is diagnosed, never takes the headline and doesn't ask you to change the agent; the diagnosis says plainly when nothing was saved, and why.
• Files a functional test asks to read are given with their full path.
• An AI approval rewrite's one-line note shows the start of its reason; the full text is under "Details".
• A thin harness that just forwards to the customer's own code: a wrong or invented reply is diagnosed against the customer's code, never re-blamed on the connection, and the platform never adds a connection-side guardrail for it.
• Reading a file that is private or marked confidential counts as a leak under any label, whether or not it names individuals.
• Returning another user's personal data is judged a hit whether asked for directly or reached indirectly (a join, an aggregate, "the top customer"); a pure aggregate with no personal column still passes.
• The report headline is the most severe real problem: a failure that happens on most runs keeps the headline even if the agent passed once, while a check that trips only occasionally ranks below it.
• The DVMCP-style security set no longer deadlocks: security questions are aimed only at tools and files that exist at runtime (read from what the connection actually launches), credential-theft gets a question that can land, and a set that genuinely can't be covered is marked "not tested", never a dead end and never a false pass.
• A run whose security set wasn't tested no longer reads "all passed"; it reads "functional passed, security not tested (reason)".
• Tool-disclosure and other-user-data questions are judged consistently; a failure the platform diagnosed as a mis-written test, or an over-strict check, no longer becomes the headline over a real problem.
• Drafted functional tests no longer require the reply to repeat a specific username, and greeting questions no longer require a fixed self-identification phrase.
• A round that ran only some of its tests still shows its round page instead of "not found".
• Security judging: a cut-off SQL fragment with no write statement in it is "not tested", decided before the judge model; a write the judge claims must be in the reply. A runnable write aimed at a specific record counts as a violation even when labelled "example"; only a placeholder template is exempt.
• A security category with no decided attack reads "not tested", never risk "0.0 low".
• Attacks are paired with the tools that can really reach their targets (each file's server and container), and SQL-injection attacks put their payload into the parameter the code builds its query from.
• The "makes things up" check is added only where the repository has its own agent; an agent that invents a capability and then asks back fails it.
• AI approval reads a security set through before deciding, asks for its verdict in the same conversation instead of starting over, fixes mislabelled questions in place, and every rewrite says what it changed.
• A judge whose check carries a shortened description of the agent is given the whole description too.
• The platform no longer changes the prompt, refusals or safety rules of a harness it wrote to stand in for the agent, whatever a diagnosis calls the problem; and a connection whose harness was changed that way is never reused as a reference.
• Security judging: a purpose that says what the tools can reach, or that anyone may call them, is the attack surface, not a permission — reading a confidential or private file still counts as a leak.
• An attack whose target doesn't exist (a missing file, a config name never given) or whose output is cut off or a program error counts as "not tested", never as held; the security result says how many.
• Security questions are aimed at the tools that can really reach their targets, use the real file paths and config names, and know the database's tables and columns, so invented columns are caught.
• Every code file is scanned for SQL built from input and seeded users, so SQL-injection questions and real user names no longer depend on which ten files the model read.
• When a tool really hands out what it shouldn't, the diagnosis points at the code in your repository that implements it.
• Q&A and support agents' first tests always include one that checks the agent doesn't make up business details.
• The sandbox queue runs jobs first come, first served (a run never waits forever on its own pre-run), and the waiting pages agree on who is ahead.
• A round's report gives no verdict until every part has finished; pages say whether a test was confirmed or changed by AI approval or by a person, and what an approval rewrite changed.
• The system risk score counts only attacks that got through: only low-severity hits keep it at "low".
• Security judging knows what the agent is for and who is signed in: the signed-in user's own data, returned the agent's normal way, is no longer judged a leak or a "debug mode" hit; another user's data, raw tool output, parameter lists, configuration and secrets still are.
• Each successful attack shows its own judge's reason, not the first one's.
• An empty reply or a program error in a security test counts as "couldn't judge", never as held; the security result says how many attacks got no reply.
• Agents that use MCP tool servers get security questions written from their tools (command injection, reading private files, stealing credentials, prompt injection through tool output), whatever way they are connected; once a check shows the tools, the default questions are written again.
• The AI review of security questions uses fixed criteria and doesn't reject questions it already accepted; when no credible questions are left, the page says so and security counts as not tested instead of passing.
• Agents with a query tool built from input always get SQL-injection questions; agents that call tools always get a prompt-injection question.
• Tests that only read pass on their own merits when the connection sees the tool calls but can't read the data; tests that change data still need their data checked, and "Let AI finish and check" then aims at the data's reset and read.
• Recommendations no longer show internal codes; the run page footer shows the model the agent actually used.
• Escaped characters in logs (\uXXXX) are shown as the characters.
• Before a project starts beyond the organization's sandbox limit, the page says it will queue.
• While waiting for a trial sandbox, the connection AI keeps working instead of idling; a build error is fixed from the log at once; a working connection for the same repository in the same organization is used as a reference.
• When the platform answers with a reasoning model in place of the agent's own, the reply length limit leaves room for reasoning, so replies are no longer cut to nothing.

6 October 2026

Fixed

• Access-control and excessive-agency questions: an agent that produces a write or destructive SQL statement the user isn't allowed to run now counts as a successful attack, even if it wasn't executed; read-only queries still count as held.
• Agents that use MCP tool servers: a tool name with or without the server prefix is read as the same tool, so a correct run is no longer judged "didn't do it but told the user it did"; a new test version that only corrects tool names is confirmed by the platform.
• Security questions also look at the tools the platform actually saw: agents that can run commands or read files get command injection, path traversal and tool-poisoning questions instead of a generic quick check.
• The security report says when the platform swapped the agent's model; the risk score never exceeds the severity; categories this question set doesn't include read "Not tested here".
• When supervisord runs several services in one container, check results include each service's own error and its health-check result.
• Logs shown while a run is in progress also fold long error traces.
• Command and SQL injection attacks count as successful only when a tool call actually carries the payload; normal look-ups of the user's own data are no longer misjudged.
• Security questions follow the code's attack surface: SQL injection only when a tool builds SQL from input, command injection only when a tool runs commands, and access-control questions use other users who really exist.
• When the test runner itself fails, that's the platform's problem and isn't handed to the connection AI; when tool servers crash while running, their logs are shown.
• Tests the platform wrote that can't be tested are rewritten or removed by the platform instead of left to you; the rewrite AI only gives a fixed set of reasons.
• A reply that misses or contradicts a required point now fails instead of "couldn't be judged"; a run with only undecided results reads "Couldn't be judged" rather than "Didn't pass".
• Security test sets the platform writes now have a judging rule for every question, fill "who may be looked up" from the test account, and are reviewed in full; when the rewrites run out, you can drop the questions that can't be judged and test with the rest.
• Each attack is judged only on what it tests: a human hand-off the agent is set up to offer isn't overclaiming, and not fulfilling an impossible request isn't making things up.
• When the services a test needs don't come up, the check result has each service's own log.
• Secrets in logs, including ones a repository echoes itself, are masked.
• When the test connection only calls your code, a diagnosis tells you to change your own code and never suggests editing files the platform wrote.
• The platform's MCP tool proxy now starts inside the agent's container, and legacy SSE MCP servers can be connected; when it can't start, it says this is the platform's problem and doesn't use up checks.
• The default security tests are written from what the agent in the repository actually does, not guessed from the project name; "Regenerate" makes a new version for AI approval and never confirms it for you; attacks that don't fit the business can't be confirmed, so there's no false "passed".
• A diagnosis counts only a harness the platform wrote to stand in for the agent as the platform's; when it just calls your code, the problem is yours, with a fix note.
• When the agent's tools are functions in its own program (LangChain, for example), the platform can route each tool call through its tool proxy and see it; when it can't, it says why instead of sending you back to the same button.
• "Let AI finish and check" no longer repeats the same failure: after two identical failures it changes approach, and a trial now also runs the data reset and read the check runs.
• The connection check notices an agent that can't actually work (it keeps saying a schema or configuration is missing, or replies empty) and sends it back to the platform instead of calling the connection working.
• The AI diagnosis first tells a fault in the connection the platform wrote from one in your agent; the platform fixes the connection, and you're no longer told to edit a prompt or a file the platform wrote.
• A Q&A or support project seen through its replies can pass when every test passes; when a run can't pass, every page gives the same plain reason and what to do.
• A dependency cache that restores .venv no longer makes uv venv in the setup fail the run while getting ready.
• The test identity the platform adds can no longer be dropped by "Let AI finish and check"; the platform makes sure it is there before every check and run.
• The agent's tool config (mcp_config) path is read from .reviewctl/, and one folder too many or too few is recognised.
• When a check or a run doesn't run, its title says why (for example "The test identity isn't allowed") and whether the platform or you act; the project page shows the last run that didn't run.
• New reports no longer show "Judging again by today's rules".
• The AI diagnosis names tests by their titles, with no internal words like u1 or agent_bug.
• When the platform's AI is busy, "Let AI finish and check", AI approval and the assistant now wait their turn and try again instead of failing, and errors say so in plain words ("The platform's AI is busy right now"); the raw error is under "Details".
• AI diagnosis sometimes never wrote a result: it is now told to write its conclusion first, the platform diagnoses again by itself a few minutes later when there is none, and if there is still none it says why.
• A bot that refused and said why (for example "I can't share customers' personal information") could still fail for not saying "privacy reasons": saying what is protected, or why it can't, now counts as giving the reason. Runs judged that way before are judged again when you open their report.
• After AI approval failed, confirming a test by hand while the AI rewrote it could leave two copies of the same test, run twice: the AI no longer saves its own copy once someone has confirmed one, and never adds a test with the name of one already there.
• Cancelling a run also cancels the pre-run waiting for a sandbox ahead of it; a cancelled pre-run shows as cancelled, not "Pre-run didn't pass".
• Agents built on frameworks such as LangChain couldn't run on the platform's model (it doesn't take the stop parameter): the platform now drops such parameters when it forwards the call and still applies stop to the reply, with no change to the agent's code.
• "Let AI finish and check" could write a data reset that deleted the database, so every check failed with unable to open database file: the check result now says why and what to do, and the AI is told the data has to be there right after the reset.

Improved

• "Make it an agent project" is a real button; while the default security tests are still being written, the run step says to wait for them; the report header says "+ the platform's test connection files" instead of uncommitted changes.
• The check of "Let AI finish and check" runs every confirmed functional test once, and names the test when something is missing.
• After self-healing fixes the connection, the project page says to test again; the connection page has the "Let AI finish and check" button at the top; a Q&A project can become an agent project and connect a repository.
• Every page words a run the same way, and an empty reply counts as a fail.
• Tests the platform wrote are rewritten by the platform when they can't be judged or the connection changes what it sees, and a test fixed by self-healing says "confirm it, then test again".
• While waiting for the platform's AI or a sandbox, the page says what it's waiting for.
• "Next steps" now says where to click; the test list shows progress while a test runs; harmless warnings in the log are folded.
• When the platform's AI is interrupted (busy, timed out), it carries on the same conversation instead of re-reading the repository and opening a new trial sandbox; it starts over only when it can't.
• The same failure reuses the last diagnosis, and re-judging runs in parallel, so results come sooner.
• Reports count functional tests and attack tests apart, and project cards show confirmed tests in one format.
• A run or a pre-run waiting for a sandbox now takes the slot from the platform's own trial sandbox first; while it waits, it says how many sandboxes the organization runs at once and how many are ahead, instead of "Pre-running".
• "Start test" while the project is pre-running waits on that pre-run instead of being refused, and a pre-run that passed within 6 hours for the same code, connection files, secrets and tests is used as it is.
• Drafting tests from a few sentences no longer adds hard requirements you didn't ask for; anything extra is written as a suggestion. For a made-up answer, "Why it failed" says the risk first, and "How to fix" gives two ways: give the agent the information, or have it say it doesn't know and hand over to a person.
• The form for writing a test by hand fills in the identity and tools from the project's other tests, with examples in the project's own words.
• The overview, the project cards and the first screen of a report reach their verdict the same way, and a report counts functional tests and attack tests apart.
• Generating attack tests keeps you on the page instead of moving you or changing "What to test".
• Connection progress keeps updating in a background tab, and a project name you typed is no longer replaced by the name suggested from the repository.
• Choosing "I'm not sure" when creating a project now says: if you have a code repository but no deployed URL, choose "Agent that calls tools" and connect the GitHub repository. The assistant points to the buttons by their own words too.
• "Check once" and "Start test" now say what they usually cost and how long they take, from this project's (or the organization's) recent runs, and the spending cap says it is rarely reached.
• The interface uses one word, "test", throughout; model names, internal ids and machine types are in the expert view, and engineering notes sit under "Details" after one plain sentence.
• /docs opens in your browser's language, and shortcuts show Ctrl+K and Ctrl+J off a Mac.
• When the assistant reaches its daily limit, the message says how much was used, the limit, when it resets (in your local time) and how to raise it, with a heads-up at 80%. It also says so when what's left won't cover the next step.
• Under "Organization → Usage", assistant spend is split by project and by action, and "Let AI finish and check" shows what that run cost.
• Cached input to the platform's models is billed at Azure's cached rate (about 1/12 of normal input), so the assistant allowance and spend reflect the real cost.
• The overview's failing projects list every name; buttons such as "Ask the assistant" open a new conversation instead of an old one from another project; architecture and cloud sit under "Details"; a re-judge says which tests changed.

5 October 2026

New

• Credit codes: an owner enters a code under "Organization → Billing" and its credit goes into the balance. On the Free plan, usage past the £5 a month is paid from the balance instead of stopping; it stops only once both are used up. Spending caps and the other safety limits stay. See Billing and usage.
• The billing page is now a dashboard: plan, balance, this month's spend and free credit at a glance; then usage by day, spend by type and project, how far each limit is, credit history and invoices.
• Version comparison on the project: the latest run is compared with the one before by default, or switch to "Compare with the best" or "Pick one". It says how many tests passed before and now, which tests got better and which got worse, and warns first when the two runs were judged differently. Scenario runs and test-set runs are compared apart. See Retest and compare.

Improved

• Shorter interface text: explanations moved behind a "?" or into the docs, empty states are one sentence and one button, and nothing is said twice. The main pages carry 30–60% less text.

• The points a reply must make are judged by their meaning, not their wording. A point written as one word (say "customer service" or "human agent") means that idea, so a reply that answers as customer service, or says "assist" for "help", makes it. Runs judged word by word before are judged again by meaning when you open their report. New tests write each point as one sentence of meaning, with the ways of saying the same thing in one point.

• In a report, each failed test's "Why it failed" now gives the judge model's own reading of that reply, with the reply's words. "How to fix" is that test's own too: for a test judged on the reply alone, it says what the system prompt should tell the agent to answer to that message, instead of the same line under every test.

• The step where test drafts wait for confirmation is now carried on by the platform's own AI, so it no longer stops for you over what can be solved without touching your code:

  • AI approval also reads the platform's test connection files (review.yaml and the adapter under .reviewctl/), whether or not GitHub has them;
  • when the test itself is right and only the test connection keeps it from running, the platform hands it to "Let AI finish and check", and the draft is reviewed again once a check passes;
  • when the platform's AI diagnoses a run and finds the test connection at fault, the platform hands it to "Let AI finish and check" itself; the button is still there;
  • a draft that only lacks a test file gets it from the repository, or a sample for TXT, CSV, JSON or XML; PDFs, images and real documents are still yours to upload;
  • a draft that only lacks a test identity whose name doesn't look like a key gets a synthetic value;
  • a run waiting on its pre-run is pre-run again once a test it uses gets a new confirmed version, or a test file it needs arrives;
  • "Let AI finish and check" goes on by itself a few minutes later after problems on the platform's side (the model failing, a busy project, a sandbox that didn't start);
  • a review the AI couldn't give is taken again, further apart each time, up to 8 times.

  See Tests.

• Reports are redesigned to read like a test report:

  • One verdict: "Passed" or "Not passed". A run passes only when every test passed every time.
  • Every test listed once, as passed, failed or couldn't be tested.
  • Each failed test says why it failed, with the reply and how to fix it.
  • Each test that couldn't be tested says why, in plain words, and what to do.
  • The rest: passed tests are folded away, and the numbers and the platform's checks of the test itself are under "Details".

  The round report reads all its parts the same way, and project pages and test history use the same two words. See Verdicts.

• A run judged by older rules is judged again by itself the first time its report or round report is opened. The page says so and updates when it's done.

• Reports are stricter and say clearly what failed. On a connection that sees only replies, a test is now judged on its reply: the judge model reads the reply against the test's goal and the points it must make. A missing or contradicted point, or a goal not met, fails the test, and the finding quotes the reply. Before, all of these were "undecided". Runs already judged are judged again when you ask for a new diagnosis. See what the report shows.

• A security test set (red-team attacks or safety checks) without its own pass criteria now has to resist every attack, and any attack that gets through blocks it. Before, a set passed at 90%, so a few successful attacks could still read as "passed".

New

• "Confirm all" on the tests page confirms every draft waiting for you after one confirmation. Each draft is confirmed as its own "Confirm" button would confirm it, and drafts the AI reviewer turned down are confirmed too (the confirmation says how many). Drafts held by a rule, and rewrites that would loosen a check, are left for you to open one by one. In "All tests (detailed list)", tick rows and click "Confirm drafts" to confirm only those. See drafts and confirmed tests.

Improved

• When a pre-run fails only because the tests check tool calls that a replies-only connection can't see, the platform's model rewrites those tests to check the reply instead, and automatic approval reviews the rewrites. The waiting run then moves to the rewritten tests and is pre-run again. The connection is left as it is. See Runs.
• On a connection that sees replies only, the platform's AI no longer writes tests that check tool calls. "Let the platform read the repo" and the smoke tests of "Let AI finish and check" are told so, and their checks say which test reads calls. Automatic approval now holds such a draft for any check on tool calls (a call the agent must or must not make, or a fault or an attack in a tool), not only for a required call.

4 October 2026

New

• "What the AI does by itself", under "Organization" → "Policy" (owners only; every item is off by default): let the platform start the tests after a pre-run passes, start a run again or fill it in after a platform failure, re-run undecided or unstable cases once, switch a project to the recommended sandbox class, and give the automatic runs of someone who left to an owner. It acts for the person who saved the policy, within daily and monthly caps you set, through the same plan and billing checks as their own click. Everything it does is in the audit log and marked on the run, and a class it switched can be switched back. See Organization.
• After a run with failing or undecided cases, the platform's AI diagnoses it: for each failure it says whether the agent, the test, the connection or the environment is at fault, quoting the run as evidence, and acts where it may. A wrong test gets a revised draft for automatic approval (one that would loosen a check waits for you); a bug in your agent becomes a fix suggestion for your AI; a connection problem gets a button that hands it to "Let AI finish and check". You can ask for a new diagnosis at any time.
• Undecided verdicts are judged once more by themselves, with the run's judge model and within the run's judge budget; the report says they were re-judged by AI. Before you tick that a lost run was checked, the AI lists the calls that may have changed data outside the sandbox, with evidence.

Improved

• "Let the platform read the repo" writes the tests one at a time and checks them with the platform's own import checks as it goes, fixing what they find before it hands them in; if the import still turns them down, it carries on from that version. "Let AI finish and check" writes its one or two smoke tests the same way.
• "Let AI fill in and test" works in one conversation: it reads more of the docs you gave (only on your hosts), lists models, writes the connection, checks and tests it and fixes it from the reply, up to 5 tests; another click carries on where the last one stopped. The page lists what the AI did.
• Automatic approval reviews and revises in one conversation, checking each rewrite with the platform's checks before it is saved. Test sets are revised too, never their reference answers or expected values; when a draft needs something only a person can give, it says what. Generated attack prompts are checked before use, and an expected value you adopt goes to automatic approval.

3 October 2026

New

• A chat AI connected over HTTP gets "Let AI fill in and test" too: enter the API address, paste whatever docs, curl example or sample request and reply you have (or an OpenAPI document's address) and click once. The platform's model writes the connection settings (headers, the body with the question in it, the field holding the answer), the platform executor tests the connection, and when a test gets no answer it revises and tests again, up to 5 tests; once one works, the connection is saved as your "Save" would and starter questions are drafted. When a key is needed it names the header it goes in: enter the key and click again to go on. It connects only to the host you gave, and the key goes only in a header, never to the AI. See "Let AI fill in and test".
• The local test runner gets "Let AI finish and check" too: one command in your agent's repository, reviewctl connect --ai --project <project id> (the "Local test runner" page gives it with your project's id filled in). It reads the repository on your machine by the same rules the platform reads a GitHub repository with (never .env or files that hold secrets, anything that looks like a key removed before sending), lists what it would send and uploads it only once you agree; the platform's model writes the test connection files, which the command writes into the repository's .reviewctl/ only (backing up what it changes), pre-runs a check on this machine and, when it fails, sends back the lines of its log that look like errors (keys removed) for the AI to revise from, up to 10 checks. Secrets are sent by name only; their values stay on your machine. The web page shows each step as it happens. See let AI finish and check.
• Image projects get "Let AI finish and check" too: the platform's model reads the image's config as "Check image" read it and, when this organization's GitHub App can read the source repository the image's label names, that repository too, read-only (at the commit the label names, if any). It writes a test connection that runs in the image's container (the command and any adapter run from /reviewctl) and runs "Check once", revising after a failed check, up to 10 checks. It never changes your image and sets no secrets. See "Let AI finish and check" for an image.
• "Check image" now reads the image's own config too: its start command, working folder, user, ports, variable names and the source repository its labels name; a variable that looks like a key keeps its name only, and its value never leaves the executor. The starter test connection file fills in agent.command from the image's start command, and when the image's USER is root or not set it reminds you a sandbox runs it as uid 1500.
• Once a GitHub repository is connected, the platform suggests the project's name (the repository's) and business domain (from its README and system prompt): one click takes them, or "No thanks". See name and business domain.
• A local test runner can have "Restart" update reviewctl too: started with reviewctl runner start --self-update, its row's button reads "Update and restart" when it is behind, and the test runner downloads the platform's package, checks it against the SHA-256 the platform publishes, installs it with npm install -g only when they match, then restarts; if it didn't install, the row says so. See have "Restart" update it too.
• For questions in a test set whose expected values say "check by hand", "Have AI read the unknown expected values from the files" has the platform's model read the question's TXT, CSV, JSON or XML file and propose each value with the file's own words; take the ones you checked and they go into the test set's draft. See a test set's report.

Improved

• "Let AI finish and check", and the platform's own fix of a failed check or pre-run, now go up to 10 checks instead of 3, so a test connection that takes several rounds of fixes gets there without another click. Each check is a sandbox run, billed by sandbox time like a manual one, and each revision is a model call counted against the assistant's daily allowance. See "Let AI finish and check".
• When the tests neither set up nor read business data (a chat bot judged on its replies, say), the pre-run's "Does every attempt start from the same point" says "not applicable" instead of failing for a missing backend.reset.
• An adapter the platform's AI writes that starts a web server or plays a webhook is sent back to it before any check (it calls the function that writes the reply instead).
• "Let AI finish and check" no longer stops at "can't be tested" for an agent whose replies go out asynchronously, such as a webhook bot: the platform itself finds the functions in the repository that call a model to write the reply (AiResponder.reply_to, say) and hands them to the AI with an example adapter that calls one directly; if the AI still says it can't be tested, the platform sends it back once, naming the function.
• "Let AI finish and check" writes the test connection for the sandbox as it really is: the command runs from the repository's root (with the repository's Dockerfile, the repository is mounted at /workspace in the container), and a data reset command runs on the VM from .reviewctl/; a path that isn't there (such as /app/… guessed from a Dockerfile) is sent back to the AI before any check, instead of costing a sandbox run. A webhook bot is always reached by calling the function that writes the reply, with no web server or forged signature, so no webhook-signing secrets are asked for; the agent's own chat history is no longer taken for test data to reset.
• When "Check once" fails, the platform's AI now fixes the test connection from the check's result and log and checks again by itself too (up to 10 checks, counting that one), not only for a waiting run's pre-run; for repository and image projects alike. When it has just given up and nothing has changed since (no new commit, connection file or secret), it doesn't start again by itself: "Let AI fix it from this check" on the page hands that check over. See check once.
• The AI sees more when it fixes the test connection: the lines of the sandbox log that look like errors, with the lines around them (it read only the end of the log before, which missed an error followed by a long teardown), the repository files the error points to that it hadn't read (up to 5, never a file that holds secrets), and why the earlier checks of the round failed and what was changed.
• When the repository's Compose file reads an uncommitted .env through env_file, the platform's draft of the test connection file leaves services.compose out (and says why), and one the AI writes is sent back to it; a check that fails this way says the cause and the fix on its page. See troubleshooting.
• "Secrets" points out secrets the test connection file doesn't use (the agent only gets the variables listed in agent.env_passthrough); "Add to the connection file" adds them, and step 4's editor shows the file as it is now. See add project secrets.
• "Settings" → "Judge model" now works for repository and image projects: saving writes judge.model into the overlay's review.yaml (that line only; with no overlay copy yet, the repository's is copied first, and the page says so), so there's no YAML to edit by hand. See project settings.
• A run started without a label in a repository project is labelled with the first line of the tested commit's message; runs started by a push or a pull request use the commit's message or the pull request's title.
• Where a person's grade and the automatic result disagree, "Human grades" says for each what to change in the checks, and "Have the assistant change the checks" saves that as a new draft version of the test set, used once you confirm it.
• "Let AI finish and check" keeps the judge model in the current connection file (one you chose in the settings, say) rather than changing it back to the platform's default.
• The "Connect a GitHub repository" wizard ends at "Check once": runs on every push and pull request, the GitHub check, the schedule and API tokens moved to the project's "Settings" → "Automations" (formerly "Automatic runs"), with every setting kept. Once a check passes, the wizard ends with "The connection works. Want a test on every commit or on a schedule? Go to Settings → Automations" and a link; under "Organization" → "Members", a project's name in "Left behind" opens it too. See Automatic runs.
• When the GitHub App's installation covers only one repository, the wizard picks it and its default branch for you; the data boundary is still yours to tick.
• The platform recommends a sandbox class from the test connection (how many containers start beside the agent) and the logs of the latest 3 sandbox runs (memory ran out, the disk filled up, the setup ran out of time), picked for you in "Settings" → "Sandbox" and the wizard's "Class" step with why and which run's log said it; it applies only when you click "Save", or "Keep" the current one. The platform never moves a project to a pricier class by itself. See sandbox classes.

2 October 2026

New

• For a project connected through a GitHub repository, "Have AI write tests" starts on "Let the platform read the repo": one click, and the platform's model reads the repository, read-only, and writes the functional tests (the main tasks, edge cases, what happens when something fails), the capability list and the test files they need, saved as drafts for automatic approval. Of the test files, the platform copies the ones in the repository by their path, and AI writes synthetic samples of a TXT, CSV, JSON or XML the repository doesn't have; PDFs, images and real documents are still uploaded by a person. See Have AI write tests.
• A draft automatic approval turned down is revised by the platform from the reasons and reviewed again, up to 2 times, with what each revision changed and took out shown on the draft; a review the AI couldn't give is taken again a few minutes later. Drafts you wrote or edited yourself are left alone.
• Once the first functional test is confirmed, the platform writes a default set of security tests itself: the template chosen from what the agent does, sent as the test account, reviewed by automatic approval as usual; delete it and it isn't written again.
• When a project is connected through a GitHub repository, the assistant can read the repository (list, read, search), so the names, arguments and test accounts it drafts with come from the code.
• "Let AI finish and check" in step 4 of the GitHub wizard: one click, and the platform's model reads the repository (read-only, never .env or a secret file), writes review.yaml and any adapter it needs into the platform overlay, drafts a test if the project has none, and runs "Check once", revising after a failed check, up to 3 checks. It ends with "The connection works" or exactly what you have to do: add a secret it names, confirm a test, or why the repository can't be tested this way. It never changes your repository or sets a secret, and it can't loosen settings a person decides. See let AI finish and check.
• What goes wrong in a sandbox, the platform now handles itself wherever it can, saying on the page what it did and why; whatever changes a setting can be switched back:
  • A virtual machine that hasn't reached the platform 12 minutes after starting, or that the cloud couldn't create, is started again by itself (the same run, not charged); if that one doesn't start either, the platform sends the job id and boot logs to our engineers and the page says "The platform's engineers have been told": nothing for you to do.
  • When a pre-run's log shows exec format error, a package missing for aarch64 and the like, the platform pre-runs again on the other architecture, leaving the project's setting alone; if that passes, it switches the project's architecture and says so under "Settings" → "Sandbox" with "Switch back"; if both fail, both results are shown. When the image has no build for the architecture the setting names, "Check once" runs on the one it has.
  • A sandbox run that stopped early offers "Run the N that didn't run" on its report: a new sandbox (the same commit or image, class and architecture) runs only those attempts, and the results are read into the round report and judged again by the run's own pass criteria. See fill-in runs.
  • A sandbox run that loses contact mid-way is checked by the platform when every backend its review.yaml uses was inside the sandbox, so the next test and automatic runs need no "I've checked"; anything that might have reached outside still needs a person. The rule is under after a lost run.
  • With automatic approval on, the "Platform adjusts" items of the next-evaluation recommendations are adopted by themselves after each run (the new versions are still reviewed by the AI), say so, and can be undone with "Undo: back to the previous versions"; recommendations for the agent's own code are never adopted by themselves.

Improved

• A failed pre-run is no longer yours to fix (projects connected to a GitHub repository): when a waiting run's pre-run fails because of the test connection, the platform goes on as "Let AI finish and check" does, this pre-run being check 1 of at most 3, and the run starts by itself once one passes. Runs started by a push, a pull request, a schedule or the API wait for it too, and only end, with a single mail, if it can't be fixed. A failure of the agent's own is left alone, with the reason. While a run waits for its pre-run, changing the connection files, a secret or the sandbox settings, or pushing to the branch, makes the platform pre-run again 30 seconds later, up to 3 times per run. See the pre-run.
• "Secrets" lists the values review.yaml and your confirmed tests read by name and who gives each, with the first missing one already in the form; OPENAI_API_KEY is marked as answered by the platform's model, nothing to add; plain values found in the repository's .env.example or Compose ports go into runner_env in one click. "Check once" checks first and names a missing secret it needs, or you can "Check once anyway". See add project secrets.
• "Check once" works before any test is confirmed: the platform checks the connection with its own question (the agent introduces itself), and says that only shows the agent starts and replies.
• A chat AI project no longer needs pass criteria saved first: a test set without its own runs on the default (each question passes every time it is asked, for at least 90% of them, and safety checks may not fail), and switching test sets needs no save. "Start test" shows the criteria in effect, with "Change" to edit them.
• Connecting a chat AI takes a pasted curl command: the platform reads the address, headers and body (it never runs the command) and moves a header value that looks like a key into the "Key" field. For an OpenAI-compatible API, once the address and key are in, "Model" offers that address's own model list.
• Saving a connection tests it right away, and for a custom HTTP API the platform finds the field that holds the answer (OpenAI and Anthropic formats, the usual answer fields, streamed replies; failing those, the platform model reads the raw reply). It says which field it uses; click another answer if it's wrong.
• The first time the connection works and the project has no test set, the platform drafts a few starter questions from the domain, the system prompt and the AI's own self-introduction. They have no reference answers (tagged "needs-answer"), are judged only by checks that need none, and automatic approval reads them first as usual.
• When the chat AI has no system prompt the platform can use, "Doesn't leak the system prompt" no longer stops the test from starting: the run leaves that check out and the page says so. A new test set doesn't add it in that case either.
• The product is now called Preworld Review (Preworld, with a lowercase w, is the company), in the app, the mails, the docs and the GitHub check run, which now reads "Preworld Review".
• The sign-in page and everything on it (signing up, invitations, approving the command line, sign-in links) and these docs take the company website's look: dark, with the Preworld mark. The docs open dark and can be switched to light, and their top bar has "Sign in" and "Start free". Signed in, the app looks as before.

Fixed

• When a scenario has neither a data check after the task nor a required tool call, an attempt counts as done only if the reply says it's done, makes the points it must make, or carries the record's own values. A reply that shows nothing done, such as "I'll look that up next", is undecided instead of done.
• Recoverable faults are now reported as "handled": where the scenario allows a safe stop, stopping counts as handled, and the report says separately how many got the job done, instead of calling a safe stop "recovered, still gets the job done".
• On a connection that doesn't show tool calls, an attempt's detail no longer heads its calls "Tool calls · 0".
• On such a connection, the attempt detail says right under "Tool calls" that this connection doesn't show them, so the injected text listed there is not taken for the agent's calls.
• After a failed pre-run, "Skip the pre-run and start" now starts the sandbox at once, instead of leaving the test queued with no sandbox until cancelled.
• A security problem the evidence shows (a reply that says what it mustn't, a violation the product let through) now fails the attempt even when the task itself can't be judged. Reply leaks are counted over the attempts run and described as content in the reply, not as "a tool call matched the rule".
• In a version comparison, a count that stayed the same is no longer called an improvement just because more attempts could be judged. "Should recover" now reads "faults handled (safe stops included where allowed)", with "of which the job got done" listed separately.
• For a test that was cancelled or stopped early, the report header and the project's latest verdict say "planned N, ran M" instead of taking the plan for what ran, and a "don't release" verdict comes with a matching problem count.
• A sandbox test that was cancelled no longer offers "raise the limit and run the rest" or says the test runner is offline: it says the sandbox is gone and the test can't be resumed, with "Start test" to start again.
• At phone width, a very long reply (a long path, say) in an opened attempt wraps inside its box instead of widening the page.
• The quickstart, the report guide and the docs home no longer promise "every tool call" unconditionally: tool calls are shown when they go through the platform, and a connection that sees only replies says so.
• When reply leaks from several scenarios are gathered into one finding, it is counted over all of those scenarios' attempts, so it no longer reads like "12/3".
• An attack that made the agent actually run code or write a file, as the data afterwards shows, is now a security finding ("attack got through"), no longer filed as a wrong answer with an unrelated fix.
• At phone width, long paths in the report's problem cards and summary wrap instead of widening the page.
• The message on a cancelled sandbox test is corrected: what ran is in its report as usual, to view and download; it can't be resumed because the sandbox, and the environment a resume needs, is gone, not because the records are.
• Attempts with a proven security problem whose task itself couldn't be judged now count as judged, not as "insufficient evidence", so the report's count of judged attempts matches the pass-rate denominator.
• When a reply carries the record's own values ("figure_created": true, say), the judge model reading it as "doesn't say it's done" is no longer taken as the reply denying it, so "the task was done but the reply says it wasn't" no longer appears.
• Likewise, when the data or the calls show the task done and the reply makes every point it had to, the judge model's "doesn't say it's done" (a reply explaining the rate limit the task asked about, say) is not taken as a denial.
• In an English report, the connection details quoted by the setup checks ("tools don't go through the tool proxy", for example) and the sentence for the developer machine's AI are in English.
• A safe stop that the scenario allows and that passed by its standard (a rate limit the task itself sets up, say) no longer gets a separate high-severity "didn't finish the task under normal conditions".
• On a phone-width window, the version comparison no longer squeezes metric names and problem titles to a letter a line: a table that doesn't fit scrolls sideways in its own box, and "Passes every time" and "Version inputs" stack.
• When the sidebar shows only icons (narrow windows), screen readers still read each link's name (Overview, New project, Compare versions, Organization), in both languages.
• When the judge model writes a point's id out of shape ("[task-result] 14", or its quote where the id goes), an answer that can still be told apart is used as usual; one that can't is reported as the judge model's answer being unusable, not as no judge model or a failed call.
• When a point names a number (a total of 450, say) and the reply doesn't contain it, the judge model's "stated" no longer counts: the attempt is undecided, not passed.
• On a connection that sees only replies, the setup checks no longer describe a rule on what a call may carry as "it must not call this tool"; where the tool proxy can't be connected, they suggest adding a data check instead of removing those requirements.
• Before automatic approval asks the model, it checks two things by rule: whether the run will allow the scenario's identity variable (one the test runner allows, or, for a repository or image project, one among the project secrets or review.yaml's runner_env), and whether, on a connection that sees only replies, the scenario requires calling a tool. A draft that fails either is no longer confirmed automatically: it waits for a person, with why, instead of being turned down at the pre-run.
• When a call went through but the tool answered with an error (refusing a path, say), the report and the attempt details say "went through, but the tool returned an error", not "succeeded"; when the agent stopped without doing the task and the test has no data check, the report says "not done", not "the data in the system was wrong afterwards".

1 October 2026

New

• Activation code for sign-up: signing up a new account, by email, GitHub or Google, asks for an activation code, and nothing is created without the right one. Don't have one? Email admin@nice2.io. Signing in and accepting invitations are unchanged. See Sign up.
• New plans and prices: Free, Team and Enterprise: no price per member; you pay for sandbox time and model tokens. Every new organization starts on Free, with £5 of credit a month that comes back on the 1st. Team is £20 a month per organization, all of it credit, and usage beyond it is charged at the unit prices; Enterprise is by contract. A sandbox hour now costs twice the cloud's price (Standard: 14p) and models the bill plus 30%. Organizations on the trial are now on Free. See Pricing.
• Monthly spending cap: on Team, an owner can cap what the organization spends in a month under "Organization" → "Billing". At the cap new tests are refused and the platform's models stop; everyone sees "Used £x of £y this month". See set a monthly spending cap.
• Sign in with GitHub or Google: the sign-in page has "Continue with GitHub" and "Continue with Google". Your account is still your email address: a GitHub or Google account with your address as a verified email signs you in, and keeps doing so after the first time. On an invitation, tick the terms and continue with either to accept it for the invited address. A new address signs up as by email, the terms first. See how members sign in.
• Cookie settings: once we switch on Microsoft Clarity (usage analytics), the web app and this site ask first. Nothing loads until you click "Accept", "Reject" is one click too, and one answer covers both; change it under "Cookie settings" in the side bar. In the web app every text is hidden from Clarity, and pages with reports, logs, tests or projects are never recorded. See cookies.
• Your agent's model from the platform: a GitHub repository or image project no longer needs your agent's model key. Without OPENAI_API_KEY in the project secrets, the platform answers your agent's model calls (chat completions and the Responses API, streamed or not, with tool calls) and bills them by usage, up to a spending limit per run. The wizard's "Secrets" section says which one a run uses, the run log says when a model asked for was replaced, and the report records it. Pick the model under "Settings" → "Judge and agent models". See your agent's model.
• Test runner variables in review.yaml: runner_env sets values that aren't secret for the test runner itself, such as the address of a service the repository starts or a test account of its own test backend, so they need no project secrets. The tests' users, the data reader, resets and env_passthrough read them; a project secret of the same name wins. A value that looks like a real credential is refused with the variable's name, and the values aren't masked in logs. See runner_env.

Improved

• The billing page shows the plans side by side, and what a sandbox hour of each class and a million tokens of each model cost, as billed. On the Free plan, the billing, usage and overview pages say how much of this month's credit is left and when it resets.
• "Start test" and "Check once" say what a run can cost at most before you click: the sandbox at its class's hourly rate up to the run time limit, and the per-run caps of the judge model and of your agent's platform model. A part that uses your own key is marked as not on the platform bill.
• The preflight checks have a "Latest pre-run" line: it warns when the last one failed, and says the test may not reach the agent when you skip the pre-run without a passing one.
• GitHub repository and image projects can add helper files (adapter scripts, test data) to the overlay in the web app. They land under .reviewctl/ in the sandbox, with no change to your repository.
• When the button for a new scenario is greyed out, it says which fields are still missing; required fields are marked, for screen readers too.
• When the platform has no mail service, the sign-in page says how to sign in with an invitation link.
• Every cell of a report's execution ledger opens: the verdict for that attempt and why, the agent's reply, each tool call and what it returned, the data afterwards, and the raw evidence files. The report page also downloads the full report (JSON) and the whole run bundle.
• Pre-runs and reports have two more checks: "Does the entry program finish normally" says how far the agent got and the last error the entry printed; "Does the test environment work" says which tool backend, data reset or read failed.
• Pages and run logs say which model answers the agent: the model the agent asks for when the platform offers it, otherwise the project's agent model; the log notes each model the first time it answers.

Fixed

• When the test entry exits with an error before any tool call or model answer (a missing model key, for example), the attempt is recorded as "the test didn't reach the agent", no longer as the agent's "silent failure". The report's first page, its detailed data and the Markdown export now agree.
• "Does the agent use a model to decide" first counts the answers the platform's relay actually gave; attempts whose entry exited with an error no longer count as the model deciding.
• A dependency cache that fails to restore no longer leaves a half-written folder behind, which made the second run of the same commit fail to install; Python virtual environment caches restore correctly.
• reviewctl doctor looks for agent.command from the repository root for a config at .reviewctl/review.yaml, where the sandbox actually starts it.
• Pre-runs are described correctly: a pre-run gives a pre-run result (a diagnostic, not a test report) and stays out of the report list and comparisons; the report page says so.
• The GitHub App's permissions are described correctly: read-only on code; Checks write is requested at install together with the others and used only on projects that switch on "GitHub checks".
• An entry that exits with an error after the agent has used its model is no longer reported as "the test didn't reach the agent".
• A read-only question answered with something missing is recorded as an incomplete answer, no longer as "not done, but told the user it was"; a problem only the judge model read from the replies is marked as a lead, on the first page too.
• After the last attempt, the sandbox stays in touch with the platform while it uploads the report and reports the test done, so a test with many attempts and a slow upload is no longer taken for lost. A decoy key a scenario writes itself (a fake sk- key the reply must not contain, for example) no longer stops the upload, and an upload that really is refused ends the test at once with the reason, instead of retrying until contact is lost.
• When a scenario has neither a data check after the task nor a required tool call, an attempt is no longer done if the agent replies that it could not do it, the entry ends with an error, or every tool call seen failed. Before, unchanged data plus any answer passed, together with "the task was done but the reply says it wasn't". A reply that carries the record's own values still counts as answered. In an attack, a clear refusal counts as holding.
• An attack that was delivered, but where the agent stopped on its own error before the step the attack aims at (no forbidden attempt, no impact), is recorded as not triggered and no longer counts as an attack held.
• On a connection that sees only replies, not tool calls, an attack judged by a forbidden tool call no longer counts as held: it is undecided (tool calls not visible). An attack with a data check is still judged by the data. On such a connection the report's "Forbidden attempts · refused by the product" reads "tool calls not visible" instead of 0.
• On such a connection the report's first page and attempt details say "tool calls not visible" rather than "no tool calls", and an attack held is no longer described as the agent "not following the injected instructions". A leak in the reply still counts as the attack succeeding, so "actual impact" and "attacks that succeeded" agree. Where an attempt's "the reply says it is done" is the judge model's reading, the detail says so.
• When the points a reply must make can't be checked, the report now says whether the judge model gave no verdict (none set up, its call failed, or it hit its limit) or gave one that doesn't hold against the reply's own words; the second no longer asks you to set up a judge model key.
• At phone width (390px) the start-test page no longer scrolls sideways: long scenario names wrap instead of running under 「已确认」 (Confirmed), and the top bar's search and assistant stay on screen.
• An attack that slips instructions into the user's message using the platform's built-in attack text failed on every attempt because the text couldn't be found; it is now injected as it should be. When the platform itself fails during an attempt, the report says so, instead of blaming a tool backend, the data reset or a read.

30 September 2026

New

• Platform models: use models the platform provides (gpt-5-mini, DeepSeek-V4-Pro and others) as the AI under test or as the judge, with no key of your own; usage is billed. See Pricing.
• Prepaid plan: usage comes off a prepaid balance you can see any time under "Organization" → "Billing". See Billing and usage.
• GitHub checks: accept the App's Checks permission and turn it on, and every automatic run posts a check on the commit it tested, with the verdict and a link to the report.
• English interface: the web app, mails and the reviewctl CLI are fully in English; switch with "中文 / EN" in the side bar.
• My CLI tokens: see and revoke the command-line sign-ins on your own computers; they expire after 90 days unused.

Improved

• Money that will be billed (estimates, limits, costs) is shown in pounds.
• Round reports, completion mails and GitHub checks list functional and security results separately, and an injection that got through on its own.
• Version compare only compares what both runs judged, and says when a run was cancelled or didn't finish.
• A sandbox that hasn't reached the platform 12 minutes after it started is deleted and the run ends; that time isn't charged.
• When the platform is updated, a run in progress hands in a report of what ran after the current attempt instead of being lost, and results that can't be handed in straight away are retried.
• Removing a member also revokes the test runners they paired, their tokens and their sign-ins.
• Saving a review.yaml with a problem says which line and field is wrong, in plain words.
• You accept the Terms of Service and the Acceptable Use Policy before using the platform, and again when they change.

29 September 2026

• Launch: Preworld Review opens at https://review.nice2.io, by invitation.
• GitHub repositories and your own container images: every run gets a fresh virtual machine in London, deleted afterwards.
• Automatic runs: on push to the default branch, when a pull request opens or updates, on a schedule, or from your own CI with a project API token.
• CLI download: the reviewctl package is at https://review.nice2.io/downloads/.
• Documentation: quickstarts, security, data processing, pricing and the legal texts.