Release gate

The release gate turns your rules into one verdict per experiment: "Passed the release gate", "Failed the release gate", or "No release gate set". It is worked out on every experiment, kept up to date while a run goes on and when a run is re-judged, and shows wherever someone decides whether to ship.

Set the rules

Open Settings → Release gate. Each rule is one line; "When it fails" makes it "Blocking" (the version can't ship) or "Warning only". Add one with "Add a rule", then "Save gate". The page previews the verdict on the latest experiment as you edit ("Preview: …"), and saving works the recent experiments' gates out again under the new rules.

Rule	What it checks	New rule starts as
"Scorer threshold"	A scorer's mean over the rows it decided, or the number of rows above 0 (for lower-is-better checks such as "Attack succeeded"), compared with ≥, ≤ or = a value. Any scorer: built-in, the project's, or one scored on your side in the SDK.	mean ≥ 0.8, blocking
"Test verdict is Passed"	The sandbox or test-runner run's own verdict, with all its trust checks (the pre-run, attacks that never reached the agent, the second judge, the platform's own faults). Only for experiments from runs; always blocking; at most one.	blocking
"No regression against the baseline"	How many rows got worse than the same row in the baseline on any scorer both have ("Rows allowed to regress"). Skipped when there is no baseline.	0 rows, warning only
"Undecided at most"	The share of rows that ran and stayed undecided. Undecided rows the platform caused itself (its own failure, a lost sandbox, the platform model or judge failing, the judge's cap) are left out.	10%, warning only
"Tags that must run"	Every tag listed (comma-separated) has at least one row that ran, so "passed" can't mean the security rows were left out.	security, blocking

A gate passes when every blocking rule holds. It is blocked when a blocking rule fails, or can't be told (all its rows undecided, say). Rules that don't apply to an experiment (a test verdict on an SDK experiment, no regression without a baseline, a scorer the experiment neither has nor names, like "Attack succeeded" on a run of functional tests only) are skipped; when none applies, the gate says there is no rule to check. Warning-only rules that fail are listed but never block.

A count held under its line ("Attack succeeded = 0", or "≤ N") can't be told while rows the scorer left undecided could take it over the line: the rule shows "? Not sure" with what it got and how many more are undecided ("0 rows, 10 more undecided"), never a tick, and holds the gate like any rule that can't be told. "See N rows" opens the undecided ones; the experiment page, the GitHub check and the pull request comment say the same.

Low-severity problems only warn

A "Scorer threshold" rule can tick "Low-severity hits only warn; they don't count toward this rule" (it matters for "Attack succeeded"; low_risk: "warn" in the API). With it ticked, a row whose attack got through only at low severity is left out of the rule, the gate is decided as usual, and a warning is added:

• Low severity means the attack "got through" only by saying internal details, with no one's data, no secret, no system-prompt text and no action (naming its own tools and how to fill their parameters, say); or it rested on a scenario the platform assumed for the test ("this is the platform's assumption": another user on a product that keeps no users apart). The report still lists them as "Low" problems.
• Medium, high and critical hits count as before, and still fail the gate.
• A gate that passes with low-severity problems reads "Release gate passed, with N low-severity problems: … Not a serious hole: whether to ship is your call", never "failed"; the rule's line says "N more low-severity rows, warning only".

The default "Attack succeeded = 0" rule has it ticked (see below). Rules a person saved stay as they were saved: without the tick, a low-severity hit still fails the gate.

The same goes for a sandbox or test-runner run's own verdict ("Test verdict is Passed"): the platform's default pass criteria for a security test set only warn of low-severity hits, and the run reads "Passed, with low-severity problems"; pass criteria you set yourself under "Settings → Pass criteria" still count every hit.

The default rules

Until someone saves rules, the project has default ones written from what it has, and kept in step with it:

• "Test verdict is Passed" (blocking), when the project has sandbox or test-runner tests;
• task_done mean ≥ the lowest pass line of the saved pass criteria's key tasks (blocking), when there are key tasks;
• attack_success rows above 0 = 0 (blocking, low-severity hits only warn), when it has security tests;
• "No regression against the baseline", 0 rows (warning only);
• the key tasks' scenarios as "Tags that must run" (blocking), when there are key tasks.

Once a person saves, the rules are theirs, and the platform never changes them.

Where it shows

Where	What
An experiment	The banner at the top: the verdict, each rule's actual value with ✓ or ✗, "See N rows" for the rows that broke a rule, "Edit gate". With low-severity problems, the line under the verdict says "N low-severity problems: … whether to ship is your call" in the warning colour.
The experiments list	A chip per experiment: "Can ship", "Failed" or "None"; "Can ship · low-severity" when it passed with low-severity problems (hover to see which); "Scoring" while the platform's scorers are still scoring an SDK experiment's rows (the gate is decided once they are done).
Overview	The first block: "v13 can ship" or "v13 failed the release gate" ("v13 can ship, with low-severity problems"), and whether that changed from the version before.
GitHub	The check on a push or pull request round concludes success (passed and the round finished), failure (blocked) or neutral (no rules, or the round didn't finish and nothing broke for sure), with the scores, the counts and the top problems. With low-severity problems the check is still success, and its title and the table say "Release gate passed, with N low-severity problems …". On a pull request, the same table is posted as one comment, updated each round, when the App's installation lets it write pull requests. See automatic runs.
CI	reviewctl eval --launch, the GitHub Actions and the SDK's preworld eval exit 0 (passed), 1 (blocked) or 2 (not judged). With low-severity problems it is still 0, and a warning line is printed (a warning annotation on GitHub Actions, warning in --json). See CI and the GitHub Action.
API	GET /v2/experiments/<id>/gate, with ?wait= to long-poll until it's final. See v2 routes.

Examples

• Ship only if quality holds and security never fails: "Scorer threshold" correctness mean ≥ 0.85 (blocking); attack_success rows above 0 = 0 (blocking); "Tags that must run" security (blocking).
• Catch regressions on a pull request: "No regression against the baseline", 0 rows, blocking, with the project baseline set to the last release.
• Keep the platform honest: "Undecided at most" 5% (blocking), so a run that couldn't judge a tenth of its rows doesn't pass.

Next steps

• Experiments: the rows behind each rule.
• CI and the GitHub Action: block a merge on it.
• Full run reports: how a sandbox run reaches "Passed".