API: tokens and errors

The API lets your code upload experiments and traces, your CI start a run and wait for its release gate, and your scripts read what the web app shows. This page covers tokens, how requests are signed in, the /v1 routes CI uses, and how the API answers errors. Every /v2 route (datasets, scorers, experiments, the gate, traces, online scoring, alerts, the Playground, human review) is in v2 routes; the OTLP and Gateway endpoints are in OpenTelemetry and the AI Gateway.

Routes are under https://<platform address>/v1/ and /v2/, take and return JSON, and need a token in the Authorization: Bearer <token> header.

Tokens

Token	Starts with	How you get it	What it can do
Project API token	pj_	The project's "Automations" → "API tokens"	Start runs of that one project and read its jobs and runs; read its datasets, scorers, experiments, gate and traces. With "Allow uploading logs and experiments" ticked (the ingest scope), also create datasets, rows, versions and experiments, upload experiment rows and traces, and call the OTLP and Gateway endpoints. It can never change scorers, the release gate, online scoring rules or alerts, rename a dataset or delete its rows, or use the Playground or human review.
CLI token	rt_	reviewctl login (Command line)	Read what you can read in the web app, and upload runs. It can't start runs.
Dev AI connection token	dl_	reviewctl dev connect trades a pairing code for it and saves it per repository under ~/.reviewctl/dev-links/; or, choosing MCP over HTTP in "Settings → Dev AI", the page shows it once	Call the dev AI routes (/v1/dev/*) only, in the one project it was made for, as far as the connection's permissions go. Lasts 30 days.
Test runner token	rn_	reviewctl runner register trades a pairing code for it and saves it in ~/.reviewctl/runners/<project id>.json	Only for the local test runner itself: claiming jobs, heartbeats and results. No other route.

Each acts as the person who made (or paired) it. Runs started with a token are that person's, billed like runs they start by hand. A token stops working when it is revoked or when that person leaves the organization. A project API token also stops working when its project is deleted.

Create a project API token

1. Open the project, then "Automations".
2. Under "API tokens", enter a "Name" (for example GitHub Actions). Tick "Allow uploading logs and experiments" for a token the SDK, OpenTelemetry or the AI Gateway will use. Click "Create token".
3. Copy the token with "Copy token". It is shown only this once; the platform keeps only its hash.

The page also shows a ready curl command with your project's id and, for an uploading token, the OTLP and Gateway addresses. The list shows each token's name, what it can do, who made it, when, and when it was last used. To revoke one, click the bin beside it. Only a person signed in in the browser can create or revoke tokens. Tokens made before v2 keep what they could do; to upload, make a new one.

How a request is signed in

• Tokens: Authorization: Bearer <token>. The OTLP and Gateway endpoints also take x-api-key: <token>, as Anthropic's SDKs send it.
• The browser: the session cookie. A browser request that changes something must come from the platform's own pages (Sec-Fetch-Site: same-origin, or an Origin of the platform); one from another site answers 403 cross_site. Requests from CI, scripts and servers send neither header and aren't affected.
• A project token works only for its own project: another project's route answers 403 project_token_not_allowed. A token without the upload scope answers 403 token_scope_missing on an upload. An object in another organization, or a deleted project, answers 404 not_found, never 403.

Start a run from CI

POST /v1/projects/<project id>/jobs

curl -X POST https://<platform address>/v1/projects/prj_01J8ZK3Q9MXYZ0000000000000/jobs \
  -H "Authorization: Bearer $REVIEW_PROJECT_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" \
  -d '{"commit": "4f2c9e1b7a0d3c5e8f6a1b2c3d4e5f6a7b8c9d0e"}'

It starts a round of every confirmed test, as "Test new version" → "Start test" does, and the round becomes one experiment. A pre-run comes first, and the run starts once the pre-run passes (a chat AI's test sets go to it directly; when each scenario runs once and has only its normal condition, the run itself is the pre-run). All fields are optional:

Field	Type	What it does
commit	The full 40-character sha	A GitHub repository project: test this commit instead of the head of the bound branch.
image_ref	Image reference	An image project: test this image, read and pinned by its digest for this run only.
scope	all, functional, security or failed	Which confirmed tests to run: all (default), the functional or the security ones, or those whose last result didn't pass.
repeats	1 to 5	Times to run each test. Default: what the project's pass criteria need.
notify	true or false	Email the token's maker when the run ends. Default true.

Idempotency-Key (8 to 128 characters) is optional: a retry with the same key gets the first launch back instead of starting another.

The answer is 201 for a new launch, 200 for a repeated key:

{
  "created": true,
  "job": {
    "id": "job_…",
    "project_id": "prj_…",
    "status": "waiting_trial",
    "trigger": "api",
    "commit_sha": "4f2c9e1b…",
    "batch_id": "…",
    "run_id": null,
    "report_url": null
  },
  "experiment_id": "exp_…",
  "gate_url": "https://<platform address>/v2/experiments/exp_…/gate"
}

experiment_id is the experiment the round becomes, named at launch (its page is at /p/<project id>/experiments/<experiment id> once the round's first run is in); gate_url is its release gate, to wait on (see wait for the release gate).

A project whose connection is a test runner needs it online. A project connected in the browser to a chat AI sends every confirmed test set to that chat AI. With commit on a project that isn't a repository project, or image_ref on one that isn't an image project, the call is refused.

Read a job and its report

Route	What it returns
GET /v1/jobs/<job id>	The job: status, error_code and error, run_id and report_url once its run is uploaded, trial (its pre-run) and batch (the jobs started with it). A sandbox job (and its trial) also has log_last_at and cancel_by. waived says why the platform charges nothing for the job: auto_fill is the platform's fill-in run after a sandbox was lost part way; model_retest a retest of attempts only the platform's model left untested; otherwise null. A fill-in job's fill_of is the run it finishes; with fill_mode undecided it retests only what that run couldn't test (see tests that couldn't be tested).
GET /v1/runs/<run id>	The run: gate (the verdict), trust, status, finding_count, blocker_count, key_missing, the times and report_url.
GET /v1/projects/<project id>/runs	The project's runs, newest first, each as above. ?limit= up to 200 (default 50); ?pack=<test id> for the runs of one test.

A job's status goes from waiting_trial (waiting for its pre-run) or waiting (waiting its turn behind jobs started with it) to queued, claimed and running, and ends as succeeded, failed, cancelled or lost. When a pre-run of a run started from the API doesn't pass, the run ends as failed with error_code trial_failed and the reason in error, so the next call can start. A run that is what its pre-run would run, with no separate pre-run, starts as queued with self_check true, its own pre-run result in trial_result; when its tests didn't reach the agent, it goes back to waiting_trial (or ends with trial_failed as above), and its trial is the run it was, with self_check true too. When a sandbox's preparation stalls, the job ends as failed with error_code prepare_stalled (a step printed nothing for 15 minutes) or prepare_timeout (not prepared as the run's time limit came near), and error names the step and the last line it printed; see preparation that stalls.

For a sandbox job, log_last_at is when the sandbox last sent a log line, null before it has sent any. cancel_by is the deadline once a cancel was requested: if the sandbox hasn't stopped by then, the platform deletes it; null when no cancel was requested. Jobs on a test runner or the platform executor don't have these fields.

A run's gate takes one of four values. passed_in_scope and passed_reply_level are both shown as "Passed"; with the second, only replies were checked. blocked means a test failed, shown as "Not passed". incomplete means there's no full verdict yet: a test couldn't be decided or didn't run, the connection saw too little, or a key task of the pass criteria wasn't selected this time; pages show "Not passed (…)" when the connection saw too little, and "Couldn't be judged" otherwise. key_missing is how many key tasks of the pass criteria have no result at all in this run, that is, weren't selected. A trust of untrusted means the test never reached the agent ("Didn't run"), whatever the verdict says. In GET /v1/projects/<project id>/compare, which the project page's version comparison uses, each run's verdict is passed, failed or incomplete ("Couldn't be judged"), as on the report's first screen.

In the whole report JSON (see below), each undecided attempt gives its specific cause in verdict.open (the judge model's call failed, two judge models read it differently, the tool the attack goes through failed before it could act, and so on), an attempt the platform re-ran has verdict.rerun, and the run's re-runs are listed in reruns. A re-run caused by the platform's own problem has platform in both (model_relay: the platform model failed; fixture: the platform's own test backend failed), and in reruns also spans, when each try started and finished. How often the second judge model read again and how many it settled is in verdict.second_judge; the checks the two read differently and that were settled by strength are in verdict.second_judge.split, each with both readings (readings), whose reading decided (by), and why: stronger, or the platform check that overturned the stronger one (number, denial). The second judge model's usage is in usage.judge.second. On a test runner that judges several attempts at once, usage.judge.concurrency has how many judge calls may be out at once (n), how many went out with 1, 2, … calls out (let_out), and each stretch the judge went one call at a time (one_at_a_time: from, to, null when it lasted to the end, and why: rate_limit, unavailable, timeout or near_cap, with the status that said so). A near_cap stretch sends as many calls at once as the job's spending limit lets out: one more beside the calls out while those fit under it at their most (since 5 November 2026 the new call's own most isn't counted). at_most says how many, 1 when even the one call out doesn't fit under it at its most (older reports leave it out there); it ends when more fit again. See Reports.

A round with both scenarios and test sets is several jobs: one for the scenarios and one per test set, each with its own run. Follow every job in batch. The round's report is at https://<platform address>/round/<batch_id>.

Wait for the release gate

GET /v2/experiments/<experiment id>/gate?wait=<seconds> is a long-poll: the platform holds the request up to wait seconds (at most 60) and answers as soon as something changed, once the gate is final, or when the time is up — so CI asks again only until final is true, never every few seconds. The answer is the gate (status pass, blocked or none, each rule in rules) with experiment_status (pending until the round's first run is in, then running, done, cancelled, failed or invalid), final, conclusion (success, failure or neutral, as the GitHub check concludes) and report (each scorer's average and change against the baseline, the problems' titles, the page). The project token reads it.

GATE=$(curl -fsS -X POST … | jq -r .gate_url)
for i in $(seq 60); do
  G=$(curl -fsS -H "Authorization: Bearer $REVIEW_PROJECT_TOKEN" "$GATE?wait=60")
  [ "$(echo "$G" | jq -r .final)" = true ] && break
done
case $(echo "$G" | jq -r 'if .final then .conclusion else "neutral" end') in success) exit 0;; failure) exit 1;; *) exit 2;; esac

reviewctl eval --launch --project prj_… and the GitHub Action's launch: true do this, print the gate's table and exit 0 (passed), 1 (blocked) or 2 (not judged); see CI and the GitHub Action. The answer's fields are in v2 routes.

A simple wait on the job itself, with jq:

JOB=$(curl -fsS -X POST … | jq -r .job.id)
while :; do
  J=$(curl -fsS -H "Authorization: Bearer $REVIEW_PROJECT_TOKEN" https://<platform address>/v1/jobs/$JOB)
  case $(echo "$J" | jq -r .status) in succeeded|failed|cancelled|lost) break;; esac
  sleep 30
done
RUN=$(echo "$J" | jq -r .run_id)
curl -fsS -H "Authorization: Bearer $REVIEW_PROJECT_TOKEN" https://<platform address>/v1/runs/$RUN | jq -r .gate

With a CLI token you can read two more routes. GET /v1/runs/<run id>/report returns the whole report as JSON, or as Markdown with ?format=markdown: the report the pages read, with what re-judging changed, what the AI diagnosis set aside as the platform's cause and what a person counted as passed written in. With ?fills=1 it returns the report as the round reads it: fill-in runs and "Retest only what couldn't be tested" read in, the latest try counting, judged again by its pass criteria, with how many in folded. GET /v1/runs/<run id>/bundle returns the whole run bundle the test runner uploaded (.tar.gz); past the retention period it answers 410 with the code evidence_expired.

Test connection and project settings

These are routes the web app itself uses, listed here for reference when you script:

Route	What it does
PATCH /v1/projects/<project id>	Takes auto_rerun (true or false): switches re-runs by the platform on or off; on by default. The project's details have it too.
GET /v1/projects/<project id>/overlay	The test connection files in the platform overlay, each with written_by: person (you), dev (your developer machine's AI) or platform (the platform); null for a file saved before the field existed that the audit log can't tell.
PUT /v1/projects/<project id>/overlay/<path>	Saves a test connection file. When the file reads fine but won't run as written somewhere, it's saved all the same, and the answer has warnings (each with a zh and an en sentence).
GET /v1/projects/<project id>/wiring-warnings	The same warnings, for the review.yaml a sandbox reads now (the overlay's, else the repository's): { "warnings": [...] }.

Rerun what couldn't be tested

The routes behind rerunning an experiment's undecided rows (see undecided rows). A CLI token can read the GET.

Route	What it does
GET /v1/runs/<run id>/fill	What a fill-in run would run: left, planned, parts, the earlier fill-in runs in fills (each with fill_mode), or why it can't in refused. ?undecided=1: what "Retest only what couldn't be tested" would run, with free (the platform's problem, not charged) and tests, the version of each test it runs (was: the version that test ran, where the platform fixed it after).
POST /v1/projects/<project id>/jobs	{ "kind": "fill", "run_id": "…", "undecided": true, "tests": [{ "scenario": "…", "condition": "…" }] }: retests only what the run couldn't test, all of it when tests is left out; without undecided it runs the attempts that didn't run. Needs an Idempotency-Key. 422 fill_nothing_undecided when nothing is left undecided.

Dev AI routes

The coding AI on a developer's machine calls /v1/dev/* with a dl_ token, in the one project the connection was made for, as the person who made it. The reviewctl dev commands, the reviewctl dev mcp tools and the platform's MCP over HTTP all go through these routes. How to use them: Using the platform from a coding AI.

Only a person in the browser can make a connection or change its permissions:

Route	What it does
POST /v1/projects/<project id>/dev-links	{ "allow": {...}, "http": true, "replaces": "<id>" }, all optional. Without http: answers with a pairing code code and the connect command connect (valid 10 minutes). With "http": true: the connection is made at once, and the answer has http.url, http.token (the dl_ token, in this answer only) and the registration lines http.claude and http.cursor. replaces: the id of a code still waiting in this project that this one takes the place of (the page's "Get a new code"): it is voided as this one is made, and the answer says so in replaced; a connection already made is never voided this way.
GET /v1/projects/<project id>/dev-links	Open connections and codes still waiting: each with its permissions allow, what it used today spent_today_usd, and what it did last.
PATCH /v1/dev-links/<id>	{ "allow": {...} }: changes the permissions from the connection's next call; goes into the audit log.
POST /v1/dev-links/<id>/resume	Resumes a paused connection.
DELETE /v1/dev-links/<id>	Disconnects it.

allow is { "draft", "wiring", "prerun", "launch", "confirm", "assist", "gate", "daily_usd" }: the first seven true or false, in order "Write test drafts and upload files", "Change the test connection files", "Pre-run", "Start formal tests", "Confirm tests", "Have the platform's AI finish the connection" and "Change the pass criteria"; daily_usd is the most it may spend a day, in US dollars (0 to 1000), or null for no daily limit (only the organization's own credit and caps apply). Without allow, the dialog's defaults: everything but confirm and gate on, daily_usd 5, or null on a repository or image project. A connection made before this update has draft and prerun only.

Routes for a dl_ token:

Route	Tool	Permission
GET /v1/dev/status	get_status
GET /v1/dev/prompt	get_prompt
POST /v1/dev/answers	submit_answer	draft
POST /v1/dev/files	upload_file	draft
GET /v1/dev/tasks, POST /v1/dev/tasks/<id>/finish	get_tasks, finish_task
GET /v1/dev/wiring, PUT /v1/dev/wiring	get_wiring, put_wiring_file: each file with written_by; what the connection saves is recorded as dev, and the answer carries any warnings	wiring to save
GET /v1/dev/overlay	reviewctl dev pull: the test connection files as one .tar.gz
POST /v1/dev/connect-assist, GET /v1/dev/connect-assist	start_connect_assist, get_connect_assist	assist to start
GET /v1/dev/tests, GET /v1/dev/tests/<id>	list_tests, get_test
POST /v1/dev/tests/review	review_drafts	draft
POST /v1/dev/tests/<id>/confirm	confirm_test	confirm
POST /v1/dev/trials, GET /v1/dev/trial	try_run, get_setup	prerun to start
POST /v1/dev/jobs	run_tests	launch, held to daily_usd (null: to what the organization has left)
GET /v1/dev/jobs/<id>, POST /v1/dev/jobs/<id>/cancel	get_job / wait_job, cancel_job	only jobs this connection started can be cancelled
GET /v1/dev/runs, GET /v1/dev/runs/<id>/report, GET /v1/dev/runs/<id>/diagnosis	list_runs, get_report, get_diagnosis
GET /v1/dev/fixes	get_fixes
PUT /v1/dev/gate	set_gate	gate
POST /v1/dev/disconnect	reviewctl dev disconnect
GET /v1/dev/plan	get_plan: the test plan approved before, and what next
POST /v1/dev/plan, PUT /v1/dev/plan/answers, POST /v1/dev/plan/next-round, POST /v1/dev/plan/write, PUT /v1/dev/plan	start_plan, answer_plan, next_plan_round, write_plan, edit_plan: removed; they answer 410 with the code plan_removed, whose message says what to do instead
GET /v1/dev/v2/experiments, GET /v1/dev/v2/experiments/<id>, GET /v1/dev/v2/experiments/<id>/rows	list_experiments, get_experiment, get_experiment_rows
GET /v1/dev/v2/traces, GET /v1/dev/v2/traces/<id>, GET /v1/dev/v2/datasets, GET /v1/dev/v2/gate	list_traces, get_trace, list_datasets, get_gate
POST /v1/dev/v2/datasets/<id>/rows, POST /v1/dev/v2/scorers	add_rows, create_scorer_draft	draft
POST /v1/dev/mcp	MCP over HTTP: Streamable HTTP, no session, JSON answers; the same tools without upload_file. POST only.	per tool

The test plan step was removed (see test plans): GET /v1/dev/plan only reads the plan approved before (plan, in_force) and what next (next), with no approve_url any more.

Tests, jobs and runs are looked up by id in the connection's own project only. Common errors: 403 dev_not_allowed (the connection lacks this step's permission; message names it and who should switch it on, where), 403 not_this_connection (the job wasn't started by this connection), 402 dev_daily_cap (this formal test may cost more than is left of today's cap), 423 dev_paused and 423 trial_paused (5 failed submissions or pre-runs in a row: waiting for a person to click "Resume"), 422 estimate_unavailable (the most this formal test may cost can't be worked out), 422 trial_run (get_fixes or get_diagnosis named a pre-run's run, which has neither: its problems are in get_setup), 409 run_not_ready (a run its job named before its report is in: the message says which job to wait for). Every answer to a dl_ token carries the connection's language in the x-reviewctl-locale response header.

Billing and credit

With a CLI token (or in the browser) anyone in the organization can read its billing. <org id> is your organization's (GET /v1/me has it); another organization's answers 404.

Route	What it returns
GET /v1/orgs/<org id>/billing	The plan (monthly_cents, included_usage_cents, max_concurrent_sandboxes, max_run_minutes, assistant_daily_usd) and subscription, this period's usage against what the plan includes (usage_this_period: usage_cents, included_usage_cents), wallet (on Free, Pro, Ultra and Prepaid: included_cents and included_left_cents, this period's credit and what is left of it; balance_cents, credits_cents, used_cents; left_cents, what may still be spent now; null on Enterprise), can_redeem, top_up (min_usd, max_usd, presets; null where card top-ups aren't available), can_checkout, can_change_plan, plans (on sale by card), catalog (the plans compared), the spending cap (spend_cap.cap_cents) and invoices (total_minor in the invoice's currency: cents for usd, pence for older gbp ones).
GET /v1/orgs/<org id>/usage?month=YYYY-MM	The usage page: totals (usage and cents by type), by_project, by_day, allowance (the month's sandbox hours) and not_charged. not_charged is what the platform ran again because of its own problem and didn't charge: cents is what it would have cost, and each by_reason item has a reason (auto_fill: the platform's fill-in run; rerun: re-runs by the platform; sandbox_lost: a lost sandbox's time that left no result; model_retest: a retest of what only the platform's model left untested; judge_fallback: what the second judge model would have cost above the first's, used while the first was rate limited; platform_trial: the sandbox time of a pre-run or check that failed only for the platform's own part), cents and sandbox_seconds. That money isn't in total_cents; the sandbox time is in totals.sandbox_seconds but not in allowance.used_hours. The current month without month. See the platform's problem isn't charged.
GET /v1/orgs/<org id>/billing/overview?month=YYYY-MM	The billing dashboard: wallet, period (this period's start, end and spend), usage (as the usage page above, with not_charged; the current month without month), limits (included, spend_cap, sandbox_hours, assistant_today, autonomy, each null when it doesn't apply), credits (the credit lines, newest first: source is stripe for a card top-up, code or admin) and draws (what the balance paid each period). Amounts are in US cents; assistant_today gives the day's limit both in model bill (cap_usd, used_usd, left_usd) and as billed (cap_cents, …).
POST /v1/orgs/<org id>/credit-codes/redeem	{ "code": "PW-…" }: an owner in the browser only (a CLI or project token is refused). 201 with cents and the new wallet; 200 with "duplicate": true when this organization redeemed it already.

All amounts are in US dollars: fields ending in _cents are US cents. Every limit (daily_usd, cap_usd, daily_cap_usd, agent_cap_usd, platform_judge_cap_usd, an estimate's judge_cap_usd and the like) is in what you are billed, when set as well, the same figure the pages show (since 26 October 2026; a few were in model bill before); in usage, judge_usd, assistant_usd, target_usd and the like are dollars of model bill, what is billed being the _cents beside them. Until October 2026 these fields were in pence (*_pence) and an invoice's total was total_pence.

A refused code answers 422 with credit_code_invalid, credit_code_expired, credit_code_revoked or credit_code_used. On a plan billed by invoice (Enterprise) the answer is 409 credit_code_plan. After 10 tries an hour for the organization, or 20 from one IP address, it is 429 credit_code_attempts. See redeeming a credit code.

Paying is for an owner in the browser (a CLI or project token answers 403 session_required). Each route answers { "url": "…" }, the Stripe page to open:

Route	What it does
POST /v1/billing/checkout	{ "plan_id": "pro" } (or "ultra"): Stripe's checkout page for a subscription. 409 already_subscribed when a subscription is live already (change plan instead), 409 managed_by_hand on a plan we manage by hand, 422 plan_not_available.
POST /v1/billing/top-up	{ "amount_usd": 25 }, whole dollars from 5 to 1000: Stripe's checkout page for a one-off payment. The balance goes up once Stripe confirms the payment, usually within seconds of coming back to the billing page. 400 top_up_amount, 409 top_up_plan (a plan billed by invoice), 503 top_up_off (card top-ups not switched on).
POST /v1/billing/portal	{} opens Stripe's page for the card, the invoices and cancelling; { "flow": "change_plan" } opens it on changing between Pro and Ultra. 409 no_customer before the organization's first payment, 409 no_subscription for change_plan without a live subscription.

All three answer 403 owner_required for a member, 503 billing_off where paying on the web isn't switched on, and 502 stripe_error when Stripe doesn't answer. See billing and usage.

Alert webhooks

Give an alert a webhook under Automations → Alerts, and the platform POSTs JSON to it once when the alert starts firing, once when it resolves, and on "Send a test" (only a 2xx counts as delivered; redirects aren't followed; 10-second timeout). The signing secret is shown once, when the webhook is saved.

• Header x-preworld-event: alert.firing, alert.resolved or alert.test.
• Header x-preworld-signature: t=<unix seconds>,v1=<hex>, where v1 is the HMAC-SHA256 of <t>.<raw request body> under the signing secret. Check the signature, then that t is at most 5 minutes old.
• Body: { "type", "alert": { "id", "name", "project_id", "project", "condition" }, "state", "reading": { "value", "breached", "words": { "zh", "en" } }, "at", "url" }.

import { createHmac, timingSafeEqual } from 'node:crypto';
const [t, v1] = header.split(',').map((p) => p.split('=')[1]);
const want = createHmac('sha256', secret).update(`${t}.${rawBody}`).digest();
const ok = timingSafeEqual(want, Buffer.from(v1, 'hex')) && Math.abs(Date.now() / 1000 - Number(t)) <= 300;

Errors

An error answers with an HTTP status and one JSON object:

{ "error": { "code": "project_busy", "message": "This project already has a job queued or running. Start again once it ends or is cancelled." } }

code never changes with the language; message is for people. Send x-reviewctl-locale: en or zh to choose the language of messages. Without it, a project API token is answered in its maker's language, else the organization's; other requests are answered in Chinese. Some errors add detail, a parser's own words. A malformed body is 400 invalid_body, with the validator's message in English.

Status	Common codes	What to do
400	invalid_body, idempotency_key_invalid, bad_filter (with position), bad_cursor, invalid_characters	Fix the request. A query parameter out of range (say limit=1000) is invalid_body too.
401	unauthorized	The token is wrong or revoked, or its maker left: create a new one.
402	trial_ended, trial_credit_used, subscription_inactive, prepaid_used, sandbox_quota, spend_cap_reached	The plan refuses new runs and spending; see when tests can't start.
403	project_token_not_allowed, token_scope_missing, session_required, cross_site, org_suspended, terms_required	A project token called a route it can't or lacks the upload scope; a step only a person in the browser can take; a browser request from another site; the organization is paused; or, for a CLI token, its person must accept the terms in the browser.
404	not_found, version_not_found, row_not_found, baseline_not_found	Not in your organization, or a wrong id.
409	project_busy, idempotency_conflict, lost_job_unacknowledged, edit_in_scenario_editor, experiment_finished, derived_experiment, idempotency_key_reused	A run is already waiting or going; a key was reused for a different launch or batch; a run lost contact mid-run and a person must confirm on the web that they've checked the test backend (for a sandbox run whose backends were all inside the sandbox, the platform checks it itself; see after a lost run); a scenario's row changes in the scenario editor; rows can't be added to a finished experiment, or to one made from a run.
422	nothing_to_test, commit_not_for_project, image_ref_not_for_project, no_runner, no_chat_ai, chat_ai_ambiguous, unknown_scorer, unknown_dataset, unknown_baseline, invalid_scorer	The project can't run as asked, or the request names something the project doesn't have; the message says why.
429	rate_limited, assistant_allowance	Too many sign-in attempts or Gateway calls (wait the seconds given), or the assistant's daily allowance is used up.
5xx	internal	Try again later; if it persists, see status.

Rate limits

Project API token calls to /v1, /v2 and OTLP have no request quota. What limits runs is the project and the organization:

• One launch at a time per project: while a run is waiting or going, another is refused with 409 project_busy.
• Sandboxes beyond the organization's limit wait in a queue (limits).
• The AI Gateway takes 600 calls a minute per organization by default, then answers 429 rate_limited with Retry-After (Gateway limits).
• The sign-in endpoints (sign-in links, invitations, GitHub and Google sign-in, CLI device codes, test runner pairing) take 20 requests a minute per IP address, then answer 429 rate_limited.

Next steps

• v2 routes: datasets, scorers, experiments, the gate and traces.
• CI and the GitHub Action: wait for the release gate in CI.
• Command line: reviewctl eval for test sets run inside CI.
• Billing and usage: what runs cost and why a launch is refused.
• Troubleshooting: symptoms and fixes.