review.yaml reference

review.yaml tells the test runner how to start your agent, route its tool calls, reset and read your test backend, and judge the results. This page lists every section and field the platform accepts.

Where the file lives

Connection	Where	Where the agent command runs
GitHub repository	.reviewctl/review.yaml in the repository, or in the platform overlay, which is laid over the repository's .reviewctl/ folder	The repository root
Your own image	The platform overlay only (.reviewctl/review.yaml); the overlay is mounted read-only at /reviewctl in the container	The image's own WORKDIR
Local test runner	Anywhere on the test machine; commands take --config (default review.yaml in the current folder)	The folder the file is in (its parent, when that folder is .reviewctl)
Chat AI over HTTP	Not used

Paths in agent, attach, backend and scenarios are relative to the folder review.yaml is in. Paths in environment and services are relative to the repository root and can't leave it.

Paths are read from .reviewctl/

In a project connected through a GitHub repository or with its own image, review.yaml is in .reviewctl/, so these are read from .reviewctl/, not from the repository root:

• backend.reset's command runs in .reviewctl/, and the scripts it names are looked for from there;
• backend.reset's body_file and script, backend.state.reader's path and catalog;
• scenarios.dir;
• agent.version's prompt_files and tool_config_files.

For a script at .reviewctl/seed.mjs in the repository, write [node, seed.mjs]. [node, .reviewctl/seed.mjs] looks for .reviewctl/.reviewctl/seed.mjs. A file at the repository root takes ../ in front, such as ../prompts/system.md. The agent's own command (agent.command) is different: it runs from the repository root (see the table above).

Such a mistake doesn't stop the file from loading, so the platform warns about it rather than refusing it: when the overlay is saved, under "Test connection file" in the wizard under Targets, and in the "Preflight checks" of "Test new version: all options". The warning names the line, the path looked for and how to fix it. It also warns about these mistakes that raise no error:

• a file in agent.version.tool_config_files or prompt_files that exists nowhere: the platform skips it, and "Tool configuration" or "Prompt" in the version comparison won't change with it;
• a runner_env variable that never reaches the agent and that the platform doesn't use either (see runner_env).

Rules for the whole file

• It is YAML with schema_version: 1.
• Unknown fields are refused at every level. Check the spelling of a field the error names.
• Secret values don't go in the file. A field whose name ends in api_key, apikey, secret, token, password or passwd can't hold a literal value. A value that looks like a key (sk-…, sk-ant-…, ghp_…, Slack tokens, Bearer …, JWTs) is refused wherever it appears. Fields whose names end in _env hold variable names, so they are fine, and so are references written env:NAME or ${NAME}. Put the variable name in agent.env_passthrough and the value in project secrets or on the test machine. Values that aren't secret can go in runner_env, which checks its values instead of its names.
• Environment variable names use capital letters, digits and underscores, and start with a letter.
• A model is written <provider>:<model-id>, for example openai:gpt-4.1-mini. The test runner can call the providers openai, anthropic, azure and openai-compatible.
• To check a file: run reviewctl doctor on the test machine. On the web, the overlay editor refuses a file that wouldn't load and says which line and field are wrong.

Minimal example

Eight sections are required: schema_version, project, agent, attach, backend, judge, scenarios and budget.

schema_version: 1
project: { id: prj_01J8ZK3Q9MXYZ0000000000000, name: support-agent, domain: generic }
agent:
  driver: command
  command: [node, dist/agent.js, --message, "{{task}}"]
  env_passthrough: [OPENAI_API_KEY]
attach: { mode: endpoint_only }
backend: { mode: own }
judge: { model: "openai:gpt-4.1-mini" }
scenarios: { dir: scenarios }
budget: { per_attempt: { minutes: 5, model_calls: 30 } }

This runs the agent and judges its replies, but the platform sees no tool calls and no data. Add attach routes and a backend reset and reader to test those.

Full example

A repository project whose agent calls REST tools on a test backend started with Docker Compose:

schema_version: 1
project: { id: prj_01J8ZK3Q9MXYZ0000000000000, name: expense-agent, domain: expense }

environment:                         # GitHub repository only
  setup: [pnpm install --frozen-lockfile]

services:                            # the test backend, next to the agent
  compose: docker-compose.test.yml

agent:
  driver: command
  command: [node, dist/agent.js, --message, "{{task}}", --user, "{{identity.user_id}}"]
  env: { AGENT_VARIANT: baseline }   # plain settings
  env_passthrough: [OPENAI_API_KEY]  # values come from project secrets
  model: "openai:gpt-4.1"
  version: { git: true, prompt_files: [../prompts/system.md] }

attach:                              # the tool proxy
  mode: http_proxy
  base_url_env: EXPENSE_TOOLS_BASE
  upstream_base_url: http://127.0.0.1:47591/tools
  routes:
    claim_list: { method: GET, path: /claims }
    claim_submit: { method: POST, path: /claims/{claim_id}/submit }

backend:
  mode: own
  reset: { type: http, url: http://127.0.0.1:47591/reset, state_ref_path: $.state_ref }
  state: { reader: { kind: http, base_url_env: EXPENSE_STATE_BASE, catalog: scenarios/state_queries.yaml } }

judge: { model: "openai:gpt-4.1-mini" }
scenarios: { dir: scenarios }
gate:                                # local runs only; the platform uses the project's pass criteria
  key_tasks:
    - { scenario: submit-claim, condition: normal, pass_k: { k: 3, min: 1 } }
  security: { block_on: [verified_impact, verified_violation] }
budget:
  per_attempt: { minutes: 3, model_calls: 30 }
  total_model_calls: 600

Sections at a glance

Section	Required	GitHub repository	Your own image	Local test runner
schema_version	Yes	Used	Used	Used
project	Yes	id is replaced by the project's own	id is replaced by the project's own	id must be the project's id
environment	No	Used	Not used: the image is the environment	Not used
services	No	Used	Used	Not used
agent	Yes	Used	Used (inside the image)	Used
attach	Yes	Used	Used	Used
backend	Yes	Used	Used	Used
judge	Yes	Used	Used	Used
scenarios	Yes	dir only	dir only	dir; include and required for reviewctl review
gate	No	Not used: the project's pass criteria apply	Not used	reviewctl review only
budget	Yes	Used	Used	Used
cloud	No	Not used	Not used	reviewctl review and reviewctl push
runner_env	No	Used	Used	Used

schema_version

Always 1.

project

Field	Type	Default	What it does
id	prj_ and 26 characters	Required	The project's id, as the web page shows it. reviewctl runner register needs it to match the project you pair with. In a sandbox, the platform replaces it with the id of the project that started the run, so one repository can serve several projects.
name	Text	Required	A name for reports run locally.
domain	Text	Required	The business domain, such as expense or generic. orders-refunds is the domain of the built-in mock backend.

environment

How a sandbox prepares a checked-out repository. GitHub repository projects only; leave it out and the runtime is detected from the lock files at the repository root.

Field	Type	Default	What it does
runtime	node@20, node@22, node@24, python@3.11, python@3.12, go@1.23 or java@21	Detected from pnpm-lock.yaml, package-lock.json, yarn.lock, uv.lock, poetry.lock, requirements.txt, pyproject.toml, go.mod, pom.xml, build.gradle (first match); the Node version the repository asks for (.nvmrc, .node-version, .tool-versions, mise.toml, or engines.node in package.json when it needs 23 or later) picks node@20 or node@24, .python-version picks the older Python	Chooses the default setup commands and cache folders; node@24 also runs the setup and the agent on Node 24 (Node 22 otherwise).
setup	List of shell commands, at most 20, each at most 500 characters	By lock file: pnpm install --frozen-lockfile, npm ci, yarn install --frozen-lockfile, bun install --frozen-lockfile (bun.lock or bun.lockb, with the sandbox image's bun; on a test runner without bun, an older image, it runs as npx --yes bun install --frozen-lockfile, and either line written in setup runs in the form the machine has), uv sync --frozen, go mod download, …	Run in order in the repository root after the checkout, with CI=1 and the project secrets as environment variables. A command that prints nothing for 15 minutes is stopped (see preparation that stalls); otherwise the run's time limit bounds them. Once every command has exited, the install is over and the log says "Dependencies installed".
cache	List of folders, at most 20	node_modules for Node.js, .venv for Python, none for Go and Java	Kept between runs for 30 days, keyed by the runtime, the architecture, the lock file and the setup commands. Saved after the install and restored at the start of the next preparation, each a step of the platform's own taking at most 8 minutes; one that fails is skipped and never fails the run.
arch	amd64 or arm64	None	Accepted; not used today. The sandbox's architecture is the project's setting under "Settings" → "Sandbox".
container	See below	None: the agent runs on the virtual machine	Build the repository's own container and run the agent in it.

Runtimes on the machine image

On the virtual machine (without container), a sandbox installs with the runtime, setup and cache written here; a field left out is detected from the lock files. runtime chooses the defaults and the cache key, not the binaries: the machine image always has Node.js 20 and 22 (node is 22; node20 and node22 are on the PATH too), Python 3.11 and 3.12 with uv, Go 1.23 and Java 21.

setup and the agent are different users

setup and a command data reset (backend.reset with type: command, run before each attempt) both run as the test runner's own user. The checkout and what setup installs (.venv, node_modules) belong to that user.

The agent runs as another, unprivileged user. Each attempt runs in its own fresh copy of the checkout, and the agent can write to that copy. A file it opens from its working directory, such as sqlite3.connect("app.db"), just works, and the next attempt starts from a new copy. The attempt's reset runs in the same copy, and the state reader reads the copy's files. So a relative path in review.yaml points to the same file for the reset, the agent and the checks.

.git, node_modules, .venv and the cache folders aren't copied. The copy links to the checkout's own folders, which the agent can read and run but not change. The agent can also write to the directory in the environment variable REVIEW_ATTEMPT_DIR, a new, empty directory for each attempt.

If the checkout is over 500 MB without those folders, it isn't copied. The attempts then run in the checkout itself, which the agent can only read (the run log says so), and files it writes have to go under REVIEW_ATTEMPT_DIR. When the agent can't write, it fails with an error such as attempt to write a readonly database. The pre-run quotes that line under "Does the entry program finish normally" or "Does the test environment work".

The agent may not be able to read what the reset makes. In a sandbox, files and folders made by the test runner's user are readable by that user only (umask 077). Before the agent starts, the platform opens only the attempt's copy of the repository to it, so the agent can't read what a reset makes outside that copy (a folder beside the repository, say). The agent then fails with the system's permission error (EACCES), which can look like the agent refusing on its own when it never read the file. Make what the agent should read inside the copy, or open it up in the reset itself (chmod 755 for a folder, 644 for a file). The run log of a run with a command reset says so too.

environment.container

Field	Type	Default	What it does
dockerfile	Path	One of dockerfile and devcontainer is required	The Dockerfile to build.
devcontainer	Path		A devcontainer.json. Its image, or build.dockerfile, build.context and build.args, are used, plus containerEnv and postCreateCommand. Devcontainer features are not supported.
context	Path	The Dockerfile's own folder	The build context. Only with dockerfile.
workdir	Absolute path	/workspace	Where the checkout (each attempt's own copy of it) is mounted in the container; every attempt runs there. Not /reviewctl-scratch.

postCreateCommand, then setup, run once in the built container. The built image is kept per commit for 30 days, so the same commit isn't built twice. The agent runs in the container as the same unprivileged user, behind the same network rules, as on the virtual machine.

services

The test backend a sandbox starts next to the agent with Docker Compose. Either a Compose file in the repository:

services:
  compose: docker-compose.test.yml

or the services listed by name (a lowercase name, up to 31 characters):

Field	Type	Default	What it does
image	Image reference	Required	The container image to start.
env	Map of NAME: value, at most 30	None	Environment variables for the container.
ports	List of port numbers, at most 10	None	Ports published on 127.0.0.1 for the agent.
ready	Shell command	None	Run in the service's container every 5 seconds; the run waits until it exits 0, for example pg_isready -U postgres.

Services must be up within 3 minutes. The agent reaches them only on 127.0.0.1, through published ports. In your own Compose file, publish the ports the agent uses on 127.0.0.1 (for example "127.0.0.1:47591:47591") and make the server in the container listen on 0.0.0.0.

When backend has no reset, the sandbox puts the services back as they were when the job started, before every attempt after the first. It removes their containers and volumes, starts them again from the same images and waits for them the same way. A file that one attempt deletes in a service, or data it changes, is back for the next attempt. The run log says how long each restore took. A host directory a service mounts is not put back. With a backend.reset, the sandbox uses that reset instead, unless restore asks for the services to be made again too:

Field	Type	Default	What it does
restore	all, or a list of the Compose file's service names	None	With a backend.reset: before each attempt's reset, these services (or all of them) are made again from their images, with their volumes; the others are left as they are. Only with compose. Without a backend.reset, every service is put back anyway.

No service, and no container a reset or setup command starts, may get control of Docker or of the sandbox machine itself. That rules out:

• a bind of /var/run/docker.sock or /run/docker.sock, of a folder holding it, or of a symlink to it
• privileged, or pid, network_mode or userns_mode set to host
• capabilities such as SYS_ADMIN in cap_add, or seccomp/apparmor unconfined
• host devices other than a GPU's
• the machine's /proc, /sys, /dev or home folders, or its system folders mounted writable
• a DOCKER_HOST variable

The sandbox refuses such a Compose file before anything starts, whether you wrote it or the platform did, and names each service and what it holds. A container started that way between attempts is removed before the agent runs, and the attempt fails with the same reason. If you want fresh services before each attempt, don't run a service that makes containers. Leave out backend.reset, or use restore.

An attempt may start containers next to the services, for example a fresh copy of them that its reset makes from their images or on their network. These are removed with their volumes once the attempt is over, before the next reset. Each attempt still gets its own, and they don't pile up until the sandbox runs out of memory.

A sandbox run also hands in a partial report after each attempt. If the sandbox is lost before the run ends, the last partial report becomes the run's report. What ran is judged as usual, and the attempts it never got to are marked as having no result.

agent

How to start the agent for each attempt.

Field	Type	Default	What it does
driver	command, http or harness	Required	How the test runner talks to the agent.
command	List of arguments	Required with command	The command line, one process per attempt. Placeholders below.
endpoint	See below	Required with http	A running HTTP endpoint.
harness	Path to a JavaScript module	Required with harness	A module that exports runTask(input, wiring, signal); it runs inside the test runner.
env	Map of NAME: value	None	Plain settings for the agent, such as AGENT_VARIANT. Never secrets.
env_passthrough	List of variable names	None	Variables the agent may receive from the test runner's environment: project secrets in a sandbox, the machine's environment or .env on a local test runner. In a sandbox with no OPENAI_API_KEY secret, the agent gets OPENAI_BASE_URL and OPENAI_API_KEY for the platform's model whether or not they are listed.
model	Model reference	None	The agent's own model, for the report. The judge model must be a different one.
version.git	true or false	true	Use the git commit as the code version.
version.prompt_files	List of paths	None	Files whose contents make up the prompt version. Paths are read from the folder of review.yaml; a file that isn't there is skipped, with a warning.
version.model_env	Variable name	None	The variable holding the agent's model, when model isn't set.
version.tool_config_files	List of paths	None	Files whose contents make up the tool configuration version. Paths, and a missing file, as above.
reply	Text	stdout_last_json	Accepted; not used by the test runner today.
framework	Text	None	Informational only.

The agent process gets only agent.env, the env_passthrough variables, basics such as PATH, HOME and LANG, and what the test runner adds: the tool wiring (below), the address for asking the user, REVIEW_ASK_USER_URL (see When the agent asks its user), REVIEW_PREAPPROVED_TOOLS when the test pre-approves tools (see pre-approval), REVIEW_PREPARE_URL for marking what the wiring prepares (see preparing an attempt), REVIEW_MAX_MODEL_CALLS, REVIEW_MAX_WALL_MS, ACTOR_TOKEN when the test's user has a token, REVIEW_FILES and REVIEW_FILES_DIR when the test has files, and in a sandbox with no OPENAI_API_KEY secret, OPENAI_BASE_URL and OPENAI_API_KEY for the platform's model (these two win over agent.env). Nothing else from the machine reaches it.

The command driver

Placeholders in command:

Placeholder	Filled with
{{task}}	The task text of the test
{{identity.user_id}}, {{identity.role}}	The test's user and role
{{actor_token}}	That user's token, from the variable the test names
{{max_model_calls}}, {{max_wall_ms}}	The limits of this attempt
{{files_json}}	The test's files as JSON: [{name, path, mime, size, sha256}]

The agent writes its result as the last line of standard output, one JSON object: text (the reply), and optionally usage (model_calls, input_tokens, output_tokens), stop_reason (error or max_steps) and error. JSON lines on standard error are kept as the transcript.

{"text": "Claim C-1042 is submitted.", "usage": {"model_calls": 4}}

The http driver

Field	Type	Default	What it does
endpoint.url	URL	Required	Where each task is sent, as POST with {"prompt", "actor_token", "max_steps"}.
endpoint.reply_path	Path starting with $	Required	Where the reply is in the JSON answer, for example $.reply. An answer sent as server-sent events is read from its last final event.
endpoint.send_files	none or base64	none	base64 adds the test's files to the body as files: [{name, mime, sha256, base64}].
endpoint.method, endpoint.body_template, endpoint.reset_url			Accepted; not used by the test runner today.

The answer may report usage.model_calls, as a command's last line does.

Conversations of several turns

A test can have its user say more than one thing: a script of messages (a scenario's task.turns) or a simulated user (simulated_user: who the user is, what they want done, at most how many turns, and optionally when to stop). The simulated user is played by the project's judge model. It reads each reply and writes the next message, or says it is done; it never sees the test's expected results or its checks. The test's prompt (or the question's input) is the first message; with a simulated user and no input, the simulated user writes the first message itself (on a chat AI only: on a test runner the question needs its input).

The entry is called once per turn, with the conversation so far:

Driver	What each turn gets
command	One process per turn. {{task}} is this turn's message, {{turn}} its number (1 is the prompt), {{history_json}} the messages before it as [{role, content}] (the user's, and the agent's own replies). The same in env: REVIEW_TURN, REVIEW_HISTORY.
http	One POST per turn. Beside prompt (this turn's message) the body has turn, history (the messages before it) and messages (history plus this message as the user's: the chat format).
harness	runTask is called once per turn, with input.turn and input.history.

A task of one message is sent exactly as before: no turn, history or messages. The turns share the attempt's wall clock and model-call budget (the time the simulated user takes between turns isn't counted), and every tool call of every turn goes through the test runner's proxy. A fault or attack can aim at one turn: match.turn on a fault or a tool-result attack (its nth then counts that turn's calls), or turn on a task-input attack (which of the user's messages carries it). The attempt keeps the whole conversation in conversation.json.

The multi-turn fields

review.yaml itself has no multi-turn fields: it says how the entry receives turns (above). The conversation is set in the test:

Where	Field	Type and limits
A scenario (under task)	turns	The user's messages after the prompt: a list of 1 to 11 strings, each up to 20,000 characters.
	simulated_user	See below. The prompt counts as turn 1. Not together with turns.
A test-set question (a case)	turns	The user's messages, the first included: 2 to 10 strings, each up to 20,000 characters (instead of input).
	messages	A pasted conversation, [{role, content}], 1 to 50, ending on the user's turn (instead of input).
	simulated_user	See below; input, when given, is its first message (required on a test runner). Not together with turns or messages.
A dataset row (v2)	turns	[{role: "user" | "assistant" | "system", content}], up to 50 (a conversation recorded or imported with the row)
	simulated_user	See below.
A fault or an attack	match.turn (fault, tool-result attack), turn (task-input attack)	1 to 12; 1 is the prompt. It can't be past the conversation's last turn.

simulated_user:

Field	Type	Default	Meaning
persona	Text, 1 to 2,000 characters	Required	Who the user is, in a few sentences.
goal	Text, 1 to 2,000 characters	Required	What the user wants done in this conversation.
max_turns	Whole number, 1 to 12	6	The most messages the user sends, the first included.
stop_when	Text, 1 to 300 characters	—	When the user stops besides the goal being met; absent, when the goal is met or clearly can't be.

task:
  prompt: "I bought the wrong size shoes."
  # goal, goal_text … as in any scenario
  simulated_user:
    persona: "An older customer shopping online for the first time; short sentences, no jargon."
    goal: "Return the shoes and find out when the refund arrives."
    max_turns: 6
    stop_when: "Support says when the refund arrives."

A test runner needs the multiturn capability for these (a current reviewctl has it); on a test runner, a test set's scripted turns and pasted messages don't run yet, while a simulated user does when the question has its input. The scorer conversation_goal judges whether the simulated user's goal was met (scorers). Each conversation ends by the script running out, the simulated user stopping, max_turns, or an error, and the report says which.

When the agent asks its user

Some agents ask their user before they act: for permission before running a command or changing a file, or to confirm something. On a test runner there's nobody to answer, and unless that is wired, the agent fails (run headless, for example, with HOST_CAPABILITY_UNAVAILABLE), and the test never reaches what it tests.

For every attempt, the test runner gives the agent an address, REVIEW_ASK_USER_URL (in wiring for the harness driver). The wiring passes the agent's own question to the platform:

POST $REVIEW_ASK_USER_URL
{"question": "Allow running the bash command ls -la?", "kind": "permission"}

• question: the question as the agent asked it.
• kind: permission for a permission request; leave it out for a business confirmation ("Submit this expense claim?").

The platform answers {"answer": "yes"} or {"answer": "no"}, and the wiring hands that back to the agent. How it answers follows each test's own setting: where the test allows or requires asking the user, the answer the test sets (yes by default; one set to test both runs the condition once with yes and once with no). Where the test doesn't allow asking the user, the answer is no, with "forbidden": true.

Each question and its answer go into the attempt's record (a confirmation with its kind, marked forbidden where the test doesn't allow asking), and the attempt page says "asked the user to allow "…"; the user answered yes", or "asked the user "…"; this test doesn't allow asking the user, so the platform answered no". The verdict counts a question in the record as the agent asking its user, as it does a question in the reply.

The rules for the wiring:

• Pass the question on; never approve for the user: don't answer yes itself, don't switch off the product's own permission checks, don't turn on a "skip confirmation" option. The one exception is a tool the test pre-approves.
• With no REVIEW_ASK_USER_URL (the test runner's reviewctl is too old), don't guess an answer: end the attempt, with the last line {"text": "", "stop_reason": "error", "error": "the agent asks its user, and the test runner gives no REVIEW_ASK_USER_URL"}.
• When the wiring itself can't run a tool (the host lacks a capability the tool needs, for example HOST_CAPABILITY_UNAVAILABLE when run headless, or the wiring's own relay fails), answer that call with HTTP 503 and {"error": {"code": "host_error", "message": "<the host's own error>"}}. The report then counts an attack whose tool never ran as never reaching its target (the platform runs it once more first), not as the agent's error. An error whose message carries HOST_CAPABILITY_UNAVAILABLE counts the same whatever its code, unless it is a 4xx. A tool that ran and failed, or the product's own permission check refusing it, goes back with its own error, never host_error. When the error is the agent's own permission layer wanting to ask first, run headless, with nobody to answer (it carries HOST_CAPABILITY_UNAVAILABLE or cannot prompt for permission), the platform doesn't re-run it: another try is the same. See pre-approval below.

"Let AI finish and check" wires an agent to these rules, see GitHub repository.

Pre-approval in a test

Some agents, run headless, ask for permission before they use a tool. Nobody can answer, so the tool never runs and the test never tests it. To test it, the test can pre-approve that tool:

task:
  human_confirmation:
    preapprove: [bash]

• task.human_confirmation.preapprove: a list of 1 to 10 tool names, for this test only. It's a precondition a person sets and a person confirms: a person clicks "Pre-approve <tool> in the test" on the experiment's undecided rows and the platform saves a new draft version that says it (or a person writes it into the test). Automatic approval never confirms it, and the platform never pre-approves anything by itself (see settling what couldn't be tested).
• For each attempt of that test, the test runner gives the agent REVIEW_PREAPPROVED_TOOLS: the tool names, comma-separated (in wiring for the harness driver). A test without it gets no such variable.
• The wiring pre-approves exactly those tools, for that attempt only, in the agent's own permission settings: an allow list it starts with, say, or a permission rules file for that attempt. Other tools still ask through REVIEW_ASK_USER_URL. It doesn't switch off the product's own permission checks, turn on skipping confirmation for every tool, or change the agent's own settings in review.yaml.
• When the test runner's reviewctl is too old to know the field, the page says to update reviewctl.

Preparing an attempt

Some tests need something in place first: an earlier session, a conversation before the attack, data to find. The test connection prepares these; they aren't what the agent does in the test:

• Prepare data with backend.reset. It runs before each attempt, and the platform never counts it as the agent's calls.
• Hand the agent an earlier conversation as fixed messages, for example by having the adapter read a history written under .reviewctl/ and pass it in as the conversation so far. Don't have the agent really call its tools just to make a history.
• When the agent really has to run first to make it (say, only the agent itself can write its memory), the wiring POSTs {"phase": "prepare"} to REVIEW_PREPARE_URL before that run, and {"phase": "task"} after it, before handing the agent the test's message:

POST $REVIEW_PREPARE_URL
{"phase": "prepare"}
… run the agent on the preparing conversation …
POST $REVIEW_PREPARE_URL
{"phase": "task"}
… hand the agent the test's message …

The tool calls in between still go through the tool proxy to the tools, with no fault or attack injected, no forbidden rule checked and no call budget used, and are recorded as set up by the test's wiring. Whether an attack got through, whether a forbidden action happened, and every call count read only the calls the agent made itself in the attempt; the attempt page marks the prepared calls "Set up by the test's wiring: not the agent's own", and the judge model reads them apart from the agent's. What the preparation writes to the database is still there for the data checks afterwards: prepare the data you check or count with backend.reset. Every attempt has REVIEW_PREPARE_URL (in wiring for the harness driver); without it (the test runner's reviewctl is too old), don't make such a history inside the attempt.

attach

The tool proxy: the agent's tool calls go through the test runner, so each one is recorded and faults and attacks can be injected into it.

Field	Type	Default	What it does
mode	mcp_wrap, http_proxy, fixture or endpoint_only	Required	mcp_wrap for MCP servers, http_proxy for REST tools, fixture with the built-in mock backend, endpoint_only when tools aren't observed.
mcp_config	Path	Required with mcp_wrap	The agent's MCP configuration file (mcpServers). It is read, not changed.
servers	List of server names	All in mcp_config	Which MCP servers to route through the proxy; the rest pass through as they are.
apply	env or in_place	env	env hands the agent the wrapped configuration's path in REVIEW_MCP_CONFIG; in_place writes it over mcp_config for the attempt and puts the original back afterwards.
base_url_env	Variable name	Required with http_proxy	The variable the agent reads its tools' base URL from; the test runner sets it to the proxy.
upstream_base_url	URL	Required with http_proxy	Where the real tools are.
routes	Map of tool name → {method, path}	Required with http_proxy	Each tool's REST route. method is GET, POST, PUT or DELETE; path starts with /, and {name} segments become arguments. Tool names use letters, digits, _ and -, up to 64 characters.
enforce_forbidden_writes	true or false	false	Block writes a test forbids instead of only recording them.
process_egress	true or false	false	Also route the agent process's other outbound HTTP(S) through the proxy (HTTP_PROXY, HTTPS_PROXY), so every host it contacts is recorded; only requests carrying a planted canary are blocked.

attach.mode: fixture goes with backend.mode: fixture, and only with it.

What goes through the proxy as a tool is the step where the agent acts: running a command, running code, reading or writing a file, sending a request, calling a business API, each call one thing to do. A step the program under test takes after every reply (parsing the reply into edits and applying them, handing the commands the reply lists to whatever runs them) is called even with nothing to do, so don't wire that step itself as a tool: to see what it does, wire the step that really writes the file or runs the command, or send it through the proxy only when it has something to do. An agent that declares no tools but whose code runs commands or code, reads or writes files, or reaches outside addresses for the user needs that step wired too: "replies only" isn't enough for it.

Write each route in routes on its own line. Two routes joined by a comma on one line (read: {method: POST, path: /read}, bash: {…}) aren't valid YAML, so saving is refused, and the error names the line: move what follows the comma to the next line, indented like the route before it.

backend

Your test backend: how to reset it before each attempt and read its data afterwards for the data checks.

mode: own

Field	Type	Default	What it does
reset	One of the reset types below	None	Runs before each attempt. Without it there is no reset, and data checks can't be trusted.
state.reader	One of the readers below	None	Reads the data after each attempt.
recheck_after_ms	0 to 30000	3000	Reads the data once more this long (in milliseconds) after the check, on one attempt per scenario, to catch a result that arrives late. 0 turns it off.

Reset types:

type	Fields	What it does
command	command (list of arguments)	Runs the command in the folder of review.yaml (.reviewctl/ for a repository or image project, see paths are read from .reviewctl/), as the test runner's own user, with the test runner's environment (runner_env included) plus REVIEW_WORK_DIR and REVIEW_ATTEMPT_SHORT. The agent may not be able to read what it makes: see the warning under environment above.
http	url; method (POST or PUT, default POST); body; body_file (a JSON file merged under body); state_ref_path (where the answer names the state reference, such as $.state_ref)	Calls your reset endpoint with a JSON body.
sql	url_env (variable holding the database URL); script (path to a SQL file)	Recreates the database and runs the script. SQLite only today: sqlite:///path, file:///path or a plain path.

In reset arguments and body strings, {{attempt_short}}, {{attempt_id}} and {{work_dir}} are filled in.

Readers:

kind	Fields	What it reads
sqlite	path, catalog	A SQLite file.
sql	url_env, catalog	The database in the variable's URL; SQLite only today.
http	base_url_env, catalog	Your read-only state server: GET {base}/tables?ref=<state_ref>&names=a,b answers {"<table>": [rows]}, and POST {base}/query with {ref, queries: [{id, query, params}]} answers {"<id>": value}. Needs reset.state_ref_path.

catalog is the path of the named data queries (state_queries.yaml) the tests use.

Data queries: state_queries.yaml

A test's data checks ("what the data should look like afterwards") read the system through named data queries. Each name and what it reads is an entry in state_queries.yaml, which lives in the scenarios.dir folder (with dir: scenarios, as the platform writes it, .reviewctl/scenarios/state_queries.yaml); point catalog at the same file. The test runner (or the sandbox) tells the platform the names, and each SQL query's :name parameters, every time it takes work; the queries themselves stay with it. A test that names a query it hasn't reported fails to import.

# name: what the reader runs. A name is lowercase letters, digits and _, starting with a letter
claim_count: SELECT COUNT(*) FROM claims WHERE employee_id = :employee_id
claim_amount: >-
  SELECT amount_cents FROM claims
  WHERE invoice_no = :invoice_no

• With kind: sqlite or sql: one read-only SQL query. Write each parameter as :name; the test's where fills it, and a parameter it leaves out stops the read with an error. The value checked is the first column of the first row, or null when there is no row. Return numbers for counts and amounts, and text for statuses.
• With kind: http: the text is sent as it is, as query in POST {base}/query, with the test's where as params; your server answers each id with one value. It can be a name your server knows (claim_count: claim_count) or a query it runs itself.
• No queries yet: {}. reviewctl init writes that, with these examples as comments.

A test names a query in its data checks. In a scenario file:

expected_state:
  - id: claim-saved
    query: claim_count
    where: { employee_id: e_alice }
    op: eq                 # eq, neq, lt, lte, gt, gte, in, exists
    value: 1
    applies: [completed]   # completed, safe_stop or any (the default)

In the JSON a dev AI hands in (submit_answer, in the format get_prompt gives) the same check is "data_checks": [{ "name": "…", "query": "claim_count", "where": { "employee_id": "e_alice" }, "op": "eq", "value": 1, "when": "completed" }]. An attack condition's goal.impact_state and verdict.side_effect_queries name queries the same way. reviewctl scenarios validate checks every name a scenario uses against the file.

The file is a test connection file: on a local test runner change it in the repository; in a sandbox save it with put_wiring_file or on the web. Then pre-run, so the test runner reports the new names before tests use them.

mode: fixture

The built-in mock backend, with no system of yours.

Field	Type	Default	What it does
fixture.pack	name@version	Required	The mock backend, today orders-refunds@1.
fixture.expose	List of mcp, rest	[mcp]	How the agent reaches its tools.
fixture.tool_aliases	Map of your tool name → the mock's	None	When your agent names its tools differently.
recheck_after_ms	0 to 30000	Off	As for own.

judge

The judge model grades replies for the checks that need one. It must differ from agent.model.

Field	Type	Default	What it does
model	Model reference	Required	The judge model. In a sandbox, if it names a model the platform provides, the platform judges with the project's model (models) and bills it as judge model usage. Any other model needs its key as a project secret. On a local test runner, it's called from the machine with the machine's key when the machine has one; without one, a model the platform provides is judged by the platform with the project's model, as in a sandbox, and billed by usage. See judge model keys.
second_model	Model reference	None	The second judge model: when the first can't decide on a reply, it reads it again with the same prompt and quoting rule (see the second judge model). Runs the platform judges in a sandbox don't need it: the platform uses the strongest model it provides from another family than the first, and one written here is used instead. On a local test runner its key has to be on the machine; without it, without a key, or when it's the same as model or agent.model, nothing is read again, and the report says no second judge is set up. When the two judges read something differently, the stronger one decides: strength follows the platform's model catalog (vertex:gemini-4-argon, azure:gpt-6.1-sol, vertex:gemini-3.8-flash, azure:gpt-6-luna, strongest first), and a model outside it ranks below them. When model and second_model are both outside the catalog, which is stronger can't be told, and a disagreement leaves it undecided.
embedding_model	Model reference (openai: or openai-compatible:)	None	For the similarity and relevance checks on a local test runner. Without it they stay undecided there. Runs the platform judges don't need it, because they use the platform's embedding model (similarity checks).
effort	low, medium or high	low	Accepted; not used today.

scenarios

Field	Type	Default	What it does
dir	Path	Required	The folder with the data queries (state_queries.yaml), seeds/ and attacks/. For reviewctl review it also holds the scenario files.
include	List of scenario ids	['*'] (all)	reviewctl review runs only these.
required	List of scenario ids	None	reviewctl review stops with an error when one is missing.

Runs started from the platform use the tests confirmed on the web, not the scenario files in dir.

gate

The pass criteria for runs started with reviewctl review on the test machine. Runs started from the platform use the project's pass criteria instead. Leave key_tasks empty and a local run gives numbers but no verdict.

Field	Type	What it does
key_tasks	List of {scenario, condition, pass_k: {k, min}}	Each key task must pass at a rate of at least min (0 to 1) over k runs.
reliability	{recover_min_rate, max_duplicate_side_effects}	How often the agent must recover from a fault, and how many repeated actions (such as a double refund) are allowed.
security	{block_on, max_attack_success, low_risk}	block_on lists verified_impact and verified_violation; a verified problem of those kinds means it doesn't pass. max_attack_success is the share of attacks in a test set that may get through (default 0). low_risk: warn: a test set's low-severity hits (internal details only, or resting on a scenario the platform assumed) only warn and don't fail it; the platform's default pass criteria for a security test set have it, and without it every hit counts.
regression	{max_functional_drop}	Test sets: the largest drop (0 to 1) in the pass rate of normal questions allowed against the run compared with.
evaluators	List of {scenario, id, min_rate}	Test sets: a check must pass at least this often.
tags	List of {scenario, tag, pass_k: {k, min}}	Test sets: the questions with a tag must pass.
metrics	List of {scenario, metric, min_score}	Scored test sets: a group of checks must average at least min_score.

budget

Field	Type	Default	What it does
per_attempt.minutes	Number	Required	Wall-clock limit of one attempt; past it the attempt is stopped.
per_attempt.model_calls	Whole number	Required	Model calls one attempt may make; the agent gets it as REVIEW_MAX_MODEL_CALLS.
total_model_calls	Whole number	None	Model calls for the whole run; once used up, the attempts left are recorded as not run. An agent that doesn't report its calls is counted at the per-attempt limit each time. On runs started from the platform it is raised to cover every planned attempt.
usd_estimate_cap	Number	None	Accepted; not used today.

cloud

Field	Type	Default	What it does
auto_push	true or false	false	Upload each local reviewctl review run to the platform when it ends (needs reviewctl login).
api_url	URL	None	The platform address reviewctl push uses.

runner_env

Test runner variables: environment variables review.yaml sets for the test runner itself, for values that aren't secret, such as the address of a service the repository starts or a test account of its own test backend. A repository whose test backend comes with its own test accounts then needs no project secrets for them.

runner_env:
  EXPENSE_TOKEN_ALICE: tok-e_alice              # a test employee of the repository's own test backend
  EXPENSE_STATE_BASE: http://127.0.0.1:47591    # the state server its Compose file starts

What reads them	How
The tests' users	A test's user env:NAME reads the variable. A test runner allows the names listed here as identities.
backend.state.reader, backend.reset	base_url_env and url_env name the variable.
Reset commands	They run with the test runner's environment, these variables included.
The agent	Only the names listed in agent.env_passthrough, as for any variable. agent.env is different: it sets variables for the agent only.

environment.setup and the services don't get them.

The agent gets a runner_env variable only when it's listed in agent.env_passthrough. One that isn't listed and that the platform doesn't use either (it isn't the data reader's or the reset's address variable, the agent's model variable or a test user's identity, and neither the reset command nor the scripts it runs mention it) gets a warning when the overlay is saved, under "Test connection file" and in the "Preflight checks": "X in runner_env doesn't reach the agent… To hand it to the agent, add it to agent.env_passthrough". It's a warning only; nothing is refused.

• You can set at most 30 variables. Names use capital letters, digits and underscores, and start with a letter. A value is one line of at most 2 KB.
• A project secret, or a variable set on the test machine, with the same name is used instead of the value here. That way you can override a repository's value without changing the repository.
• A value that looks like a real credential stops the file from loading, and the error names the variable. That covers key shapes such as sk-…, sk_live_…, ghp_…, github_pat_…, xoxb-…, AKIA…, AIza…, GOCSPX-…, re_… and Bearer …, a JWT, a private key (-----BEGIN), and 40 characters or more that look random. Put such a value in project secrets, or in the environment of your own test runner.
• These values aren't hidden. Anyone who can read the repository sees them, and logs and reports don't mask them.
• A sandbox's run log lists the names of the variables set here, without values. reviewctl doctor lists them too, and says which ones the environment overrides.

Update reviewctl on your own test runner

A reviewctl from before 1 October 2026 doesn't know runner_env and refuses the whole file as having an unknown field. Update it, then restart the test runner. In a sandbox, a run whose review.yaml sets runner_env ends at once on a machine image from before that date, and says the reviewctl in the sandbox image is too old for it. Start the run again once the platform's images are updated.

Next steps

• Add project secrets for the variables your file names.
• Sandboxes and limits: classes, architecture and what the machine image contains.
• Command line: reviewctl init writes a first review.yaml and reviewctl doctor checks it.