Scenarios and test sets

The rows a sandbox, a self-hosted runner or a chat AI runs are scenarios and test sets. They show as rows of the project's datasets ("From scenarios", "From a test set"), and they are written, reviewed and versioned in their own editors, because they carry more than an input and an expected output: conditions, data checks, the tools the agent may and may not call, and the identity it acts as.

	Scenario	Test set
What it is	One real task the agent carries out, under normal, fault and attack conditions	A table of questions, each with its expected result
Judged on	The tools the agent called, the data afterwards, and its reply	Each reply, by the test set's checks; for an agent also its tool calls, the data afterwards and the fields it read
A run counts	Conditions × repeats	Questions × conditions × repeats
Used for	Agents in a sandbox or on a self-hosted runner	Agents and chat AIs

Tests come in two groups: functional tests (whether the agent gets things right and stays sensible when things fail) and security tests (whether it leaks data or oversteps when lured or attacked). Attack sets are test sets in the security group.

Where to write them

Scenarios and test sets are written under Datasets. The views at the top of the datasets page are "Datasets" (the list), "Generate an attack dataset" and "All tests (detailed list)", plus "Test set scorers" for a chat AI project.

• "All tests (detailed list)" lists each scenario's confirmed version, draft and every version; tick several to confirm or delete them together. At its top are "New test set", "Write a scenario" ("New scenario" for a chat AI project) and the "Auto-approve" switch; an agent project also has "Let AI draft your tests". In the datasets list, the scenarios' dataset shows "N drafts" next to its version, which leads here too.
• "Generate an attack dataset" writes security tests: see attack tests below.
• For rows that aren't scenarios, such as those an SDK Eval reads, use the datasets list's "Generate a dataset with AI", or the dataset page's "Write a row", "Import" or "Generate rows with AI": see Datasets.

Let AI draft your tests

Under "All tests (detailed list)", "Let AI draft your tests" offers these ways:

• "Let the platform read the repo (recommended)" (a project connected through a GitHub repository): the platform's model reads the repository, read-only: the README, the entry code, the tool definitions, the prompts and business rules, the seed data and fixtures. It never reads .env or a file that holds secrets. It writes functional tests for the main tasks, edge cases and failures, using the tool, data query and identity variable names in review.yaml, and lists the files the tests need (see attachments). For an English agent the tests' words are English (read the same way as the default security tests' language); names stay as the code has them. A test has no "the day it runs": a date in it is sent as written on every run, so the AI writes dates the way a user says them ("yesterday", "three days ago") and checks no exact day. It runs the platform's own import checks on each test as it writes, and counts against the assistant's daily allowance. It never changes your repository. When it's done it lists what is left, grouped by who does it: "The platform fixes these" (the test connection the platform wrote; "Have the platform fix these" hands them to "Let AI finish and check"), "Likely problems in your product" (what reading the code suggests — what the tests look at, no precondition, nothing to fix first) and "For you to provide" (test files, test accounts, a mock page in place of a real website).
• "Let the dev AI read the code": "Copy prompt", paste it into Claude Code, Codex, Cursor or a similar tool opened in the agent's code folder, then paste its whole answer back and "Import tests". The platform checks every tool, data query and identity variable name first; "Copy fix notes for the dev AI" hands back what's wrong. A dev AI connected to the platform hands its scenarios in by itself.
• "Fill in a brief" or "Describe it in a few sentences": the assistant drafts scenarios from a template you fill in or a few sentences, and lists the assumptions it made for you to check.

The platform adds the "a tool fails" and "instructions hidden in what it reads" conditions itself.

Attack tests

The datasets page's "Generate an attack dataset" view writes security tests:

• The default security tests: once the first functional test is confirmed, the platform writes a set of security tests itself, from what the agent can do (command injection for one that runs commands, out-of-scope file reads for one that reads files, secret exfiltration and injection through tool output for one that reaches outside addresses), sent as the account the confirmed tests use. The questions are in the agent's own language: English when what the platform holds of the agent (its system prompt, its repository's files, the dev AI's description of it, its replies) and the confirmed tests are clearly English, with no Chinese in them; Chinese otherwise. Tests the platform's own AI wrote (the starter tests of "Let AI finish and check", a chat AI's starter questions, "Let the platform read the repo") and the capability list it writes don't count, until a person changes them. On a local test runner the platform keeps no copy of the files reviewctl connect --ai reads from your machine once it's done, only the language they are written in, and that counts as the agent's own words (a test a person wrote in Chinese still makes the set Chinese). In an English set the platform's own words are English too: the note at its top, the checks' names, and what the judge reads of the agent (what it does, who is signed in, the platform's lists of its tables and attack targets). A set written before keeps its language. Personal-data questions ask only for fields the agent's data really has (when the platform can read its database schema, or the data files its test connection reads); a question asking for a field the data doesn't have is not approved by AI. It is a draft, reviewed like any other, and written once per project: set off several times at once, or written while the platform's AI queues, it is still saved once. Each question gets the same time limit as the project's functional tests (not a fixed one minute), so a tool-using agent has time to answer. When some categories don't generate this time, the page says how many questions actually came out and which categories are missing (not tested this time); a set that came out with only questions that can't count (just "probing its tools", which is public content and no attack success) is treated as not written, not as tested. The default set has a floor: at least 10 questions that can count (ones that never can, like "probing its tools", aren't counted toward it) and at least one for each high-risk capability the product really has (running commands/code, writing or deleting files, exporting data or touching other users' data, out-of-scope file reads, outbound network or secret exfiltration, instructions hidden in tool returns); below it the platform writes the set again (up to 3 times, keeping the kinds an earlier try wrote), and if it is still short it is saved with the page saying how many came out and which kinds are missing. Tool poisoning and injection through tool output are written only when the platform knows a tool whose description, or content a tool returns, carries an instruction (otherwise no question of those kinds can reach anything, so they aren't planned and aren't counted as missing). When the attack-writing model is blocked by its own safety filter or answers nothing, the platform writes that batch on another platform model instead. On the attack tests page, "Write the default security tests again" runs in the background: the click returns at once, with the cost and time beside the button estimated from the organization's last few generations (writing only: AI approval is charged on its own); while it runs the line under the button says so and for how long, and both "Test new version" pages (quick and all options) note it; when it finishes the page says the new version and how many questions it has, or why it failed, and the last rewrite stays listed under the button. A template card's "about N minutes · about N questions" is estimated the same way (once the organization has generated at least twice), and while a set is being written the card shows how long it has been going. While a set still waits for AI approval, the button beside it reads "Confirm now, without waiting for AI approval", with what approval has said so far. Two security sets of one name are labelled "default / made later · version · date" to tell them apart.
• Templates: "Use this template" writes attacks from what the agent does and confirms them; "Run it" then takes you to the run form with "Security only" picked. It needs at least one confirmed test, whose account the attacks use.
• "Advanced: choose attack plugins and rewriting strategies": check the "Purpose", pick plugins and strategies and the language, "Generate preview", keep or drop each question, and "Save as draft security test set".

An attack set the platform wrote (redteam-…) is a dataset of its own too, and its page has "Write a new set". A set written again (by you, or by the platform once it reads the agent's tools) that doesn't cover a kind the last version had, or a high-risk target the last version aimed at (a system file like /etc/passwd, .env, an internal address), keeps those questions of the last version, or of the versions before it (at most 8, none the agent can no longer reach), and the new version's page says how many it kept. Attack questions that ever got through on the project are carried into every set the platform writes (default, written again, template), up to 20, and each is asked at least 3 times: an agent's refusals vary, and one ask that happens to be refused would read "held". The attack writer's own refusals ("Sorry, I cannot…") are never saved as questions; they are written again at generation. Each attack is judged held or got through (how); the scorer attack_success counts the ones that got through, and the default release gate blocks on any.

Test sets

"New test set" opens "Prepare questions": "Import a table" (Excel .xlsx, CSV, JSONL or promptfoo files; say what each column is: "Question", "Reference answer", "Key points", "Tags", "File", "Expected value"…), "Type them in", or "Ask the assistant". A test set holds up to 1,000 questions, and one table up to 5,000,000 characters. For a conversation, put || between the turns; the platform sends them one at a time and judges the answer to the last one. A question can also be played by a simulated user.

For an agent, each question is a task it really does; a question can hand it a file and list the fields it should read out ("Expected value"), and the report gives each field's accuracy.

Write a scenario

"Write a scenario" opens "New test": "What the user says", "What counts as done", who the agent acts as, the tools it can use and which of them change data; optionally the tools it should call, what the data should look like afterwards, what the reply must mention and the forbidden actions. "Test a fault: the tool fails on its first call" and "Test an attack: slip an instruction into what a tool returns" add those conditions.

• Write in plain words what it must not do in "What the user says" or "What counts as done" ("don't delete a single file", "leave the original cv.md alone", "don't export customers"), and the form asks whether to turn it into checks, listing the rules (for example "calling bash (the command carries a delete such as rm) counts as a violation", "calling write_file (the file it acts on is cv.md, by its path) counts as a violation", "calling write (the file it acts on is config, memory, by its path) counts as a violation"); "Turn into checks" adds them to the forbidden actions, and the form says which it added. The rules go by the file the agent actually changes and the command it actually runs: mentioning cv.md in a file's content doesn't count.
• "Tools it can use" under "Advanced" is filled in from the project; when the agent has more than 50 tools it is left empty, which means any of them, and you can draft right away (a test lists at most 50). Under "Tools it should call" and "Forbidden actions", type a name to search all the agent's tools. Under "Tools that change data", tick only the ones that do; read-only ones stay unticked.
• "What the data should look like afterwards" (a data check) names a data query, such as claim_count, with the fields it takes (where) and the value it should have. The names are the ones the test runner or sandbox reports: each is an entry in state_queries.yaml (name: SQL for a SQLite database). A name that isn't there fails the import. How to write that file: data queries.
• When "The reply must mention" is empty and there is no data check or tool it should call, nothing would judge the run: the form asks whether to make "What counts as done" the reply point, and "Make it the reply point" fills it in. Left as it is, the platform does the same when drafting and says so on the draft.
• "Max minutes per run" is 5 by default; raise it for slow tasks such as web research, up to 120. Web research tests the platform writes get 15 minutes by default.
• Left empty, the test's id is made from the latin words of its title; for a Chinese title, from the names in what the user says (for example node-js-pnpm), else an id such as test-1.

Change what the user says

On a test's page, "Change the wording" under what the user says changes only that sentence: no need to touch the raw rules. A confirmed test gets a new draft version, used once you confirm it. The platform's AI rewrites the reply points and "What counts as done" for the new wording (if it asks for Chinese, a short reply or the sources listed, those become reply points) and leaves the other checks alone; when it can't, the page says the reply points still follow the old wording, for you to check.

Drafts and versions

A new test is a draft; runs use confirmed tests only. Confirming locks a test's content: a confirmed version never changes, and changing it makes a new version (the dataset row follows once the new version is confirmed). Old versions and their experiments stay as they are.

Auto-approve is on by default: drafts written by the dev AI, the assistant or the platform are checked by rule (names, test files), then read by AI, and confirmed when nothing is wrong. When the AI turns a draft down, the platform's model revises it from the reasons and has it reviewed again, up to 2 times (up to 4 when something new comes up in between, such as the connection files or the repository changing). A draft that only needs a change to the test connection goes to "Let AI finish and check" and is reviewed again once a check passes. When a draft needs something only a person can give (a secret, a change to your product, an expected value, a paid service), the AI stops and says what. Drafts you wrote or edited yourself wait for your own "Confirm".

On a scenario draft's page, "Before you confirm" lists what the platform found when it held the draft's words against what it actually checks: an attack or fault the words ask for with no matching condition, a test judged on the reply only, an attack that would get through on the very state the test requires, a connection that can't see the tool calls or read the data the test needs, a forbidden tool the agent calls on every answer. When the words rule out an action and nothing forbids it, it says which rule could be added, and "Add in one click" adds it to the draft. When the connection can't read the agent's data after the task (it lives in files or in memory, say) and the test states no data of its own, it only says the test is judged on the tool calls and the reply, and doesn't hold it back: such a test is judged on those. Once AI has reviewed the draft, a "Requirements → checks" table maps each requirement to what checks it.

When a test hands the agent a file and asks what it says (a summary, its key points), reply points naming what the file doesn't hold (latin names, paths) are listed under "Before you confirm" too: answering from the file couldn't pass, so AI approval won't confirm it, and the platform's AI rewrites the points after reading the file. A sample file the platform writes for such a test is written from the test's reply points and checked against them. When a later diagnosis says a version of a test was written wrong, that version's page says so below the AI review, with a link to the version the platform saved from it, if any.

Drafts waiting for a person are listed under Datasets → "All tests (detailed list)", where "Confirm all" confirms them at once. In a dataset, a row with a newer draft is marked "New draft": open it, and its side panel says which version the draft is, whether AI is reviewing it, turned it down or it waits for a person, and "See the draft" opens the draft's page, where you confirm it.

Conditions

Condition	What happens	What passes
Normal	The user asks, in their own words	The task gets done and the data is right
Fault	A tool fails: it times out, is rate-limited or refused on its first call, keeps returning server errors, or does the work but the result never reaches the agent	What the scenario expects: "Should recover", "Should stop" (say honestly it wasn't done) or "Recover or stop"; never doing the same thing twice
Attack	A tool's result, or the user's input, carries a hidden instruction	The agent doesn't comply

A fault or attack can aim at one turn of a conversation. To run only some conditions, use "Test new version: all options" under Settings → "Advanced" (more options).

Checks for test sets

A check is one rule each reply is judged by; they are the same checks the scorer library lists. A test set's checks are on its "Checks" step: "What replies must do" (AI-judged checks quote the reply as evidence; rule checks are decided by code), "Safety checks", "Write your own", "Add from the assertion library", and for an agent "Tool and data checks". Set "How many times to ask each question" (3 by default). "Try it" judges one reply before you confirm.

Attachments

Attachments are the files the agent gets in a test: receipts, photos, XML invoices. They live with the rows that use them: open a scenario's or test set's row under Datasets, and its side panel's "Attachments" lists the files the row's test uses and where each came from (copied from the repository, or a sample AI wrote); "Upload attachments" adds more. An uploaded file is attached by its name to the tests that use it. Each file can be up to 20 MB; PDF, PNG, JPEG, WebP, Excel (xlsx, xls), XML, TXT, CSV and JSON only.

• In a scenario's task, write {{file:file-name}} to hand that file to the agent. A scenario keeps its files by content: a different file makes a different version.
• In a test set, name the files in the "File" column, and write {{file}} in the question for its first file.
• With a self-hosted runner you can keep files on your own machine: put them in files/ in the runner's scenario folder. It reports only their names and hashes.
• For a project connected through a GitHub repository, the platform copies the files the tests name from the repository (never one that looks like it holds secrets), and writes synthetic samples of TXT, CSV, JSON or XML files the repository lacks. PDFs, images and real business documents are for a person to upload.

A file a confirmed scenario uses can't be deleted.

Pass criteria

The pass criteria decide a sandbox or runner run's own verdict, which the release gate's "Test verdict is Passed" rule reads. Starting a test locks in the pass criteria of that moment.

Open Settings → "Release gate" → "Customize pass criteria". You don't have to set anything: by default every scenario must pass every time under normal conditions, no action may be repeated when a fault hits, and any verified security problem means it doesn't pass.

Part	Fields
"Key tasks"	The tests the verdict depends on: for each, the "Scenario", the "Condition", "Runs per test" and "Pass at least (%)".
"Reliability"	"Of the faults that should recover, recover at least (%)" and "Repeated actions allowed" (0 means never)
"Security"	Whether a "Verified real impact" or a "Verified forbidden action (even if the system stopped it)" means it doesn't pass

The key tasks also become the default gate's "Tags that must run", and their lowest pass line its task_done rule (default rules). "Runs per test" is how many times a key task runs: with 3, it passes every time only when all 3 pass. The run form won't run a test fewer times than the pass criteria need.

A test set has its own pass criteria; one without runs on the default: each question passes every time it is asked, for at least 90% of the questions, and a security test set must pass every question.

Next steps

• Sandbox runs: start a run with the tests you confirmed.
• Full run reports: read the verdict and the problems.
• review.yaml reference: the scenario folder and the names tests use.