Datasets

A dataset is the set of rows a version is tested on. Each row has an input, and optionally what a right answer looks like (the expected output), tags, metadata, the conditions it runs under and, for a sandbox, the run environment it needs. Datasets are versioned, so an experiment always says which rows it ran.

Open Datasets in the project.

Kinds of dataset

Origin	Where its rows come from	Edited where
Own	Rows you write, import, generate with AI or add from logs, experiments and reviews	On the dataset's page
From scenarios	The project's confirmed scenarios: one row per scenario, with its conditions and run environment; kept in step with the confirmed versions	In the scenario's editor ("Open in the scenario's editor" on the row)
From a test set	One of the project's test sets: one row per question	In the test set's editor

A project connected before v2 has its scenarios and test sets here already: the platform reads them in the first time the project is opened, and nothing is copied that could drift, because the rows point at the scenario or question they come from.

When scenarios have drafts not confirmed yet, the list shows "N drafts" next to the scenarios' dataset's version; it leads to "All tests (detailed list)" to see and confirm them (automatic approval confirms them itself while it is on). How scenarios and test sets are written, drafted and confirmed: scenarios and test sets.

Create a dataset and add rows

1. Click "New dataset", give it a "Name" (and a "Description (optional)") and click "Create and add rows". To have AI write it, click "Generate a dataset with AI" beside it instead, say "What to test", then "Create, then have AI write rows": the new dataset opens on "Generate rows with AI" with what you wrote already in its "What to test".
2. Add rows in any of these ways:

Way	How
Write a row	"Write a row" opens the row editor.
Import	"Import": paste cells from a spreadsheet, or choose a CSV, TSV, JSON Lines (promptfoo's test files too), Excel (.xlsx, first sheet) or promptfoo YAML file. Say what each column is: "Input", "Expected", "Tags", "Metadata" or "Don't import". Several input columns become one object named by their headers. Up to 5,000 rows and 5,000,000 characters per import, 20,000 characters per cell.
Generate with AI	"Generate rows with AI": the platform's AI writes new rows, each with what a right reply says or does (where the facts can't be known, how the reply should handle them, such as asking for the booking number; nothing made up). A row with no single right answer, one judged on tone alone say, has an empty expected output and says why in its side panel. It writes from "Describe what to test", "Read the repository's code" (a project connected to a GitHub repository) or "From rows that failed lately". Pick how many (3 to 20, 8 by default). It counts against the assistant's daily allowance.
From logs	In Logs, tick traces (or open one) and "Add to dataset".
From an experiment	In an experiment, open a row and "Add to dataset".
From human review	A review queue can add every reviewed item to a dataset, with the reviewer's answers in its metadata.
From the SDK or the API	POST /v2/datasets/<id>/rows with a token that may upload (v2 routes).

Rows written by AI arrive marked "To look at", with why each was written. Open one, change what needs changing and click "Looked at it, keep", or delete it.

"Add to dataset" offers only the project's own datasets, or "New dataset…" with a name. Up to 500 traces or rows at a time.

The row editor

Click a row to open it on the right ("Row details").

Field	What it is
Conversation	"One message", "Scripted turns" (the user's messages, one per line, at least two, each sent after the agent's real reply to the one before) or "Simulated user" (who the user is, what they want done, at most how many turns from 1 to 12, when to stop, and optionally the first message). The simulated user is played by the project's judge model and never sees the expected output or the scorers.
Input	What the AI under test receives.
Expected	What doing it right looks like: the reference answer, the points to make, the fields to read out.
Conditions	"Normal" by default. "+ Fault" adds a tool and how it fails (for example timeout); "+ Attack" adds an attack and optionally a plugin. A row runs once per condition. Faults and attacks run on sandbox targets only.
Tags	Separated by spaces or commas; up to 50. The release gate can require tags to have run (security, say).
Metadata (JSON)	Anything else, as a JSON object; scorers can read it ({{metadata.x}}).
Run environment	Sandbox projects only: the services, seed and identity the row needs, data checks (`name

"Save" writes a new revision of the row; the old one stays in every version that has it. "Delete this row" removes it from the current rows, not from saved versions.

Rows that come from a scenario or a test set are read-only here: "Open in the scenario's editor" (or "Open in the test set's editor") is where they change, and the dataset follows once the new version is confirmed. A newer version not confirmed yet marks the row "New draft"; its side panel says which version the draft is and where AI approval stands, and "See the draft" opens it. Such a row also has "Attachments": the files its test hands the agent (receipts, photos…), with where each came from, and "Upload attachments" to add more (attachments).

Versions

The dataset's current rows change whenever someone edits them. "Save as vN" snapshots them as version N: an immutable list of (row, revision) pairs. Pick a version in the "Version" menu to see it, read-only. An experiment records the dataset version it ran (or "current rows" when it ran on rows not yet saved as a version), and an SDK Eval can ask for one: dataset("support questions", { version: 3 }).

Find rows

• The chips "All", "Functional", "Security" and "Failed lately" filter by tag and by the latest scores.
• The filter box takes the filter language with the fields tag, condition, source, input and output (the expected output), for example tag = billing AND input ~ "refund".
• "Latest scores" shows each row's scores in the most recent experiments that ran it.

Run a dataset

"Start an experiment" on the dataset's page:

• Scenarios and test-set questions start a sandbox (or test runner, or chat AI) run of exactly those, as they are in this version, like "Test new version" does.
• Rows you wrote here run from your code: the page shows the SDK lines, dataset("<name>", { version }) as an Eval's data.

Attack datasets

"Generate an attack dataset" opens the security-test generator: pick a template ("Use this template"), or choose attack plugins and rewriting strategies yourself, preview the questions and save them as a security test set. It shows up as a dataset "From a test set" once confirmed. An attack dataset the platform wrote (redteam-…) has "Write a new set" on its page: the platform writes a new set of attacks of the same kinds from the code, saved as a new version for AI approval. See attack tests.

The datasets page also has "All tests (detailed list)" at the top: every scenario's and test set's versions and drafts, writing scenarios and confirming in bulk; see scenarios and test sets.

Next steps

• Scorers: what scores each row.
• Experiments: run a version over a dataset and compare.
• Scenarios and test sets: the rows a sandbox runs.