v2 API routes

Every route of the /v2 API: datasets, scorers, experiments, the release gate, traces, online scoring, alerts, the Playground and human review. Tokens, scopes, errors and the /v1 routes CI uses are in API: tokens and errors; the OTLP and Gateway endpoints in OpenTelemetry and the AI Gateway.

All routes are under https://<platform address>/v2/, take and return JSON, and need Authorization: Bearer <token> (or the browser's session).

Who may call a route

Mark	Who
read	A member of the organization (browser session or CLI token rt_…), or the project's token pj_…
ingest	A member, or a project token with the upload scope ("Allow uploading logs and experiments")
member	A member only; a project token gets 403 project_token_not_allowed
person	A member signed in in the browser, clicking; a CLI token, project token, the assistant or a coding AI gets 403 session_required

A project token works for its own project only. An object of another organization, or of a deleted project, is 404 not_found, never 403.

Paging and filters

List routes take limit (default 50, at most 500) and cursor, and answer next_cursor: pass it back as cursor for the next page, until it is null. With a filter, a page is read from at most 5,000 items, so it can come back short (even empty) with a next_cursor: keep going while next_cursor isn't null. A cursor not made by a page is 400 bad_cursor. filter takes the filter language (at most 2,000 characters); a wrong one is 400 bad_filter with its position.

Datasets

Route	Who	Takes	Answers
GET /v2/projects/:p/datasets	read	limit, cursor	{datasets, next_cursor}, newest first; each with its last_experiment
POST /v2/projects/:p/datasets	ingest	{name, description?}	201 the dataset
GET /v2/datasets/:id	read		The dataset with its versions
PATCH /v2/datasets/:id	member	{name?, description?}	The dataset
GET /v2/datasets/:id/rows	read	version, filter, limit, cursor	{rows, next_cursor}; 404 version_not_found
POST /v2/datasets/:id/rows	ingest	{rows: [...]}, 1 to 1,000: a row without id is added, one with id gets a new revision. Fields: input, expected, metadata, tags (≤ 50), source, conditions (≤ 20), turns (≤ 50), simulated_user, env	{rows: [{id, revision, created}]}; 409 edit_in_scenario_editor for a scenario's row
DELETE /v2/datasets/:id/rows/:row	member		{id, revision, deleted: true}
POST /v2/datasets/:id/versions	ingest	{note?}	201 {dataset_id, version, created_at, created_by, note?, row_count}
GET /v2/datasets/:id/scores	read		Each row's latest scores, by row id
POST /v2/datasets/:id/import/preview	member	{text, format: "auto" | "promptfoo_yaml"}, text up to 5,000,000 characters	{format, total, columns, rows} (the first rows)
POST /v2/datasets/:id/import	member	The same, plus mapping: {header, columns: ["input" | "expected" | "tags" | "metadata" | "skip", …]}	{added, rows, problems, problem_count}; at most 5,000 rows
POST /v2/datasets/:id/rows/from	member	{experiment_id, row_keys} or {trace_ids}, 1 to 500	{rows}
POST /v2/datasets/:id/generate	person	{mode: "describe" | "repo" | "failures", description?, count} (1 to 20, default 8)	{rows, model, usd}; 429 assistant_allowance when the day's allowance is used up

A dataset row: {id, dataset_id, revision, input, expected?, metadata?, tags, source: {kind, ref?}, conditions?, turns?, simulated_user?, env?, scenario_ref?, question_ref?, created_at, created_by}. A condition is {kind: "normal"}, {kind: "fault", tool, fault} or {kind: "attack", vector, plugin?}.

Scorers

Route	Who	Takes	Answers
GET /v2/scorers/builtin	read		{scorers}: every built-in with its name, kind, direction, what it reads and its settings' schema
GET /v2/projects/:p/scorers	read	archived=0|1, limit, cursor	{scorers, next_cursor}, the organization's shared ones included
POST /v2/projects/:p/scorers	member	{name, description?, direction?, shared?, kind, config}	201 the scorer; 422 invalid_scorer
GET /v2/scorers/:id	read	version	The scorer (its latest version, or the one asked for)
PATCH /v2/scorers/:id	member	{name?, description?, config?, direction?, archived?}	The scorer's new version
POST /v2/scorers/:id/try	member, not the assistant	{version?} and a row: {input?, output?, expected?, metadata?}, {experiment_id, row_key, trial?} or {trace_id}	{result, row}
POST /v2/projects/:p/scorers/try	member, not the assistant	{scorer: {kind, config, direction?} | {id: "builtin:…", config?}} and a row	The same, for a scorer not saved yet
GET /v2/scorers/:id/calibration	read	version	{scorer_id, version, set, status, results}
PUT /v2/scorers/:id/calibration	member	{rows: [{input?, output?, expected?, metadata?, human_value}]}, 1 to 500	The calibration
POST /v2/scorers/:id/calibration/run	member, not the assistant		202, the calibration running
GET /v2/projects/:p/scorers/stats	read		Per scorer: experiments that used it, whether the gate names it, the last 7 days, calibration

kind and config: llm {prompt, output: "score" | "pass_fail" | {choices: [{label, value}]}, model} · code {language: "js", source} · heuristic {fn, params} · trajectory {mode} · agent {check} · human {key, scale: {min, max} | {choices}}. See Scorers.

Experiments

Route	Who	Takes	Answers
GET /v2/projects/:p/experiments	read	filter, limit, cursor	{experiments, next_cursor}, newest first
GET /v2/experiments/:id	read		The experiment
POST /v2/projects/:p/experiments	ingest	{name, source?: "sdk" | "playground", target?, dataset?: {id, version?}, scorers?, baseline_id?, environment?, git?: {sha, branch, pr}, metadata?}	201 the experiment, with url and the baseline_id it will be compared with
POST /v2/experiments/:id/rows	ingest	{rows}, 1 to 500; each {row_key?, dataset_row?: {id, revision?}, input, expected?, condition?, tags?, trials: [{trial?, output?, turns?, scores, trace_id?, duration_ms?, cost_usd?}]}. An Idempotency-Key header makes a retry safe.	{rows: [{row_key, trials}]}; 409 experiment_finished, 409 derived_experiment
POST /v2/experiments/:id/finish	ingest	{status?: "done" | "cancelled" | "failed"}	The experiment
GET /v2/experiments/:id/rows	read	baseline, filter, limit, cursor	{rows, next_cursor, baseline_id}; each row with the baseline row's scores and the delta
GET /v2/experiments/:id/compare	read	baseline	{experiment_id, baseline_id, comparable, why, judge, scorers, rows: {improved, regressed, same, new, dropped}}
POST /v2/experiments/:id/scorers	member, not the assistant	{scorers}, 1 to 50 platform scorers	202 {experiment_id, scorers, pending}; 409 derived_experiment for a sandbox or runner experiment
GET /v2/projects/:p/experiment-changes	read	ids (comma-separated, up to 60)	Rows improved and regressed against each one's baseline
GET /v2/runs/:run_id/experiment	read		The experiment a run belongs to; 404 no_experiment for a pre-run
GET /v2/projects/:p/baseline	read		{project_id, baseline_id}
PUT /v2/projects/:p/baseline	person	{experiment_id} (or null)	{project_id, baseline_id}; 422 baseline_not_usable for one not finished

A score in a trial: {scorer, value, label?, undecided?: {code, reason, reason_en?}, rationale?, judge_model?, cost_usd?}. On a row derived from a run, reason_en is the English of a reason the platform wrote in Chinese. scorer is local:<name> for one scored on your side, builtin:<slug>, or sc_…@<version>; value is 0 to 1, or a count for a lower-is-better check, or null when undecided. Platform scorers named in scorers score the rows in the background.

Release gate

Route	Who	Takes	Answers
GET /v2/projects/:p/gate	read		{project_id, rules, updated_by, updated_at, by_person}
PUT /v2/projects/:p/gate	person	{rules}, up to 50, ids unique	The gate; the recent experiments are judged again
GET /v2/experiments/:id/gate	read	wait (seconds, at most 60)	The gate's result (below)
POST /v2/experiments/:id/gate/preview	read	{rules}	What those rules would say, nothing saved

A rule: {id, kind: "scorer", scorer, stat: "mean" | "nonzero", op: ">=" | "<=" | "=", value, blocking, low_risk?: "warn"} (low_risk: low-severity hits only warn, left out of the rule), {id, kind: "release_verdict", blocking: true}, {id, kind: "no_regression", max_rows, blocking}, {id, kind: "undecided_max", fraction, blocking} or {id, kind: "required_tags", tags, blocking}.

GET /v2/experiments/:id/gate answers {experiment_id, status: "pass" | "blocked" | "none", rules: [{rule, ok, actual, rows?, skipped?, low?: {rows, what}}], baseline_id?, computed_at, experiment_status, final, conclusion: "success" | "failure" | "neutral"}. low is the low-severity rows a rule with low_risk: "warn" left out, and what they are: a passed gate with it is "passed, with N low-severity problems", still concluding success. With ?wait= (even 0) the platform holds the request up to that many seconds, answers as soon as something changed or the gate is final, and adds report: each scorer's mean, baseline, change, rows improved and regressed, the undecided rows and the top problems. Ask again until final is true. The id a CI launch answers can be read before its first run is in (experiment_status: "pending"). An SDK or Playground experiment's gate has pending: true, is not final and concludes neutral while the platform's scorers are still scoring its rows. See CI.

Traces

Route	Who	Takes	Answers
POST /v2/projects/:p/traces	ingest	{environment?, spans}, 1 to 1,000 spans	202 {accepted, traces}; 422 unknown_experiment
GET /v2/projects/:p/traces	read	since, until (ISO times), env (comma-separated: dev, staging, prod, ci, experiment), filter, limit, cursor	{traces, next_cursor}, newest first
GET /v2/projects/:p/traces/facets	read	filter, since, until (default the last 7 days, at most 31), env	Counts by status, environment, source, model, tool, tag, version and score buckets; up to 50 values each
GET /v2/traces/:trace_id	read		The trace and its spans in tree order, each with its depth
GET /v2/traces/:trace_id/spans/:span_id/payload	read	part=input|output	A large payload kept in storage, as JSON

A span: {trace_id (32 hex), span_id (16 hex), parent_span_id?, name, kind?: "agent" | "llm" | "tool" | "retrieval" | "embedding" | "chain" | "other", start, end (ISO with offset), status?: "ok" | "error" | "cancelled" (the caller called it off), model?, tokens_in?, tokens_out?, tokens_cached?, cost_usd?, attributes?, input?, output?, environment?, experiment_id?, row_key?, metadata?}. Inputs and outputs over 32 KB go to storage, and the trace answers them as {$blob, size, url}.

Online scoring

Route	Who	Takes	Answers
GET /v2/projects/:p/online-rules	read		{rules}, each with today: {day, spent_usd, scored, capped_at}
POST /v2/projects/:p/online-rules	person	{name, filter?, environments?, sampling, scorers (1 to 10), daily_cap_usd?, enabled?}	201 the rule; 409 too_many_rules past 20
PATCH /v2/online-rules/:id	person	Any of the same	The rule
DELETE /v2/online-rules/:id	person		{id, deleted: true}
POST /v2/online-rules/:id/backfill	person	{limit?} (1 to 1,000, default 100)	202 {rule_id, queued}; 409 rule_off, 409 rule_capped
GET /v2/projects/:p/online-rules/preview	read	filter, environments, sampling, scorers	The last 24 hours: traces, matching, sampled, cost per trace and per day per scorer
GET /v2/projects/:p/online-stats	read		The last 7 days against the 7 before: traces, error rate, cost per 1,000, p95 latency, each scorer's mean, 14 daily means

Alerts

Members only: alerts name people and addresses.

Route	Who	Takes	Answers
GET /v2/projects/:p/alerts	member		{alerts}
POST /v2/projects/:p/alerts	person	{name, condition, channels (1 to 3, one of each kind), enabled?}	201 {alert, webhook_secret?}: the signing secret, this once
PATCH /v2/alerts/:id	person	Any of the same	{alert, webhook_secret?}
DELETE /v2/alerts/:id	person		{id, deleted: true}
POST /v2/alerts/:id/test	person		The test message sent, with each channel's delivery
GET /v2/alerts/:id/events	member		{events}: the last 50

A condition: {kind: "score_mean", scorer, window: "1h" | "24h", below} (0 to 1), {kind: "error_rate", window, above} (a share, 0 to 1), {kind: "cost_day", above_usd}, {kind: "gate_blocked"} or {kind: "rule_capped", rule_id?}. A channel: {kind: "email", user_ids}, {kind: "slack", url} (or {kind: "slack", keep: true} to keep the saved address) or {kind: "webhook", url, keep_secret?}; answers show a Slack address and a webhook secret only by a hint. Up to 50 alerts per project. The webhook's body and signature: alert webhooks.

Playground

Members only, never a project token.

Route	Who	Takes	Answers
GET /v2/projects/:p/playground/sessions	member		{sessions, target} (up to 100)
POST /v2/projects/:p/playground/sessions	member	{name?, columns?, rows?, sample?, scorers?}, or {from: {experiment_id, row_key} | {trace_id}}	201 the session with its cells and estimate
GET /v2/playground/sessions/:id	member		The session
PATCH /v2/playground/sessions/:id	member	{name?, columns? (1 to 6), rows? (≤ 100) or sample?: {dataset_id, version, n, seed}, scorers? (≤ 20)}	The session
POST /v2/playground/sessions/:id/run	person		202; 409 run_running, 422 no_rows, 422 model_unavailable, 422 target_needs_scenarios
POST /v2/playground/sessions/:id/experiments	member		201 {experiments, baseline_id} (200 when already saved); 409 not_run

Human review

Members only, never a project token.

Route	Who	Takes	Answers
GET /v2/projects/:p/review-queues	member	archived=1	{queues}
POST /v2/projects/:p/review-queues	member	{name, description?, scorers (human, ≤ 20), required, blind, assignees, problem_tags, dataset_id, calibrate: [{scorer, from}], feed: {source: "logs" | "experiments", filter, per_day} | null}	201 the queue
GET /v2/review-queues/:id	member		The queue
PATCH /v2/review-queues/:id	member	Any of the same	The queue
DELETE /v2/review-queues/:id	member		{id, archived: true}
POST /v2/review-queues/:id/items	member	{trace_ids} or {experiment_id, row_keys} (1 to 500), assignee?	{added, already, items}
GET /v2/review-queues/:id/items	member	status: pending | done | skipped, mine=1, limit (default 100, at most 500), cursor	{items, next_cursor}
GET /v2/review-queues/:id/next	member		{item}: the next one for you, or null
GET /v2/review-items/:id	member		The item, with what it shows
POST /v2/review-items/:id/submit	person	{answers: {"sc_…": value}, tags, note}	The item; 422 required_missing
POST /v2/review-items/:id/skip	person		The item; 409 already_reviewed
POST /v2/review-items/:id/reassign	member	{assignee} (or null)	The item