Scorers

A scorer turns one row's output into a score: a number from 0 to 1 (higher is better), or for a few checks a count of problems found (lower is better). A score the scorer can't give is undecided, with its reason, and never counts as passed.

Open Scorers in the project. The list shows the project's own scorers and the built-ins its experiments or its release gate use ("Show all N built-in scorers" lists the whole library), with what each is "Used in", its "Calibration" and its "Last 7 days".

Kinds

Kind	What it does	Settings
LLM judge	The judge model reads the row and answers	"Judge prompt" with {{input}}, {{output}}, {{expected}} and {{metadata.x}}; "Output": "Score 0 to 1 + reasoning", "Pass / fail" or "Choices (a value each)", one name = value per line; "Judge model": "Follow the project" or a model of the platform
Code	Your JavaScript function	({input, output, expected, metadata}) => number or {score, rationale} (async allowed). It runs isolated, with no network, no process, require or fetch, 1 second of CPU and 64 MB of memory.
Rule	A deterministic check	"Rule": exact match, contains, regex, valid JSON (optionally against a JSON Schema), JSON fields match, edit similarity, similar meaning (embeddings) or length, with its "Parameters (JSON)"
Trajectory	The tool calls against the expected calls	"Matching": "Strict" (same calls, same order), "Unordered", "Subset" (no call beyond the expected ones), "Superset" (every expected call made), or the judge model decides whether the steps make sense
Agent behaviour	What a sandbox or test-runner run found	"Check": task_done, dup_side_effect, attack_success, recover_or_stop, data_check or forbidden_action. Its value comes from the run's report: a row on its own gives no score.
Human	A score people give in human review	"Score key" and "Scale": a range ("Lowest" to "Highest", 1 to 5 by default) or "Choices", one per line

Every kind has a "Name", a "Description (optional)" and a "Direction": "Higher is better" or "Lower is better (counts problems found)".

Create, try and save

1. Click "New scorer" (it starts as an LLM judge with a default prompt), or open a built-in and click "Copy as project scorer" to start from it.
2. Pick the "Kind" and fill in its settings.
3. Under "Try it", "Paste a row" (input, output, expected, metadata) or pick a row of a recent experiment, and click "Try". You get the score with its reasoning, the judge model and what the try cost. Trying uses the settings on screen, saved or not.
4. Click "Save". Each later change is saved as a new version ("Save as v2", "Save as v3"…); experiments keep the version that scored them, written sc_…@2.

"Archive" takes a scorer off the list; the scores it already gave stay. A judge call is judge usage, at the judge model's list price plus 30% (pricing); a try is a person's click and is refused while the organization's credit is used up.

Calibration

An LLM judge should agree with people. Its calibration set is rows a person scored: "Import a calibration set", paste a table whose header has input, output, expected, metadata and human_value (0 to 1), or choose a CSV, TSV or JSON Lines file, and "Save the calibration set". Then "Recalibrate": the judge scores every row and the page says on how many rows it agreed with the person, for this version and the earlier ones, so you see whether a prompt change helped. The list's "Calibration" column shows it (green from 95%, amber from 80%).

A review queue can add every reviewed item to an LLM judge's calibration set, with the reviewer's score as human_value.

The built-in library

Built-ins have the id builtin:<slug> and need no saving: name them in an SDK Eval ('builtin:factuality'), in an online scoring rule or in a gate rule. Open one to see what it reads and its settings, and try it with "Settings (JSON, for trying)".

Agent behaviour (from sandbox and test-runner reports):

Slug	Name	Better
task_done	Task done	Higher
tool_match	Tool calls match	Higher
dup_side_effect	Duplicate side effects	Lower
attack_success	Attack succeeded	Lower
recover_or_stop	Recovered or stopped safely: after a tool's fault, doing what the user asked with another of the test's write tools and saying truthfully what happened is a recovery too (never "didn't stop", let alone a false completion), unless the test requires a stop	Higher
data_check	Data checks	Higher
forbidden_action	Forbidden actions	Lower
factuality	Factuality	Higher (also an LLM judge on any row)

The platform's checks (the same checks test sets use): llm_criterion, reference, key_points, contains, not_contains, regex, json_valid, length, safety, tool_called, tool_not_called, field, state_check, equals, starts_with, contains_any, contains_all, is_json, contains_json, is_xml, contains_xml, is_html, contains_html, levenshtein, rouge_n, bleu, gleu, similar, latency, tokens, cost, is_refusal, tool_sequence, tool_args_match, step_count, tool_call_f1, g_eval, closed_qa, answer_relevance, context_faithfulness, context_recall, context_relevance, conversation_relevance, goal_success, conversation_goal, redteam.

From open-source libraries (MIT; prompts kept word for word):

From	Slugs
autoevals	exact_match, levenshtein_score, numeric_diff, json_diff, list_contains, summary, sql
openevals	correctness, conciseness, hallucination, plan_adherence, code_correctness, toxicity (lower), pii_leakage (lower), prompt_injection (lower), groundedness, retrieval_relevance, json_match
agentevals	trajectory_strict, trajectory_unordered, trajectory_subset, trajectory_superset, trajectory_accuracy

What a scorer reads: input, output, expected, the trace's spans and tool calls, and metadata (context for RAG checks, plan, system_prompt, canary, latency_ms, usage, expected_calls, simulated_user).

Why a score is undecided

Code	Why
judge_none	No judge model is available for this scorer.
judge_failed	The judge's call failed, after its retry and the second judge.
judge_budget	The run's judge spending limit was reached.
rule	A rule couldn't decide (a regex timed out, no embedding model).
missing_input	The row lacks what the scorer reads (an expected output, context, tool calls…).
empty_reply	There is no output.
bad_config	The scorer's settings are wrong.
code_error, code_timeout, code_memory, code_result	A code scorer threw, ran past 1 s, passed 64 MB, or returned no valid score.
report_only	An agent-behaviour scorer on a row with no run report.
human	Waiting for a person's score.

In an SDK Eval, a local scorer that throws is scorer_error, one that returns nothing is skipped, and one outside 0–1 is bad_score; a task that throws makes the row task_error.

Next steps

• Experiments: scores per row, against the baseline.
• Release gate: rules on a scorer's mean.
• Human review: human scorers and calibration from reviews.