Quickstart: an Eval with the SDK (in development)

In development

This way is in development, so you can't choose it on the platform yet. Until then, connect a GitHub repository or use a local test runner. Projects already connected this way keep working, and the steps below still apply to them.

Run your agent over a few inputs in your own process, score each output, and upload the result as an experiment. About 5 minutes.

The task runs on your machine or in your CI, with your own keys: the platform never calls your agent here. What it gets is each row's input, output and scores, plus the traces of the calls you wrap.

1. Install the SDK from the platform

The TypeScript package @preworld/sdk and the Python package preworld aren't on npm or PyPI: the platform serves them itself, at https://<platform address>/downloads/, with a new version each release. Targets → SDK in a project shows the two lines below with the platform's address and the current version filled in: copy them from there. Or look up the current version yourself:

curl -fsS https://<platform address>/downloads/sdk.json     # "version", and the file names under "npm" and "python"

Then, in your project's folder:

TypeScript

curl -fLO https://<platform address>/downloads/preworld-sdk-<version>.tgz
npm install -D ./preworld-sdk-<version>.tgz tsx     # tsx runs *.eval.ts files

Python

curl -fLO https://<platform address>/downloads/preworld-<version>.tar.gz
pip install ./preworld-<version>.tar.gz             # or: uv add ./preworld-<version>.tar.gz

Commit the downloaded file with your code: package.json (or your requirements) points at it, so npm ci or pip install in CI installs that same file, even after the platform has moved on to a newer release. To upgrade, download the new file and install it the same way. Each file has its checksum beside it (<file>.sha256). npm and pip fetch the SDK's open-source dependencies (OpenTelemetry) from their own registries.

2. Make a project token that may upload

1. Create a project ("New project" in the sidebar, then "Agent that calls tools"), or open one.
2. Open Automations → API tokens. Enter a name such as laptop, tick "Allow uploading logs and experiments" and click "Create token".
3. Copy the token (pj_…). It is shown only this once.

Then set four variables in the shell you run the Eval from:

export PREWORLD_API_KEY=pj_your_token_here        # the token you just made
export PREWORLD_PROJECT=prj_your_project_id       # in the project's address: /p/<this>/…
export PREWORLD_BASE_URL=https://<platform address>
export PREWORLD_ENVIRONMENT=dev                    # dev, staging, prod or ci (ci by default when CI is set)

The SDK never reads a .env file by itself. Targets → SDK in the project shows these lines with your project's values filled in.

3. Write an Eval

support.eval.ts

import { Eval } from '@preworld/sdk';

// your agent: here a stand-in that answers from a table
async function answer(question: string): Promise<string> {
  const known: Record<string, string> = { 'How long do refunds take?': 'Refunds arrive within 5 working days.' };
  return known[question] ?? "I don't know.";
}

Eval('support-bot', {
  data: [
    { input: 'How long do refunds take?', expected: 'Refunds arrive within 5 working days.' },
    { input: 'Can I change my delivery address?', expected: 'Yes, until the order ships.' },
  ],
  task: answer,
  scores: [
    function exact({ output, expected }) { return output === expected; },   // scored here: a number 0–1 or true / false
    'builtin:factuality',                                                   // scored on the platform by an LLM judge
  ],
});

eval_support.py

from preworld import Eval

def answer(question):
    known = {"How long do refunds take?": "Refunds arrive within 5 working days."}
    return known.get(question, "I don't know.")

def exact(output, expected):          # gets the arguments it names: input, output, expected, metadata
    return output == expected

Eval(
    "support-bot",
    data=[
        {"input": "How long do refunds take?", "expected": "Refunds arrive within 5 working days."},
        {"input": "Can I change my delivery address?", "expected": "Yes, until the order ships."},
    ],
    task=answer,
    scores=[exact, "builtin:factuality"],
)

• data is a list of rows (input, and optionally expected, metadata, tags, id), any iterable, or dataset("<name>") to use a dataset stored on the platform.
• task is your code. It gets the row's input (and, as a second argument, the trial, the row key and the expected value).
• scores mixes functions you write, which run in your process, with platform scorers by id: builtin:<slug> from the built-in library or sc_… for a scorer the project made. Name your functions: the name is the scorer's column.

4. Run it

npx preworld eval            # TypeScript: every *.eval.ts / *.eval.js under this folder
preworld eval                # Python: every eval_*.py / *_eval.py under this folder

It waits while the platform scores the rows with its scorers (builtin:factuality here), then prints a summary and the link:

preworld: the platform's scorers are still scoring: the release gate is read once they're done…
Preworld · support-bot
  experiment: support-bot · 4f2c9e1 (exp_…)
  2 rows · 2 trials · 0 task errors · 9.4 s
  scorer              mean
  exact               50%
  builtin:factuality  100%
  Release gate: pass ✓
    – at most 0 regressed rows vs baseline (actual no baseline) · warning only
  https://<platform address>/p/prj_…/experiments/exp_…

That one rule is the project's default release gate: no row may get worse than in the baseline, as a warning only. A first run has no baseline yet, so the rule doesn't apply (–). Open the link: the experiment shows each row's input, output and scores, the same as the terminal. Run it again after a change, and the second experiment is compared row by row with the first: its baseline is the project's baseline when one is set, else the latest finished experiment of the same Eval (the same name in Eval('support-bot', …)), whatever its commit or time.

--no-send runs and scores locally without uploading anything; --first 5 runs only the first five rows of each Eval.

5. Gate it in CI

The command exits 0 when the release gate passed or has no rules, 1 when it blocked, 2 when the gate couldn't be read and 3 on an error. Set rules under Settings → Release gate (for example "factuality mean ≥ 0.8"), then run the same command in CI: a blocked version fails the step. The gate is read only once the platform's scorers have scored every row, so a rule on a platform scorer never blocks on scores still to come. If they aren't done within 10 minutes (PREWORLD_GATE_WAIT, in minutes), the gate isn't read and the exit code is 2. The GitHub Action that also comments a table on the pull request is in CI and the GitHub Action.

Trace the calls inside your task

Wrap your model client and each row links to its trace in Logs:

import OpenAI from 'openai';
import { wrapOpenAI, traced } from '@preworld/sdk';
const openai = wrapOpenAI(new OpenAI());                                  // chat, responses and embeddings calls become spans
const search = traced(async (q: string) => db.search(q), { name: 'search', kind: 'retrieval' });

In Python: wrap_openai(OpenAI()), wrap_anthropic(Anthropic()) and @traced. Every option is in the SDK reference.

Next steps

• Datasets: keep the rows on the platform, versioned, and add failures to them.
• Scorers: LLM judges, code and rules, tried on a row before you save.
• Experiments: baseline, compare and side by side.