SDKs: TypeScript, Python and `preworld eval`

In development

The SDK is in development as a way to connect a project, so you can't choose it on the platform yet (see the SDK quickstart). Projects already connected this way keep working, and this reference still applies to them.

The SDKs run evaluations in your own process and upload them as experiments, and trace model calls over OpenTelemetry. TypeScript: @preworld/sdk (Node 20 or later, ESM and CommonJS). Python: preworld (Python 3.10 or later). Neither is on npm or PyPI: the platform serves both at /downloads/, versioned with each release (sdk.json names the current files), and Targets → SDK gives the install lines; see Install the SDK. A walk-through is in the SDK quickstart.

Configuration

Variable	Meaning
`PREWORLD_API_KEY`	A project token `pj_…` with the upload scope ("Allow uploading logs and experiments")
`PREWORLD_PROJECT`	The project id `prj_…`
`PREWORLD_BASE_URL`	The platform's address (default `https://review.nice2.io`)
`PREWORLD_ENVIRONMENT`	`dev`, `staging`, `prod` or `ci`; `ci` by default when `CI` is set. Other values are ignored with a warning.
`PREWORLD_LANG`	`zh` prints summaries in Chinese (else `LC_ALL`, `LANG` decide)
`PREWORLD_GATE_WAIT`	How long, in minutes, an Eval waits for the platform's scorers to finish before it reads the release gate (default `10`; `0` reads it once)
`OTEL_SERVICE_NAME`	The traces' `service.name`

Or in code: init({ project, apiKey?, baseUrl?, environment? }) (TypeScript) or preworld.init(project=..., api_key=None, base_url=None, environment=None) (Python); these win over the variables. The SDKs never read a .env file. Git details come from GITHUB_SHA, GITHUB_HEAD_REF, GITHUB_REF_NAME and GITHUB_REF (a pull request number from refs/pull/N/…), else from git.

Eval

TypeScriptPython

Eval<I, O, E>(name: string, options: {
  data: Row[] | Iterable<Row> | AsyncIterable<Row> | Dataset | (() => …),   // Row = { input, expected?, metadata?, tags?, id? }
  task: (input: I, ctx: { expected?, metadata?, tags, trial, rowKey, experimentId?, span? }) => O | Promise<O>,
  scores?: Array<ScorerFn | string>,
  metadata?: Record<string, unknown>,
  trialCount?: number,        // default 1
  maxConcurrency?: number,    // default 10
  experimentName?: string,    // default "<name> · <commit sha7>", or "<name> · YYYY-MM-DD HH:MM" without git
  baseline?: string,          // exp_…; default: the project's baseline, else the last finished experiment of this Eval (by its name)
}): Promise<EvalResult>

python

Eval(name, *, data, task, scores=(), metadata=None, trial_count=1, max_concurrency=10,
     experiment_name=None, baseline=None) -> EvalResult
await EvalAsync(...)          # the same, inside a running event loop

data: rows, any iterable (sync or async), a function returning one, or dataset("<name or ds_…>", { version }) / dataset(name, version=None) to read a platform dataset (its current rows, or that version). Only input, expected, metadata and tags of a platform row reach the task.
task: your code. Python passes the context as a second argument when the function takes one; plain functions run on worker threads, async ones on the loop.
scores: local scorer functions, called with {input, output, expected, metadata} (Python: the arguments they name) and returning a number from 0 to 1, a boolean, null/None, or {name?, score, metadata?}; and platform scorers by id, builtin:<slug> or sc_… (optionally @version), scored on the platform after upload. A scorer's column is its function's name (unnamed ones become score_N), stored as local:<name>, which is how a gate rule names it; metadata.rationale and metadata.label are kept.
Undecided: a task that throws makes the row task_error; a scorer that throws is scorer_error, one returning nothing skipped, one outside 0–1 bad_score. A row's score is the mean of its decided trials.
Row keys: <dataset row id>#normal for a platform dataset row, else the row's id, else a hash of its input; repeats get #2, #3. The same input gives the same key in both SDKs, so the baseline matches rows across runs.

What Eval does: creates the experiment (POST /v2/projects/:p/experiments), appends rows in batches of up to 100 rows or 4 MB (POST /v2/experiments/:id/rows, each batch with its own idempotency key), finishes it, waits for the release gate to be final — the platform's scorers (builtin:…, sc_…) score the rows after the finish, and the gate is read on its long-poll (GET /v2/experiments/:id/gate?wait=60, asked again until final, for at most PREWORLD_GATE_WAIT minutes) — reads the baseline's rows to compare, prints a summary and returns the result. The platform scorers' figures in the summary are the platform's, as the experiment's page shows them; when they aren't done in time, they stay "pending" and the gate isn't read (exit code 2), never blocked. Requests are retried on network errors and 408, 425, 429, 5xx (5 tries, honouring Retry-After up to 60 s; 60 s timeout each). If the platform refuses a batch, the remaining rows aren't run.

The summary:

text

Preworld · support-bot
  experiment: support-bot · 4f2c9e1 (exp_…)
  20 rows · 20 trials · 0 task errors · 14.2 s
  scorer              mean     baseline change        improved regressed
  exact               85%      80%      +5.0 pts      2        1
  builtin:factuality  90%      85%      +5.0 pts      1        0
  Release gate: pass ✓
    ✓ local:exact mean >= 0.8 (actual 0.85)
  https://<platform address>/p/prj_…/experiments/exp_…

EvalResult has name, project, experiment ({id, name, url}), baselineId, rows, trials, taskErrors, durationMs, scores (per scorer: mean, decided, undecided, pending, direction, baseline_mean, delta, improved, regressed), gate, gateError, error, exitCode and results (Python: snake_case). Eval never exits the process: exitCode(results) / exit_code(results) gives the code preworld eval would.

Tracing

TypeScript	Python	What it traces
`traced(fn, { name?, kind? })`	`@traced` or `@traced(name=None, kind="chain")`	A function: its arguments as input, its result as output, a throw as an error
`wrapOpenAI(client)`	`wrap_openai(client)`	`chat.completions.create`, `responses.create`, `embeddings.create`, streaming included (TypeScript returns a wrapper; Python changes the client you pass and returns it)
`wrapAnthropic(client)`	`wrap_anthropic(client)`	`messages.create`, `stream: true` included
`aiSdkTelemetry({ functionId?, recordInputs?, recordOutputs? })`	—	The Vercel AI SDK 5, 6 and 7: `experimental_telemetry: aiSdkTelemetry()`
`currentSpan()`, `flush()`, `shutdownTracing()`	`current_span()`, `flush()`, `shutdown()`	Send what's pending before a short-lived process exits (Eval and the CLI do)

Kinds: agent, llm, tool, retrieval, embedding, chain (default), other. Spans carry the OpenTelemetry GenAI attributes (gen_ai.request.model, gen_ai.usage.*, gen_ai.input.messages, gen_ai.output.messages…) and are exported as OTLP/HTTP protobuf to <base url>/otel/v1/traces with the token and x-preworld-environment. The SDK's tracer is its own and never replaces a global OpenTelemetry provider. Inside an Eval, every span carries preworld.experiment_id, preworld.row_key and preworld.trial, so each row links to its trace. Tracing is off without an API key and under --no-send.

The command line: `preworld eval`

bash

npx preworld eval [files or directories…] [--no-send] [--json] [--first N]     # TypeScript
preworld eval [files or directories…] [--no-send] [--json] [--first N]         # Python (or: python -m preworld eval)


Files	TypeScript: `.eval.ts`, `.mts`, `.cts`, `.js`, `.mjs`, `.cjs` (TypeScript needs `tsx` installed). Python: `eval_.py` and `*_eval.py`. Under the current folder by default; `node_modules`, `__pycache__`, virtual environments and dot-folders are skipped.
`--no-send`	Run and score locally; nothing is uploaded and no token is needed.
`--json`	JSON on standard output, the summary on standard error: `{version: 1, exit_code, evals: [{file, name, project, experiment, baseline_id, rows, trials, task_errors, duration_ms, scores, gate, gate_error, error, exit_code}]}`.
`--first N`	Only the first N rows of each Eval.

Exit codes: 0 the release gate passed or has no rules · 1 the release gate blocked · 2 the gate couldn't be read (the platform's scorers not done within PREWORLD_GATE_WAIT, or no answer) · 3 an error (no files, a file that failed, bad flags). With several Evals: any 3 gives 3, else any 1 gives 1, else any 2 gives 2.

pytest

The Python package is also a pytest plugin. Tests marked @pytest.mark.preworld (optionally tags=[…]) become the rows of one experiment, keyed by their node id; their input is the test's parameters (or its name).

python

import pytest
from preworld.pytest_plugin import check, log_outputs

@pytest.mark.preworld(tags=["math"])
@pytest.mark.parametrize("q,expected", [("2+2", "4")])
def test_answer(q, expected):
    out = agent(q)
    log_outputs(out)
    check("short", len(out) < 100)     # an assertion that is also its own score
    assert out == expected

Each test's pass score is 1 or 0, with the failed assertion as its reason. check(name, condition, message=None), log_score(name, value), log_inputs, log_outputs and log_expected add to the row. Skipped and xfail tests aren't rows.
Sent only when PREWORLD_API_KEY and PREWORLD_PROJECT are set. --preworld-no-send keeps it local; --preworld-experiment NAME names the experiment (default <folder> · <sha7>).
At the end, the experiment is finished, its gate (once final) and baseline read, and a "preworld" section printed. pytest's own exit status is never changed.

Next steps

CI and the GitHub Action: run it on every pull request with a comment.
v2 routes: the API underneath.

SDKs: TypeScript, Python and preworld eval ​

Configuration ​

Eval ​

Tracing ​

The command line: preworld eval ​

pytest ​