Preworld Review · Docs

Evaluate AI agents, before and after they ship

Datasets, scorers and experiments for every change; logs, online scores and alerts in production; and what only we do, running your whole agent in a disposable sandbox under faults and attacks, with a release gate on every version.

Start free Quickstart Ways to connect

Evaluate with the SDK

Write an Eval in TypeScript or Python, run it locally or in CI, and get an experiment with scores and the change against your baseline.

SDK in 5 minutes

See production

Point your OpenAI or Anthropic SDK at the AI Gateway, or your OpenTelemetry exporter at the platform, and every call shows up in Logs.

Logs in 2 minutes

Test the whole agent

Connect a GitHub repository. Each run gets a fresh virtual machine with your test backend, faults and attacks, and a verdict backed by evidence.

Repository sandbox

Datasets and scorers

Versioned rows from tables, logs, experiments, reviews or AI; built-in, LLM-judge, code, trajectory, agent-behaviour and human scorers.

About datasets

Release gate

Rules over scores, regressions, undecided rows and required tags decide whether a version can ship, on the experiment, in the GitHub check and in CI.

About the gate

Online scoring and alerts

Score a sample of live traces, watch scores, errors and cost on the overview, and get mail, Slack or a webhook when something breaks.

About online scoring

What you can do

Evaluate every change. A dataset holds the rows to test, a scorer turns an output into a score, and an experiment is one version run over a dataset with its scores. Compare it with the baseline, row by row, and see what got better and what got worse.
Try ideas fast. The Playground runs prompts and models side by side over dataset rows, and saves each column as an experiment.
Watch production. Logs list the traces your app sends through the SDK, the AI Gateway or OpenTelemetry, with their spans, cost and latency. Online scoring scores a sample of them, alerts tell you when a score, the error rate or the cost goes wrong, and human review queues let people grade what matters.
Test the whole agent, for real. For an agent that calls tools and changes data, the platform starts your repository (or image) in a fresh virtual machine with its test backend, runs each row under normal, fault and attack conditions, checks the data it leaves behind and every tool call, and deletes the machine afterwards. Code that can't leave your company runs on your own machine with a self-hosted runner.
Decide on evidence. The release gate turns your rules into one sentence per version: "can ship" or "failed the release gate", and why. It shows on the experiment, on the overview, in the GitHub check and pull request comment, and as the exit code of your CI step.

The ten sections of a project

Section	What it is for	Guide
Overview	The latest version and its release gate, the score trend, recent experiments, live numbers, issues the assistant found	Projects
Logs	Traces from the SDK, the AI Gateway and OpenTelemetry; filters, facets, the span tree	Logs
Datasets	Rows to test, with versions, conditions and run environments	Datasets
Scorers	Built-in, LLM judge, code, rule, trajectory, agent-behaviour and human scorers; try and calibrate them	Scorers
Experiments	Every version tested, its scores against the baseline, compare and side by side	Experiments
Playground	Prompts and models side by side over dataset rows	Playground
Review	Human review queues	Human review
Targets	How your agent connects: SDK, Gateway, OpenTelemetry, repository or image sandbox, self-hosted runner, HTTP endpoint	Targets
Automations	Push, pull request and scheduled runs, API tokens, online scoring, alerts	Automations
Settings	Judge model, release gate, dev AI, sandbox, archived tests, versions	Projects

Start here

I want to…	Go to
Score my agent from code in a few minutes	SDK quickstart
See my production calls	Logs quickstart
Test my agent's tools and data in a sandbox	Repository sandbox quickstart
Pick how to connect my agent	Targets
Block a release when scores drop	Release gate, CI and the GitHub Action
Call the API	API: tokens and errors, v2 routes
Answer a security questionnaire	Security overview, Data processing
Understand what it costs	Pricing, Billing and usage
Get unstuck	Troubleshooting, Status