Projects
A project holds one agent or app: its datasets, scorers, experiments, logs, targets, automations and settings.
Create a project
- Click "New project" in the sidebar.
- Pick what you're testing:
- "Agent that calls tools": an agent that looks things up, changes data and takes actions. Pick this for an SDK or Gateway project too, and when all you have is a code repository.
- "Q&A or customer-service bot" (in development, not shown for now): a chat AI tested on its replies through its API. See Chat AI over HTTP.
- "I'm not sure": explains the difference, and that an app already in production (a chatbot too) needs no repository: it comes in through the SDK, the AI Gateway (both in development) or OpenTelemetry under Targets. "Ask the assistant" helps you pick.
- Enter a "Name" and a "Business domain" (for example "expense claims"; left empty, it's "General").
- Click "Create project".
Then connect it under Targets (how to choose). The page you land on says how, one line each: "Connect a GitHub repository", the local test runner, and for an app already in production the SDK (run tests in your process or CI), the AI Gateway (change one base URL) or OpenTelemetry (point your exporter here). The SDK and the AI Gateway are in development: they say "In development" there and aren't links yet.
The ten sections
A project's pages are on the rail on the left, always in this order:
| Section | What's there | Guide |
|---|---|---|
| Overview | "Latest version": the newest experiment's release-gate verdict ("v13 can ship"), with up to three figures and their change; the score trend over recent experiments (or the online scores' daily means once traces are scored); "Recent experiments"; "Live · last 7 days" once traces arrive; and the issues the assistant found | Release gate, Issues |
| Logs | Traces, facets, filters, saved views, the span tree | Logs |
| Datasets | Datasets and their rows, versions, imports, AI-written rows, attack datasets | Datasets |
| Scorers | The project's scorers and the built-in library; try and calibrate | Scorers |
| Experiments | Every version tested; compare and side by side; "Test new version" | Experiments |
| Playground | Prompts and models side by side over rows | Playground |
| Review | Human review queues | Human review |
| Targets | How the agent connects, its connection check, secrets | Targets |
| Automations | Push, pull request and scheduled runs, API tokens, online scoring, alerts | Automations, Online scoring |
| Settings | Below |
"Test new version", at the top right of every section, opens the form that starts a sandbox, test-runner or chat-AI round in Experiments. Addresses of the project page from earlier releases (/p/<id>?tab=…) still work: each opens the section that does that job now.
The project's model
Pick the project's model in the dropdown next to its name, in the header: "Strongest · slower · Gemini 4 Argon" (the default), "Strong · faster · GPT-6.1 Sol", "Balanced · Gemini 3.8 Flash", "Economy · GPT-6 Luna", or "Follow the organization default (…)". Each option shows its price per 1M tokens as billed. The judge, LLM scorers that follow the project, the platform AI and an agent in the sandbox without a model key of its own all use this one model. Results you already have keep the model that judged them. See Models.
Settings
Open Settings. Pick one part at a time:
| Part | What's there |
|---|---|
| "Judge model" ("Judge and agent models" for a sandbox project on the platform's model) | The project's model, read-only, with a link to the picker in the header; and the platform's second judge, which reads again what the first can't decide, billed by usage (see Second judge model). |
| "Release gate" | The rules every experiment is judged by (Release gate), and "Customize pass criteria", the pass criteria a sandbox run's own verdict uses (pass criteria). |
| "Dev AI" | Connect the coding AI on a developer's machine, with the permissions you tick (Using the platform from a coding AI). |
| "Sandbox" | The class, architecture and run location of the cloud sandbox (below). |
| "Archived tests" | Deleted tests, to restore or delete permanently. |
| "Agent versions" | Your agent's code, prompt and model at each run, to compare before and after a change. |
| "Advanced" | "All tests (detailed list)" (bulk confirm and delete, every version), "Test new version: all options", "Runs and trends", the name and business domain, and "Delete project". |
Test runners are under Targets; triggers, the schedule and API tokens under Automations. The files handed to the agent in tests are attachments of dataset rows.
Sandbox settings
A repository or image project runs each test in a sandbox, a virtual machine the platform starts in the cloud and deletes afterwards:
- "Class": "Default (Standard)" or another class, each shown with its CPUs, memory and price per hour (sandbox classes). "Custom…" sets the cores, memory, disk and GPUs yourself (custom spec). The platform recommends a class from the test connection and the latest sandbox logs, and says why; it applies only when you click "Save".
- "Architecture": "Auto", "x86" or "ARM". Auto uses x86 for a repository and the image's own architecture for an image.
- "Run location": "Auto" or "London" (run location).
Click "Save". Limits such as how many sandboxes run at once are in Sandboxes and limits.
Name and business domain
The name shows in the sidebar and on every report. The business domain tells the AI that drafts, reviews and attacks your tests what business the agent is in. Change them under "Settings" → "Advanced".
Delete and restore a project
- Open "Settings" → "Advanced".
- Under "Delete project", type the project's name.
- Click "Delete this project".
The project leaves the list, its test runners and dev AI disconnect, its project API tokens stop working and queued tests are cancelled. Past reports still open from their links. A project with a test running can't be deleted: cancel the test or wait for it to end.
For 30 days you can bring it back: at the bottom of the project list, open "Deleted projects" and click "Restore". After 30 days the project, its reports and its evidence files are deleted permanently.
The assistant and the expert view
Click "Assistant" in the top bar (or press ⌘J) to open the assistant. It knows which page you're on, can read and work with the project's datasets, scorers, experiments, traces and gate, write rows and tests, and explain results. Whatever spends money or changes the release gate comes to you as a card, and nothing happens until you click. For a project connected through a GitHub repository, it can also read the repository, never a file that holds secrets.
The "Expert view" switch at the bottom of the sidebar shows ids, hashes, YAML and other technical details. It's off by default and remembered in this browser only.
Next steps
- Targets: connect the agent.
- Datasets and Experiments: test it.