Chat AI over HTTP (in development)

In development

This way is in development, so you can't choose it on the platform yet. Until then, connect a GitHub repository or use a local test runner. Projects already connected this way keep working, and the steps below still apply to them.

Use this way for chatbots, support bots, knowledge-base Q&A and other LLM apps that are tested on their replies only. You enter the API address and key in the browser, and there's nothing to install. The platform executor sends each question to your API, and the judge model grades each reply against your checks.

If the AI also looks things up or changes data, this tests only what it says. To test its tool calls and the data it changes, connect a GitHub repository or use a local test runner.

1. Create a project

Click "New project", pick "Q&A or customer-service bot", and name it.

2. Connect the chat AI

Choose the API type:

• OpenAI-compatible API: most model services and gateways support it (OpenAI, the major clouds, compatible APIs, a self-hosted vLLM). Enter the API URL, the model and the key. A system prompt is optional: it is sent as the first system message, and the system prompt leak check compares replies against it.
• Platform model: one of the platform's models, with no address or key of yours. Pick the model (the project's model to start with) and write a system prompt. A chat AI built this way keeps its model even if the project's model changes later, because that model is what you test.
• Custom HTTP API: your own chat endpoint. Enter the URL, any headers, and a JSON request body template. Put the key's placeholder, shown on the page, in the header where the key goes. In the body, placeholders stand for the question and for the whole conversation. Server-sent events are supported.

If you don't know how the request should look, enter just the address and let the platform's AI fill it in and test it: see "Let AI fill in and test" below.

You can also open "Fill in from a curl command", paste the curl example from your API's docs (or a command from "Copy as cURL" in your browser's developer tools) and click "Fill in the form". The platform reads the address, the headers and the body. It puts the placeholder where the question goes and moves a header value that looks like a key into the "Key" field. The command is only read, never run. A call to an OpenAI-compatible /chat/completions fills in the OpenAI-compatible form instead: address, model and system prompt.

Once an OpenAI-compatible API's address and key are in, the platform executor asks that address for its models (GET …/models), and you can pick one under "Model"; if it lists none, type the name in.

Click "Save", and the platform tests the connection right away with "Hi, what can you help me with?". For a custom HTTP API you don't have to find the field that holds the answer. The platform knows the OpenAI and Anthropic reply formats, the usual answer fields (reply, answer, output, text and so on) and streamed replies. When it can't tell, the platform model reads the raw reply and picks the answer, which counts toward the assistant's daily usage. The test result shows which field it used and how it found it. If that's wrong, click the actual answer in the raw reply.

The key is sealed as it arrives and stored encrypted. The API can't read it back, the page shows only its last four characters, and only the platform executor can decrypt it. For a custom HTTP API, you can paste your AI's system prompt in for the leak check; that copy is never sent to the AI. (For an OpenAI-compatible API, the system prompt you enter is the one sent, as above.)

Let AI fill in and test

Enter the address under "API URL", and the key in the "Key" field if there is one. Then paste whatever you have under "The API's docs, a curl example, or a sample request and reply". That can be the part of your API's docs about the chat endpoint, a curl command, one request with its reply, or a few notes. Nothing at all is fine too. If you have an OpenAPI or Swagger document, enter its address. Then click "Let AI fill in and test". The platform does the rest and shows each step on the page (Understand the API → Write the connection settings → Test the connection (n of up to 5) → Result):

• The platform's model works out from the address and what you pasted whether this is an OpenAI-compatible API or a custom one, and writes the connection settings: the address's path and query, the headers, the body with the question in it, the field of the reply that holds the answer, and whether the reply is streamed. When the address looks OpenAI-compatible, the platform executor first asks it for its models (GET …/models). If you entered an OpenAPI address, the platform executor reads that document.
• The model works in one conversation, as a developer would: it writes the settings, runs the platform's checks on them, has the platform executor test them and reads what came back. It may also have the platform executor read up to 4 more documents on the host you gave (for example …/openapi.json) or ask for the model list under another path there; the page lists everything it did.
• The platform executor tests the connection with "Hi, what can you help me with?", as "Test connection" does. When that gets no answer, the model revises the settings from the status, the error and the raw reply, then tests again, up to 5 tests. Settings already tested are never sent again.
• When the platform's AI is busy (many of your organization's tasks use it at once, or the model service is rate limiting), it waits in line until its turn, for up to 2 hours; the page says its place and about how long, so there's no need to come back and click.
• Once a test works, the connection is saved just as your "Save" would save it, with the name and system prompt you typed in the form. The audit log records it as yours, marked AI-written. If the project has no test set yet, starter questions are drafted from the AI's self-introduction (see step 3).
• When no test works, it says why, and the AI's last settings go into the form below for you to edit and save:
  • A key is missing: the API answered 401 or 403, or its docs say it needs a key. The page names the header the key goes in. Enter the key in the "Key" field and click again; the platform goes on from the settings it wrote, now with the key.
  • Can't reach it: the host name or port is wrong, or the API can't be reached from the internet.
  • 5 tests and no answer (or the AI's 2 conversations ended without one): the page shows the last result. Paste more of the API's docs and click again to continue from the AI's last settings, or edit them by hand.

The platform's code enforces these rules; they don't depend on a prompt:

• It connects only to the scheme, host and port of the address you gave. A model answer pointing anywhere else is sent back, and no request ever goes there.
• The key goes only in a header (written as ), and the platform executor fills it in when it sends the request. The model never sees the key, and anything in what you paste that looks like a key is masked before the model reads it.
• What you paste and what your API answers are only material for the model to read: instructions in them are never followed, and a curl command is only read, never run.
• Every request goes through the platform executor's egress checks (private, local and cloud metadata addresses are refused), and what it reads back is capped in size.

Its model calls count against the assistant's daily allowance, and at the limit it stops and says so. The tests call your own API, so the platform charges nothing for them. For a custom API, the platform sends only POST requests with a JSON body. Because it sends a key only in a header, an API that takes its key only in the body or the address can't be connected this way.

3. Add questions

The first time the connection works, if the project has no test set yet, the platform drafts a few starter questions from the project's domain, the system prompt and the AI's own self-introduction. It uses the platform model, which counts toward the assistant's daily usage. The questions are in the AI's own language, or in your page's language when that can't be told. The questions have no reference answers, since only you can give those. They are tagged "needs-answer" and are judged only by checks that don't need one. With automatic approval on, the AI reviews them first like any other draft. Edit or delete them, or add reference answers.

Add the questions you want to ask, with a reference answer or key points to check replies against. Start with the 10–50 most common and most important ones.

• Import a table: paste from Excel, or upload .xlsx (first sheet), CSV or JSONL. promptfoo test files work too. A test set holds up to 1,000 questions and a table up to 5,000,000 characters; split a bigger one into several test sets.
• Type them in, or ask the assistant to draft some from a description of your AI.
• For a multi-turn conversation, separate the turns with ||: the platform sends them one at a time and judges the answer to the last one.

Then choose the checks, for example: correct against the reference, covers the key points, doesn't leak the system prompt. You don't have to set pass criteria first. A test set without its own runs on the default: each question passes every time it is asked, for at least 90% of the questions. A security test set, with attacks or safety checks, must pass every question, and any attack that gets through blocks it. "Start test" shows the criteria in effect, with "Change" to edit them.

"Doesn't leak the system prompt" compares replies with the system prompt. For an OpenAI-compatible API or a platform model, that's the one entered on the connection; for a custom HTTP API, it's the copy pasted under "Its system prompt". Without one the check can't be decided, so the run leaves it out and the page says so. The test still starts.

4. Start the test and read the report

Click "Start test". The platform executor calls your API with at most 4 requests at once and 120 requests a minute by default. The report gives a verdict on the replies, "Passed" or "Not passed", with every question, reply and judgement. "Passed" here means every pass criterion was met for what the AI said; tools and data were not tested. If a run was cancelled or didn't finish, the top of the report says so and shows how many attempts ran. Attempts that didn't run count as neither passed nor failed, including when you compare with the last run.

5. Run it automatically

Under "Start test", set a schedule (at most hourly), or create a project API token so your CI can start a round after each deploy: POST /v1/projects/<id>/jobs with the token. Each automatic run sends every confirmed test set to the chat AI connected to the project. If several are connected, the run is refused with a message saying so; keep only the one to test. It runs as the person who switched it on (or made the token), is billed like a run you start, and emails that person when it ends.

How the platform reaches your API

• Requests come from the platform executor on our control plane in London, over HTTPS, from the control plane's fixed IP address.
• The executor only sends HTTP requests; it never runs your code.
• Every address a URL resolves to is checked and pinned: cloud metadata addresses and private or local addresses are refused, so the executor can't be pointed at internal systems.
• Tool calls your AI asks for are recorded in the report but never executed.

What enters our cloud

The conversations (the questions and your AI's replies), the chat AI's key (sealed), and afterwards the run bundle.

Questions and replies are sent to the judge model's provider for grading. When starter questions are drafted, or the answer's field can't be told from the reply, the project's domain, the system prompt and the connection test's reply are sent to the platform model's provider (the assistant's). With "Let AI fill in and test", these are sent there too: the API address, what you paste (with anything that looks like a key masked), the OpenAPI document and the other documents it reads on your host, and your API's replies to the tests. The key is never sent. See sub-processors.