CI and the GitHub Action

Three ways to put the release gate in your pipeline. Each exits 0 when the gate passed, 1 when it blocked and 2 when it couldn't judge (no rules, the run didn't finish, or the gate couldn't be read); 3 is an error. A gate that passed with low-severity problems it only warns of still exits 0, and prints a warning line (reviewctl eval --launch makes it a warning annotation on GitHub Actions).

You test with	In CI run	Comment on the pull request
The SDK (`.eval.ts`, `eval_.py`)	`preworld eval`	The SDK action
The project's tests on the platform (sandbox, test runner, chat AI)	`reviewctl eval --launch`, or the API	The reviewctl action with `launch: true`, or the platform's own GitHub check
A test set in the repository against a chat AI	`reviewctl eval --set … --target …`	The reviewctl action

The actions live in the platform's repository under integrations/; write to admin@nice2.io for the repository and ref to put in uses:. Tokens go in your CI's secrets, never in the workflow file.

The SDK action

integrations/github-action-sdk runs preworld eval, writes a Markdown summary to the job summary, comments it on the pull request (one comment, updated on every push), and fails the check when the gate blocks. It reads the gate once the platform's scorers are done (at most PREWORLD_GATE_WAIT minutes, default 10). The SDK comes from the file you downloaded from the platform and committed (Install the SDK): npm ci or pip install installs it from there.

yaml

on: pull_request
permissions: { contents: read, pull-requests: write }
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - uses: <owner>/<repo>/integrations/github-action-sdk@<ref>
        with:
          paths: evals
          api-key: ${{ secrets.PREWORLD_API_KEY }}     # a project token with the upload scope
          project: prj_your_project_id

Input	Default	Meaning
`api-key`	(required)	The project token
`project`	(required)	The project id
`paths`	`.`	Files or folders, space-separated
`runtime`	`node`	`node` or `python` (set up Python and install your project instead of Node)
`command`	`npx --no-install preworld eval` / `python -m preworld eval`	Another command to run
`base-url`	the SDK's default	The platform's address
`environment`	`ci`	The experiments' environment tag
`working-directory`	`.`	Where to run
`fail-on-gate`	`true`	Fail the check when the gate blocks or couldn't be read
`disable-comment`	`false`	Don't comment on the pull request
`language`	`en`	`en` or `zh` for the comment
`github-token`	`${{ github.token }}`	Needs `pull-requests: write` to comment

Outputs: exit-code, result (the --json file) and summary (the Markdown file).

The comment, per Eval: the experiment's link, rows, trials and task errors, then a table "Scorer · Average · Change vs baseline · Improved · Regressed" (platform scorers still being scored show as pending), then the release gate with each rule. A pull request from a fork, or a read-only token, gets a warning instead of a comment.

The reviewctl action

integrations/github-action builds reviewctl and runs it in one of two modes.

Launch the project's tests and wait for the gate

yaml

on: [pull_request, push]
permissions: { contents: read, pull-requests: write }
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: <owner>/<repo>/integrations/github-action@<ref>
        with:
          launch: true
          api-url: https://<platform address>
          project: prj_your_project_id
          token: ${{ secrets.PREWORLD_PROJECT_TOKEN }}   # the project's API token pj_… (Automations → API tokens)
          # commit: auto        # default: the pull request's head, else the pushed commit; none sends no commit
          # image: ghcr.io/acme/agent@sha256:…   # a project that runs its own image
          # scope: all          # all, functional, security, failed
          # wait-minutes: 60

It launches every confirmed test of the project on the platform (a repository project in a sandbox at that commit, an image project at its image, a test-runner project on its runner), waits for the release gate of the experiment the round becomes, writes the gate's table to the job summary and the pull request, and exits with the gate. A re-run of the same job sends the same idempotency key and waits on the same round.

A test set from the repository

yaml

      - uses: <owner>/<repo>/integrations/github-action@<ref>
        with:
          set: ai/support-faq.yaml        # the test set
          target: ai/target.yaml          # how to reach the chat AI under test
          min-pass-rate: 90%
        env:
          SUPPORT_BOT_KEY: ${{ secrets.SUPPORT_BOT_KEY }}   # the variable target.yaml's key_env names
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}     # only for AI-judged checks (pick the model with JUDGE_MODEL)

The test set runs from the pull request's checkout, so a changed system prompt or question is tested before it's deployed. With api-url, project and token (a CLI token rt_…), a push to the default branch is pushed to the platform's archive and pull requests are compared with its latest run.

Input	Default	Meaning
`launch`	`false`	`true`: launch on the platform and wait for the gate
`set`, `target`		The test set and target files (not with `launch`)
`commit`	`auto`	`launch`: the commit to test (`auto`, a sha, or `none`)
`image`		`launch`: the image to test
`scope`	`all`	`launch`: `all`, `functional`, `security` or `failed`
`wait-minutes`	`60`	`launch`: how long to wait for the gate
`repeat`		Times to run each test
`gate`, `min-pass-rate`		A test set's gate file, or the lowest pass rate
`baseline`	`auto`	What a test set is compared with
`push`	`auto`	Push a test set's run to the archive
`fail-on-threshold`	`true`	Fail the check when the gate blocks or isn't judged
`disable-comment`	`false`	Don't comment
`api-url`, `token`, `project`		The platform, the token, the project
`working-directory`	`.`	Where to run
`artifact-name`	`reviewctl-results`	The uploaded artifact
`github-token`	`${{ github.token }}`	For the comment

Outputs: exit-code, verdict (success, failure or neutral in launch mode), experiment-id, experiment-url, run-id, report-url and summary.

`reviewctl eval --launch`

The same launch from any CI:

bash

export REVIEWCTL_API_URL=https://<platform address>
export REVIEWCTL_TOKEN=pj_your_token_here          # the project's API token
reviewctl eval --launch --project prj_your_project_id --commit "$GIT_SHA" --wait 60 -o summary.md

Flag	Meaning
`--project prj_…`	Required
`--commit <40-character sha>`	A repository project: the commit to test
`--image <ref>`	An image project: the image to test
`--scope all\|functional\|security\|failed`	Which tests
`--repeats <n>`	Times to run each
`--wait <minutes>`	How long to wait for the gate (default 60)
`--idempotency-key <key>`	8 to 128 characters: a retry waits on the same round
`--title <text>`, `-o <file.md\|file.json>`, `--json`	The summary's title, where to write it, JSON on standard output

It prints the gate's table: each scorer's average, change against the baseline, rows improved and regressed, then each rule with ✓ or ✗ and its actual value, and the problems to deal with first. Never the agent's output, an attack's text or a secret. See Command line for test sets.

The gate from your own script

Launch with POST /v1/projects/<id>/jobs (a project token); the answer has experiment_id and gate_url. Then long-poll the gate until it's final, and exit by its conclusion:

bash

GATE=$(curl -fsS -X POST "https://<platform address>/v1/projects/$PROJECT/jobs" \
  -H "Authorization: Bearer $PREWORLD_PROJECT_TOKEN" -H "Content-Type: application/json" \
  -H "Idempotency-Key: gh-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" -d "{\"commit\":\"$GITHUB_SHA\"}" | jq -r .gate_url)
for i in $(seq 60); do
  G=$(curl -fsS -H "Authorization: Bearer $PREWORLD_PROJECT_TOKEN" "$GATE?wait=60")
  [ "$(echo "$G" | jq -r .final)" = true ] && break
done
case $(echo "$G" | jq -r 'if .final then .conclusion else "neutral" end') in success) exit 0;; failure) exit 1;; *) exit 2;; esac

conclusion is success when the gate passed and the run finished, failure when it blocked, and neutral when there are no rules or the run didn't finish and nothing broke for sure. The answer's fields: v2 routes. The launch's fields: start a run from CI.

The platform's own GitHub check

A project connected to a GitHub repository can have the platform test every push and pull request by itself and put the gate on the commit as a GitHub check, without any workflow: see automatic runs. When it also comments on pull requests, set the action's disable-comment: true so there aren't two comments.

CI and the GitHub Action ​

The SDK action ​

The reviewctl action ​

Launch the project's tests and wait for the gate ​

A test set from the repository ​

reviewctl eval --launch ​

The gate from your own script ​

The platform's own GitHub check ​

CI and the GitHub Action

The SDK action

The reviewctl action

Launch the project's tests and wait for the gate

A test set from the repository

`reviewctl eval --launch`

The gate from your own script

The platform's own GitHub check