Issues the assistant finds
Low scores and errors rarely come one at a time. Every day the assistant reads what went wrong in a project (traces that errored, scored low or got a thumbs-down, and experiment rows that failed), groups the failures that look alike, and writes each group up as an issue: a title, what goes wrong, the evidence, how severe it is and how often it happened. An issue that comes back grows its count instead of being written again.
Where issues show
- Overview: the right-hand column lists the open issues, each with its severity, title, how many times it happened and since when.
- Logs → "Issues": every issue, filtered by status ("Open", "Not seen again", "Closed", "All"), severity and words, under a line saying how many failures the latest analysis read and what it found, with "Analyze now" beside it.
An issue holds:
| Part | What it says |
|---|---|
| Title and summary | What goes wrong, in a sentence or two. |
| Severity | High, medium or low. |
| Evidence | One sentence quoting the samples it rests on. |
| Count, first seen, last seen | How often it happened, and when. When the traces carry a version, the version it first appeared in. |
| Samples | Up to 20 traces, and the experiment rows that failed the same way. |
| Status | Open; or closed by a person, with who and why: fixed, not important, or wrong. |
An issue whose samples stop appearing for 7 days is marked "Not seen again"; so is an issue made only of experiment rows none of which counts as a failure any more (a run's diagnosis found the platform's own test harness, test connection or model at fault, and they read "Not tested (the platform's cause)"), at once. It isn't closed: a person closes it.
Act on an issue
| Action | What it does |
|---|---|
| See traces | Opens Logs filtered to the issue's sample traces. An issue whose samples are experiment rows only has "See rows" instead, which opens that experiment. |
| Add to dataset | Adds the samples to a dataset (the same picker as everywhere), so the next version is tested on them. |
| Build a scorer | The project's model drafts an LLM judge that catches this failure, and opens it in the scorer editor unsaved: try it on a few rows, change it, then save it yourself. |
| Close | With a reason: fixed, not important, or wrong. |
When it runs, and what it costs
- Once a day, for each project with traces or experiments in the last 24 hours; when new traces or finished experiments come in later that day, checked every hour and read once an hour has passed since the last analysis (only what finished at least half an hour ago, so a run's diagnosis has settled the platform's own faults first); and whenever a person clicks "Analyze now". Nothing else starts it.
- Before it reads experiment rows, each experiment is brought up to date with how its runs read now: an attempt the platform counts as its own fault ("Not tested (the platform's cause)") is no failure, and is never written up as the agent's issue.
- Each analysis reads the failures since the project's last one, 7 days back at most (so the first one, daily or "Analyze now", reads the last 7 days).
- The failures are embedded with the platform's embedding model and grouped; the project's model then names each group, writes its evidence and severity, and matches it against the open issues.
- It counts against the organization's assistant daily allowance, like the platform AI's other work, and stops cleanly when that is used up. A failure of the platform's own model isn't charged.
The assistant and a connected coding AI
The in-app assistant can now read and work with datasets, rows, scorers, experiments, traces and the release gate, with the same permissions as the person using it: it drafts where it drafted before, and anything that spends money or changes the release gate still needs that person's click.
A coding AI connected to the project gets tools for the same objects: list_experiments, get_experiment, get_experiment_rows, list_traces, get_trace, list_datasets, add_rows, create_scorer_draft and get_gate. Reading needs no permission; add_rows and create_scorer_draft need "Write test drafts and upload files"; none of them spends.
Next steps
- Logs and online scoring: the traces issues are found in.
- Scorers: finish the scorer an issue drafted.