Evaluations
Score past traffic against rubrics and code.
Evaluations grade what already happened: point a rubric (or a code check) at existing spans, traces, or threads and get verdicts back. Three nouns:
- Evaluation — the editable definition: scope, variables, prompt, judge model.
- Run — one pass of exactly one evaluation over matched traffic.
- Result — one verdict about one target.
One evaluation per run is load-bearing: the old multi-evaluator sweep made every count ambiguous.
Creating, editing, and running evaluations needs the evaluations plan
feature (Pro and Enterprise); reading, cancelling, and deleting stay open after
a downgrade. Each action needs its evaluation:* permission, which owners
and admins hold and members don't.
Definitions
An evaluation names an immutable lower-snake metric key, a title, a scope
(span, trace, or thread), and a prompt with {variable} placeholders
(escape a literal as {{x}}). Variables (max 20) pull from 10 payload sources
— attribute, question, answer, chunks, tool calls, available tools, tool name,
tool input, tool output, transcript — plus five host-derived ones (citations,
abstention, output schema, resolution, tool outcome), each with an enforced
scope allowlist.
Results are categorical or numeric. Categorical takes 2–20 distinct labels,
each with a score or null — a null label is counted but left out of the
rate. Numeric takes bounds. Either may set a direction (maximize,
minimize, none). The saved agent picks whose traffic is evaluated (a run
may override it), aiModel picks the judge, and up to 10 filters narrow the
anchor span with eq, ne, contains, exists, or missing. Explanations
are on by default; turning them off stops asking the judge for one.
Two kinds author: llm judges by prompt; code runs a function and makes no
model calls — a built-in check, or source you edit, which runs sandboxed with
no network. The third kind, human, is
server-provisioned on first feedback — user ratings land in the same
verdict store. Disabled and human evaluations can't be run.
The gallery offers ready-made templates; nothing is created on your behalf,
and a template becomes a row only when you press Create. To version a judge,
clone it: the copy gets a new key (suggested as the old key plus _v2) and
its own series.
Runs and results
Starting a run samples matched traffic (run rate, else evaluation default, else everything), filters by agent/thread/window, caps at 300 targets, and executes the current definition — the key is copied, mid-queue edits apply. The 202 answers with the run id and whether it calls a model. Cancel is advisory; results already written stay, and finished runs are untouched. Deleting a run deletes its results; deleting an evaluation keeps them.
There is one result per target and evaluation: running again overwrites the verdict rather than adding one, so retries never double-count. Each result records scope, target ids, agent, timestamp, label or score with explanation, judge model and trace, and any error. Rollups aggregate by name, scope, days, and agent. A variable-preview endpoint dry-runs unsaved draft variables against a target, and the editor offers a span-tree picker for scoping.