Production traces¶

AI Eval imports OpenTelemetry GenAI spans from a live site, groups them into conversations and agent runs for review, lets you label them, and promotes the interesting ones into datasets. A dataset grown from traffic your site actually served covers inputs you would not have thought to write down. This page is for whoever does that review.

The loop:

flowchart LR
    OTLP["OTLP JSON export"]
    IMP["Import<br/>drush or admin form"]
    STORE["ai_eval_trace"]
    REV["Grouped review queue"]
    LAB["Labels<br/>pass, fail, defer"]
    DS["Dataset question"]
    GOLD["Judge gold set"]

    OTLP --> IMP --> STORE --> REV --> LAB
    LAB --> DS
    LAB --> GOLD

Where the spans come from¶

Trace review works on OpenTelemetry GenAI spans written by ai_observability, a submodule of the AI module, through the OpenTelemetry PHP SDK.

Install and enable the OpenTelemetry module.
Enable the ai_observability submodule.
In its settings turn on otel_enabled, the master switch, which is off by default. Span emission (otel_spans) is already on, so flipping the master switch starts producing one span per provider call. To capture the prompt and response text you want to review, also turn on otel_spans_store_input and otel_spans_store_output, both off by default.

Then configure an OTLP file exporter so spans land as JSON on disk. An OpenTelemetry Collector with a file exporter is the simplest setup:

exporters:
  file:
    path: /var/log/otel/traces.json
service:
  pipelines:
    traces:
      exporters: [file]

For clean per-conversation grouping, set the assistant's allow_history setting to session. The assistant API stamps a per-session thread tag on each span, and AI Eval groups on it. Without it, each call reviews as an isolated group.

Importing¶

# One OTLP JSON file, or every *.json file in a directory
drush ai-eval:import-traces /var/log/otel/traces.json
drush ai-eval:import-traces /var/log/otel/

# With an explicit batch id
drush ai-eval:import-traces traces.json --batch=2026-06-30-prod

ai-eval:import-traces (alias aeit) takes one argument, a file or a directory, and one option:

Option	Default	What it does
`--batch`	the filename without its extension	The import batch id stamped on every row from that file.

A directory is scanned for *.json and sorted, so import order is deterministic across filesystems. The file format can be a single OTLP document, several concatenated documents, or newline-delimited JSON. Spans that are not AI spans are skipped, and the command finishes with a count line:

[OK] Imported 412 / 500 spans (88 skipped, 6 truncated at source).

Import is idempotent. Rows are keyed on the span id, so replaying a file updates existing rows in place rather than creating duplicates. A missing path, an unreadable file, or a payload with no OTLP document in it exits 1 with a message rather than reporting a false success.

Smaller imports can skip Drush. The Traces tab has an Import traces action at /admin/config/ai/ai-eval/trace-review/import that accepts an upload or a server-side path and runs the same importer through the Batch API, with a progress bar and the same four tallies at the end. Uploads are capped at 8 MB per file; use Drush beyond that. The action needs operate ai eval or administer ai eval.

The batch id¶

Every imported row records its batch id, so an import is identifiable after the fact and a batch can be removed as a unit. The removal itself is not yet exposed: Drupal\ai_eval\Service\TraceStore::clearBatch() on the ai_eval.trace_store service deletes a batch and its tag rows, and there is no Drush command or admin screen that calls it. Undoing an import today means calling that service from your own code or from drush php:eval. Give a batch a meaningful id at import time anyway; it costs nothing and it is what makes the undo possible at all.

The review queue¶

The Traces tab at /admin/config/ai/ai-eval/trace-review lists trace groups, 50 per page, most recent activity first. Spans are grouped rather than listed flat, because one answer is rarely one provider call:

Group type	What it is	Rendered as
`chat`	Spans sharing a conversation key, from an assistant thread or a multi-request agent thread	A chronological conversation thread
`agent_run`	Spans sharing a trace id with no conversation key	An indented call tree, parent above child
`single`	An `agent_run` group with exactly one member	A single card

Each group card shows the type, the providers and models involved, the member count, and a rollup of its members' verdicts. Date and tag filters narrow the list, and the filters travel with you into a group and with the prev and next group navigation, so a filtered review stays inside its filter.

Opening a group gives you a verdict bar for the whole group plus one card per member trace. A card carries the request and response text, token usage, the finish reason, the span's tags and remaining attributes, the labeling sidebar, and a promote control.

Reviewing needs operate ai eval or administer ai eval.

Labeling¶

A label is a verdict plus an optional note. The verdicts are:

Verdict	Meaning
`pass`	The answer was good.
`fail`	The answer was bad.
`defer`	Come back to this one. Not a gold verdict, and it does not count toward gold coverage.

Labels come in two scopes. A group verdict is one row per group, an open-coding "first thing that went wrong" for the whole conversation or run. A span label is per member trace, and that is the one downstream machinery reads.

Span labels are keyed by trace, dimension, labeler, and round:

Dimension. Labels written from the review queue carry no dimension: they are the holistic verdict on that trace. Labels written from a judge's gold screen are stamped with that judge's dimension, and an LLM judge is validated only against labels in its own dimension. A null dimension never means "any dimension", so one trace can carry a holistic label plus one label per judged dimension without collisions.
Labeler. Each reviewer gets their own row rather than overwriting the previous one, which is what makes inter-rater agreement computable. Readers that need one verdict per trace take the earliest round 1 row.
Round. Round 1 is the working label. Round 2 is a blind re-label pass used to measure self-agreement, and it never feeds validation.

Domain tags do not live on the trace. They live on the judge, where they define which slice of the trace pool that judge is validated against, and the gold screen serves you only traces carrying one of them. See judges for the tagging and gold-set workflow.

Promoting into a dataset¶

Promotion turns a reviewed trace into a dataset question, so a production failure becomes a regression test. Each card has a promote control that appends to an existing dataset or creates a new one by name, and the review landing page has a bulk control that promotes every trace currently labeled fail or defer in one action.

The promoted question keeps the trace faithfully. A multi-turn conversation is stored as its message array; a single-message trace is stored as the bare content string, so it matches a hand-authored question and stays editable. The label's verdict and note travel with it as provenance metadata, along with the trace key.

Three guards apply:

Duplicates. Existing question metadata is checked for the same trace key before appending, and disabled questions are scanned too, so re-running a bulk promotion does not silently double the dataset.
Ownership. Promoting an annotation from a result page checks that the annotation is yours. You cannot promote someone else's.
Integrity. See below.

The integrity guard¶

Promotion refuses a trace whose content is known to be incomplete:

Flag	Condition	Why it is refused
`truncated`	The payload was cut at the observability layer's cap (1024 characters, an upstream limit that is not yet configurable)	Half a conversation grades as a different question than the whole one.
`streamed_uncaptured`	Output tokens were counted but the response text is empty	The model streamed an answer the span never captured, so there is nothing to grade.

The default is refuse, because dirty gold silently poisons every eval downstream while a missing case is visible and re-addable once the trace is re-imported clean. A refusal is logged with the trace key and the reason, skips that trace, and continues; it changes no score and no verdict. The check runs before the duplicate check, so a dirty trace is flagged even if it was never promoted.

Both conditions are also marked on the review card itself, so you know you are judging a partial conversation before you decide anything about it.