Production traces¶

AI Eval imports OpenTelemetry GenAI spans from a live site, groups them into conversations and agent runs for review, lets you label them, and promotes the interesting ones into datasets. A dataset grown from traffic your site actually served covers inputs you would not have thought to write down. This page is for whoever does that review.

The loop:

flowchart LR
    OTLP["OTLP JSON export"]
    IMP["Import<br/>drush or admin form"]
    STORE["ai_eval_trace"]
    REV["Grouped review queue"]
    LAB["Labels<br/>pass, fail, defer"]
    DS["Dataset question"]
    GOLD["Judge gold set"]

    OTLP --> IMP --> STORE --> REV --> LAB
    LAB --> DS
    LAB --> GOLD

Viewing trace review needs operate ai eval or administer ai eval. Labeling individual traces and conversations needs annotate ai eval results, validate ai eval judges or administer ai eval. Promoting traces into datasets needs annotate ai eval results or administer ai eval. Operators without these permissions see existing labels read-only; the page explains which permissions to request.

Where the spans come from¶

Trace review works on OpenTelemetry GenAI spans written by ai_observability, a submodule of the AI module, through the OpenTelemetry PHP SDK.

Install and enable the OpenTelemetry module.
Enable the ai_observability submodule.
In its settings turn on otel_enabled, the master switch, which is off by default. Span emission (otel_spans) is already on, so flipping the master switch starts producing one span per provider call. To capture the prompt and response text you want to review, also turn on otel_spans_store_input and otel_spans_store_output, both off by default.

Then configure an OTLP file exporter so spans land as JSON on disk. An OpenTelemetry Collector with a file exporter is the simplest setup:

exporters:
  file:
    path: /var/log/otel/traces.json
service:
  pipelines:
    traces:
      exporters: [file]

For clean per-conversation grouping, set the assistant's allow_history setting to session. The assistant API stamps a per-session thread tag on each span, and AI Eval groups on it. Without it, each call reviews as an isolated group.

Importing¶

# One OTLP JSON file, or every *.json file in a directory
drush ai-eval:import-traces /var/log/otel/traces.json
drush ai-eval:import-traces /var/log/otel/

# With an explicit batch id
drush ai-eval:import-traces traces.json --batch=2026-06-30-prod

ai-eval:import-traces (alias aeit) takes one argument, a file or a directory, and one option:

Option	Default	What it does
`--batch`	the filename without its extension	The import batch id stamped on every row from that file.

Drush reads any path the shell user can read and has no permission check; restrict shell access accordingly.

A directory is scanned for *.json and sorted, so import order is deterministic across filesystems. The file format can be a single OTLP document, several concatenated documents, or newline-delimited JSON. Spans that are not AI spans are skipped, and the command finishes with a count line:

[OK] Imported 412 / 500 spans (88 skipped, 6 truncated at source).
Batch: traces. To undo this import, run: drush ai-eval:clear-traces -- 'traces'

When a file has damaged parts, the command also reports them:

[WARNING] 1 part of the input could not be read as a trace document and was skipped. The ai_eval log has the position of each part.
[WARNING] In 1 file, everything after an unclosed brace was not read. Fix the brace and import it again.

A stray closing brace, stray text or a closed object that is not a trace document is skipped and counted; later documents are still imported. An object that wraps a document is skipped whole, so nested payload data is never imported. An unclosed brace or an unterminated string stops the read: documents before it are imported, and everything after it is not read. Fix the file and import it again. Drush exits 0 with these warnings when it read at least one document; a file with no usable document still exits 1.

The command ends by naming the batch id and the command that undoes the import. It does this also when a later file fails after earlier files were imported.

Import is idempotent. Rows are keyed on the span id, so replaying a file updates existing rows in place rather than creating duplicates. A missing path, an unreadable file, or a payload with no OTLP document in it exits 1 with a message rather than reporting a false success.

Smaller imports can skip Drush. The trace review screen has an Import traces action at /admin/config/ai/ai-eval/trace-review/import that accepts an upload for users with operate ai eval or administer ai eval. The server path option appears only for administer ai eval. Operators who need a server file can ask an administrator or use Drush. The page runs the same importer through the Batch API, with a progress bar and the same four tallies at the end, with a link to undo it. Each UI import gets its own batch id, such as ui-upload-20261023-142501-3fa9c2, shown in the message at the end, which also links to the undo page. Uploads are capped at 8 MB per file; use Drush beyond that.

The import page shows the same skipped-parts and unread-tail warnings, including when no AI traces could be imported.

The batch id¶

Every imported row records its batch id. To undo an import, run drush ai-eval:clear-traces -- '<batch>' (alias aect). It deletes that batch's traces and tags after asking; -y skips the question. The id is used exactly as you type it, spaces included. An unknown id lists the batches on the site with their trace counts and last import time. Review labels and annotations are kept.

You can also undo an import from the screen. The Import traces page lists earlier imports (batch id, source, number of traces, when) with an Undo this import link for each. The link opens a confirmation page, and the delete runs only after you confirm. Importing an upload needs operate ai eval or administer ai eval. Importing from a server path needs administer ai eval. Undoing needs administer ai eval: only those users see the undo links, and anyone else gets "Access denied" on the undo page.

A span that is imported again under another id belongs to the newer id only, so clearing that id removes it. Older UI imports may share ui-import-upload or ui-import-path, and clearing one of those removes all of them. A trace imported again while a clear is running can be missed; there is no lock, so avoid importing and clearing at the same time.

The review queue¶

The trace review screen at /admin/config/ai/ai-eval/trace-review lists trace groups, 50 per page, most recent activity first. Spans are grouped rather than listed flat, because one answer is rarely one provider call:

Group type	What it is	Rendered as
`chat`	Spans sharing a conversation key, from an assistant thread or a multi-request agent thread	A chronological conversation thread
`agent_run`	Spans sharing a trace id with no conversation key	An indented call tree, parent above child
`single`	An `agent_run` group with exactly one member	A single card

Each group card shows the type, the providers and models involved, the member count, and a rollup of its members' verdicts. Date and tag filters narrow the list, and the filters travel with you into a group and with the prev and next group navigation, so a filtered review stays inside its filter.

Opening a group gives you a verdict bar for the whole group plus one card per member trace. A card carries the request and response text, token usage, the finish reason, the span's tags and remaining attributes, the labeling sidebar, and a promote control.

Reviewing needs operate ai eval or administer ai eval.

Labeling¶

A label is a verdict plus an optional note. The verdicts are:

Verdict	Meaning
`pass`	The answer was good.
`fail`	The answer was bad.
`defer`	Come back to this one. Not a gold verdict, and it does not count toward gold coverage.

Labels come in two scopes. A group verdict is one row per group, an open-coding "first thing that went wrong" for the whole conversation or run. A span label is per member trace, and that is the one downstream machinery reads.

Span labels are keyed by trace, dimension, labeler, and round:

Dimension. Labels written from the review queue carry no dimension: they are the holistic verdict on that trace. Labels written from a judge's gold screen are stamped with that judge's dimension, and an LLM judge is validated only against labels in its own dimension. A null dimension never means "any dimension", so one trace can carry a holistic label plus one label per judged dimension without collisions.
Labeler. Each reviewer gets their own row rather than overwriting the previous one, which is what makes inter-rater agreement computable. Readers that need one verdict per trace take the earliest round 1 row.
Round. Round 1 is the working label. Round 2 is a blind re-label pass used to measure self-agreement, and it never feeds validation.

Domain tags do not live on the trace. They live on the judge, where they define which slice of the trace pool that judge is validated against, and the gold screen serves you only traces carrying one of them. See judges for the tagging and gold-set workflow.

How the input is read¶

The plain format is a role on its own line, followed by the message text. A role line starts a message, except directly after another role line, where it is kept as text. Text before the first role line is kept as a user message. Windows line endings work, and developer is shown as system. Empty turns with a blank line after the role are kept.

This format has no escaping. A line that is only user or tool within a message can still split it after a text line. Promoted questions keep the raw input text in metadata.raw_input, up to 8000 characters, alongside the parsed input used for replay.

Turn on Summarize payload in ai_observability to get structured input, where roles are explicit. Trace review reads either format per span and shows tool names, call ids, arguments and the matching tool results. Structured replies show their text and tool calls. Embeddings show Embeddings: no text output and cannot be promoted: an embeddings call has no answer to test, and its input is the text it processed, not a question a visitor asked. A reply that is only the role label assistant: is a tool step whose call the legacy format did not record; it is not a final answer and cannot be promoted either. ai_observability can write a string-conversion diagnostic for non-chat output objects; review recognizes that diagnostic without changing the imported output or hiding chat replies.

ai_observability limits the length of each input and each output (the length limit of ai_observability, 1024 characters by default, unless the site changed it). With summarizing on, it also keeps 256 characters per string and 6 list items by default. Turning summarizing off removes those smaller limits, not the length limit of the whole input or output. Review shows a notice when input or output was cut, messages or calls were left out, or text was shortened or replaced. Only complete messages from the retained head are recovered. Such traces cannot be promoted. Traces with tool turns in structured input also cannot be promoted yet: replay cannot send those turns faithfully.

A reply containing only tool calls can be promoted when its input has no tool turns. Its calls are kept as text in metadata.ai_response_tools; raw_input is text too, so a single-user question stays editable. Observed calls do not fill Expected tools automatically.

Promoting into a dataset¶

Promotion turns a reviewed trace into a dataset question, so a production failure becomes a regression test. Each card has a promote control that appends to an existing dataset or creates a new one by name, and the review landing page has a bulk control, Promote all, that promotes every trace labeled fail or defer in one action.

Promote all counts two kinds of label:

A label on a single trace qualifies that trace.
A label on a whole conversation or agent run qualifies one trace that stands for it: the last turn of a conversation, whose input carries the earlier turns, or the final answer of an agent run. The question records the group's key, verdict, note and optional failure mode (group_key, group_verdict, group_note, group_failure_mode) next to the trace's own label, so a turn labeled pass inside a conversation labeled fail keeps both.

For an agent run, the final answer is the last chat call by time, then stored row id (or tree order when the source has no row ids), with non-empty text and no response tool calls. If there is no such answer, promotion takes the last chat call, then the first call if there are no chats. Its input is only the visitor's first user message, not the agent's tool history; its AI response is the selected reply. metadata.raw_input keeps the original input of the call that supplied that first user message, up to 8000 characters. A run without a user message is skipped. A cut first input is refused even when the final answer is intact. The call's own verdict and note stay unchanged, even when empty. The reviewer's run label lives in the group_* fields.

A trace that qualifies both ways becomes one question. The message after Promote all says how many traces were promoted and through which kind of label, and how many were already in the dataset or refused. A busy destination counts as a refusal for that trace; the remaining traces are still processed.

A new dataset is created only when the first question is written. If nothing qualifies, the message says so and no dataset is created. A labeled conversation or agent run whose traces are no longer on the site (its import was cleared) has nothing to stand for it; the message counts those too. A new name that another dataset already has (ignoring case and surrounding spaces) is refused, and the control comes back with the current list of datasets to choose from.

The promoted question keeps the trace faithfully. A multi-turn conversation is stored as its message array; a single-message trace is stored as the bare content string, so it matches a hand-authored question and stays editable. The label's verdict and note travel with it as provenance metadata, along with the trace key.

Three guards apply:

Duplicates. Existing question metadata is checked for the same trace key before appending, and disabled questions are scanned too, so re-running a bulk promotion does not silently double the dataset.
Ownership. Promoting an annotation from a result page checks that the annotation is yours. You cannot promote someone else's.
Integrity. See below.

Promoting an annotated answer¶

On a result page, Annotate a question, save the annotation, then use Add to dataset in the same form. The new question starts from the dataset question the answer was given for:

The form prefills What a good answer says and one expected fact from that question, and says where they came from. The form trims both fact fields, so it skips a fact with a blank value or surrounding spaces; such a fact travels as it is. The question's other expectations (other facts, must_not_contain, rubric_ref, tags and the like) travel with it without being shown. Clearing the criteria drops only the criteria; clearing both fact fields drops only that fact. Changing the fact's name replaces it.
The source is the dataset the run recorded. When that dataset is unchanged since the run, the form says the question is from the dataset this run used. When it changed since, or the run recorded no dataset and only the test's current one is known, the form says it is the dataset as it is now and may have changed since the run; check the fields before adding.
The new question keeps the source question's id, such as sunday_visit. When the target already holds the original question with that id and input, nothing is added and the message links to that question. When the id is taken by a different question, the new one becomes sunday_visit-2, sunday_visit-3 and so on. Only an answer whose question cannot be found gets an annotation-<id> id, as before.
The annotation's verdict, note and failure mode, and the source question's id, are kept in the question's metadata.

After a successful promotion, the controls become Added to [dataset name] with an Edit this question link when you can edit questions. A duplicate points to the existing question; it does not save changes typed into the old form. Use Add to another dataset to reopen the controls. Errors leave the form active so you can correct them. A promotion from another result creates a new copy and the message says so.

The integrity guard¶

Promotion refuses a trace whose content is incomplete or cannot be replayed:

Flag	Condition	Why it is refused
`truncated`	The payload was cut or summarized by the observability layer	Half a conversation grades as a different question than the whole one.
`tool_turns`	Structured input contains tool calls or tool results	Replay cannot send those turns yet.
`streamed_uncaptured`	Output tokens were counted but the response text is empty and there is no tool call	The model streamed an answer the span never captured, so there is nothing to grade.
`tool_step`	The response is only a role label and no tool call was recorded	This is an intermediate tool step, not a final answer to grade.
`no_answer`	The call has a non-text response kind, such as embeddings	There is no text answer to turn into an evaluation case.

The default is refuse, because dirty gold silently poisons every eval downstream while a missing case is visible and re-addable once the trace is re-imported clean. A refusal is logged with the trace key and the reason, skips that trace, and continues; it changes no score and no verdict. The check runs before the duplicate check, so a dirty trace is flagged even if it was never promoted.

These conditions are also marked on the review card itself, so you can see why a span cannot become an evaluation case before you decide anything about it.