Skip to content

Sharing results

A completed run can leave the site in two shapes: a raw JSON dump of everything the run produced, and a portable envelope meant to be read by someone else. This page is for whoever has to hand a result to a colleague, a client, or a pipeline.

The two shapes

Raw results file EEE envelope
Produced by drush ai-eval:run --json drush ai-eval:export-envelope, or automatically on run completion
Scope Every target in one invocation One file per result row
Contains Per-question detail: the prompt, the stored response, every grader's score and reason Aggregate metrics and provenance only
Meant for Your own debugging Sending somewhere else

The raw file is described under running evaluations. It is a dump, not an interchange format, and it carries prompt and response text. Read it before you send it anywhere.

Exporting an envelope

Envelopes are EEE (Every Eval Ever) 0.2.2 JSON, one file per stored result row.

drush ai-eval:export-envelope --target=my_target --since=-7days

ai-eval:export-envelope (alias aee) takes:

Option Default What it does
--target every target Restrict the export to one target id.
--since no lower bound Only export runs at or after this time. A Unix epoch, or anything strtotime parses, such as -7days.
--output-dir results_path plus /envelopes Where to write the files.

Files are named <target>-<timestamp>-<result id>.eee.json. A row that cannot be built is reported as a warning and skipped, and the remaining rows are still written.

The command exits 1 when it was given an unparseable --since, when it could not prepare the output directory, or when it found result rows but wrote no envelopes. It exits 0 otherwise, including when there was simply nothing to export.

Sites that want an envelope for every run can enable export_envelope_on_complete in configuration. It is off by default. A failed automatic export is logged and swallowed: it never breaks the run that produced it.

What an envelope contains

  • Identity of the run. Result id, target id and label, run timestamp, the run source tag, the Drupal and AI Eval versions, and the sha256 of the dataset.
  • The model under test. Mode, agent id, provider and model id, the model identities the provider actually reported, and token usage.
  • avg_score. The score, the gate verdict with its metric, type and threshold, the error count, the duration, the question count, and the grader ids that produced it.
  • pass_rate. The rate with its Wilson 95% confidence interval, so a rate from twelve questions is visibly less certain than one from four hundred. See scoring.
  • corrected_pass_rate, only when the recorded run was scored by a single trusted LLM judge with known TPR and TNR. Absent in every other grader shape rather than emitted as an undefined number.
  • Judge provenance. The judge provider and model in force at run time, the prompt version each judge scored, whether its decoding was pinned, and a per-judge trust block carrying the trust state, TPR, TNR, the raw confusion cells and their sum. Plus whether the judge shared a model family with the evaluated model. See judges.

Envelope fields are read from the run identity snapshot recorded when the run started, not from live configuration, so an export attributes a run to the configuration it began under.

The envelope says which source it used: snapshot, snapshot_with_drift when a live-only input changed mid-run, with the drifted field names listed, or live_config_fallback for rows predating the snapshot. Changing your judge model tomorrow does not re-attribute yesterday's exports.

The snapshot records provenance, it does not freeze execution

In a stepped run the graders are rebuilt for each question, and an LLM judge reads ai_eval.settings when it is constructed. So changing the judge provider or model halfway through a run does affect the questions still to come. That is detected rather than prevented: the changed inputs are listed as drift and the envelope is marked snapshot_with_drift. What the snapshot guarantees is that the export attributes the run to the configuration it began under, not that every question ran under it.

What it deliberately omits

An envelope is evidence about a run, not a copy of it. It carries no question text, no responses, no per-question scores, no per-grader reasons, no review labels or annotations, and no user identities.

The system prompt is present as a sha256 rather than as text, for two reasons: the recorded hash identifies the prompt that actually ran even after the target has been edited, and a system prompt is often the part of an eval a site is least willing to publish. Two envelopes with the same hash ran the same prompt, which is what a comparison needs, without the prompt leaving the site.

Warning

The one exception is a result row old enough to predate the run identity snapshot. Those rows fall back to live configuration and emit the target's current system prompt as text, under generation_config.generation_args.prompt_template. Such an envelope is flagged live_config_fallback in its source metadata. Check that field before sharing an export of historical runs.