Sharing results

A completed run can leave the site in two shapes: a raw JSON dump of everything the run produced, and a portable envelope meant to be read by someone else. This page is for whoever has to hand a result to a colleague, a client, or a pipeline.

The two shapes

	Raw results file	EEE envelope
Produced by	drush ai-eval:run --json	drush ai-eval:export-envelope, or automatically on run completion
Scope	Every target in one invocation	One file per result row
Contains	Per-question detail: the prompt, the stored response, every grader's score and reason	Aggregate metrics and provenance only
Meant for	Your own debugging	Sending somewhere else

The raw file is described under running evaluations. It is a dump, not an interchange format, and it carries prompt and response text. Read it before you send it anywhere.

Exporting an envelope

Envelopes are EEE (Every Eval Ever) 0.2.2 JSON, one file per stored result row.

drush ai-eval:export-envelope --target=my_target --since=-7days

ai-eval:export-envelope (alias aee) takes:

Option	Default	What it does
--target	every target	Restrict the export to one target id.
--since	no lower bound	Only export runs at or after this time. A Unix epoch, or anything strtotime parses, such as -7days.
--output-dir	results_path plus /envelopes	Where to write the files.

Files are named <target>-<timestamp>-<result id>.eee.json. A row that cannot be built is reported as a warning and skipped, and the remaining rows are still written.

Results imported from another site or tool are not exported. They are not this site's own measurement, and exporting them would label them first party with a gate that was never applied here. The command says how many it skipped.

The command exits 1 when it was given an unparseable --since, when it could not prepare the output directory, or when it found result rows but wrote no envelopes. It exits 0 when every result found was imported. It exits 0 otherwise, including when there was simply nothing to export.

Sites that want an envelope for every run can enable export_envelope_on_complete in configuration. It is off by default. A failed automatic export is logged and swallowed: it never breaks the run that produced it.

Importing an envelope

An envelope from another site or harness can be stored here as a result: Results > Import envelope, or

drush ai-eval:import-envelope results/ other-site-run.json

ai-eval:import-envelope (alias aeie) takes files or directories (every .json file in a directory) and reports how many were imported, skipped, in conflict or rejected. An imported result shows as imported, with no gate applied.

Each file is checked for duplicates by its evaluation_id:

• The same envelope imported again is skipped ("already imported"); a different measurement under the same evaluation_id is a conflict, and nothing is written.
• An envelope this site exported from one of its own runs is recognized as that run: it is skipped with "already on this site (run #N)" when its measurement matches the run, and is a conflict when it does not. The measurement compared is the score, pass rate, scale, sample size, interval and gate verdict of avg_score and pass_rate; versions, export times and judge trust details are not compared, since they describe the export rather than the run.

The match relies on the evaluation_id, which names the test, model, start time and result id but not the site. A run recorded before run snapshots existed rebuilds its identity from the test's current settings, so once that test is deleted or its model changed, its export no longer matches and imports as a new result. An envelope that lists avg_score or pass_rate twice is never taken as the same measurement.

The command exits 1 when nothing was imported and at least one file was rejected or in conflict.

What an envelope contains

• Identity of the run. Result id, target id and label, run timestamp, the run source tag, the Drupal and AI Eval versions, and the sha256 of the dataset.
• The model under test. Mode, agent id, provider and model id, the model identities the provider actually reported, and token usage.
• avg_score. The metric details also name non-empty must_pass grader ids from the run snapshot only; old results never inherit today's list. The score, the gate verdict with its metric, type and threshold, the error count, the duration, the question count, and the grader ids that produced it.
• pass_rate. The rate with its Wilson 95% confidence interval, so a rate from twelve questions is visibly less certain than one from four hundred. See scoring.
• corrected_pass_rate, only when the recorded run was scored by a single trusted LLM judge with known TPR and TNR, validated at the run's recorded question pass threshold. A validation with only endpoint scores is exempt when both thresholds are above 0 and at most 5. Absent in every other grader shape rather than emitted as an undefined number.
• Judge provenance. The judge provider and model in force at run time, the prompt version each judge scored, whether its decoding was pinned, and a per-judge trust block carrying the trust state, TPR, TNR, the raw confusion cells and their sum, the validation threshold, not_scored answers, and distinct pass_labels and fail_labels. Missing evidence is null; legacy rows retain their numeric threshold. Plus whether the judge shared a model family with the evaluated model. See judges.

Model identity depends on target mode. Agent exports use the agent ID as model_info.name and drupal-ai-agent/<agent_id> as model_info.id. Chat exports use the model ID as the name and <provider_id>/<model_id> as the ID; without a provider, the ID is the model ID alone.

Workflow exports use <engine>/<workflow_id> as the name and drupal-workflow/<engine>/<workflow_id> as the ID. Both parts must be nonempty; otherwise the existing agent identity applies, or unknown if no agent ID is set. Rows recorded before this change export the identity they had before, including inside evaluation_id. A validated snapshot without workflow keys never borrows workflow identity from live config. Rows with no snapshot (config_json is NULL) use live runner config. Malformed snapshots still use the existing live config fallback. Nonempty workflow engine, ID, snapshot, and output path also appear under model_info.additional_details as strings. The model identity is part of evaluation_id, followed by timestamp and result row ID.

A stepped run executes the workflow, input port, and fixed inputs recorded at start. The input port and fixed inputs stay out of the envelope details. A missing agent ID yields unknown; chat without a model ID falls back to agent identity.

Envelope fields are read from the run identity snapshot recorded when the run started, not from live configuration, so an export attributes a run to the configuration it began under.

The envelope says which source it used: snapshot, snapshot_with_drift when a live-only input changed mid-run, with the drifted field names listed, or live_config_fallback for rows predating the snapshot. Changing your judge model tomorrow does not re-attribute yesterday's exports.

The snapshot records provenance, it does not freeze execution

In a stepped run the graders are rebuilt for each question, and an LLM judge reads ai_eval.settings when it is constructed. So changing the judge provider or model halfway through a run does affect the questions still to come. That is detected rather than prevented: the changed inputs are listed as drift and the envelope is marked snapshot_with_drift. What the snapshot guarantees is that the export attributes the run to the configuration it began under, not that every question ran under it.

What it deliberately omits

An envelope is evidence about a run, not a copy of it. It carries no question text, no responses, no per-question scores, no per-grader reasons, no review labels or annotations, and no user identities.

The system prompt is present as a sha256 rather than as text, for two reasons: the recorded hash identifies the prompt that actually ran even after the target has been edited, and a system prompt is often the part of an eval a site is least willing to publish. Two envelopes with the same hash ran the same prompt, which is what a comparison needs, without the prompt leaving the site.

Warning

The one exception is a result row old enough to predate the run identity snapshot. Those rows fall back to live configuration and emit the target's current system prompt as text, under generation_config.generation_args.prompt_template. Such an envelope is flagged live_config_fallback in its source metadata. Check that field before sharing an export of historical runs.