Datasets¶

A dataset is the list of questions a run asks. This page covers where datasets come from, the YAML format each question follows, and the keys that graders read. The authoritative contract is schema/dataset.schema.json and schema/README.md; nothing here contradicts them.

Where datasets come from¶

Datasets are loaded through the AiEvalDatasetSource plugin type. Three sources ship, and a target's Dataset select shows all three in one grouped list, one optgroup per source.

Source	Plugin id	A reference is	Good for
File	`file`	a `.yaml` or `.yml` filename in a discovered dataset directory	version control, review in merge requests
Config	`config`	the suffix of an `ai_eval.dataset.<ref>` config object	datasets shipped inside a module
Entity	`entity`	the numeric id of an `ai_eval_dataset` content entity	teams authoring in the browser

The file source searches the configured dataset_path, this module's data/ directory, then enabled modules' datasets/ directories in module-name order. It accepts .yaml and .yml filenames. The first matching filename wins; duplicates produce a warning. References must be bare filenames without path separators.

Only the entity source is writable. It is the one that accepts new questions, so promoting a trace or an annotated failure into a dataset writes there. See trace review.

flowchart LR
    F["YAML files<br/>dataset_path/*.yaml"]
    C["Config objects<br/>ai_eval.dataset.*"]
    E["Content entities<br/>ai_eval_dataset + ai_eval_question"]
    LD["Dataset loader"]
    RUN["Run engine"]
    PROMO["Promotion from traces<br/>and annotations"]

    F --> LD
    C --> LD
    E --> LD
    LD --> RUN
    PROMO --> E

Writing your own source plugin is covered in writing a dataset source.

Datasets shared by several tests¶

When you edit a test whose entity dataset is also attached to other tests, the guided workflow lists those tests under Used by. Choose what the change is supposed to mean:

Update the shared dataset opens the dataset editor. Every test attached to that dataset sees the saved question changes.
Create a copy for this test creates an entity dataset and keeps the other tests attached to the original. Save the test changes to attach the copy to this test.

File and config datasets are read-only. The same copy action turns either one into an editable entity dataset. That copy preserves the enabled questions returned by the source, including their IDs and question data, in load order. Disabled file or config rows are not copied because source loading filters them out. An entity dataset copy preserves every stored question row verbatim, including its ID, order, disabled state, bundle, and data. The copy is independent after it is created.

Per-question split labels are preserved. A dataset-level splits rule is not copied because editable datasets cannot store it. The copy still succeeds, but the guided workflow shows a warning and logs the omission. Without that rule, questions without a split label belong to both train and validation, never test. Add per-question labels in the source before copying if you need to keep those partitions.

That independence also applies to promotion. A question promoted into the original dataset does not appear in an existing copy, and a question promoted into the copy does not change the original. Choose the dataset attached to the test whose regression set you intend to expand.

The YAML format¶

A dataset document is an object with a questions array. The smallest valid dataset is two lines of content:

# yaml-language-server: $schema=https://git.drupalcode.org/project/ai_eval/-/raw/1.1.x/schema/dataset.schema.json
questions:
  - id: q1
    input: "What is Drupal?"

The comment on the first line is worth keeping. Editors that speak the yaml-language-server directive give you completion and inline errors against the published schema.

Required keys¶

Every question needs id and input. Nothing else is required.

Optional keys¶

Key	Type	What reads it
`criteria`	string	The prose standard the LLM judges grade against. A resolved `rubric_ref` appends to it.
`expected`	object	Grader expectations. `expected.format` is read by `format_grader`. `expected.items` is read by `recall_at_k_grader` and `reciprocal_rank_grader`. A `target_match` check can name an `expected.<name>` reference through `reference_path`.
`expected_facts`	object	Ground truth handed to `accuracy_grader` and `fact_match_grader`.
`expected_tools`	array	Assertions about the tools an agent ran. See below.
`rubric_ref`	string	`rubric/<id>@<version>`. Attaches a shared rubric.
`choices`	array of strings	Multiple-choice options. Portable interchange shape; the runner does not score multiple choice natively.
`files`	object	Filename to image URI, data URL, relative fixture path (FileSource only), or plain text. Supported images reach the model and LLM judges; other values are carried but not sent.
`disabled`	boolean	`true` excludes the question from every run.
`split`	string	`train`, `validation`, or `test`. See below.
`bundle`	string	Snake_case eval domain this case belongs to. A question lives in exactly one.
`tags`	array of strings	Free-form labels.
`metadata`	object	Case-level carry-all. Judge templates may reference `{{ metadata.<key> }}`. `metadata.inputs`, when present, must be a map: a workflow target passes it to the workflow as this question's own inputs (see targets).

Unknown keys on a question are rejected by the schema. File, config, and entity sources validate before returning questions, so a misspelled field such as expecteed_facts cannot silently remove a grading expectation. Invalid files return no questions and log the validation errors.

The optional question fields caller_uid (a non-negative integer) and user_name (a string) are schema-valid. caller_uid overrides the run-level caller identity for that question. The target invocation, including a workflow, runs as that Drupal account. Graders keep the run account.

user_name adds [Caller: name] to string input or the first user message in every mode. Search API and AI Answers recommend omitting it because it becomes part of the query.

The question editor supports Caller account, Name in the prompt, and Dataset split. Pick the caller account from the list of users, or type a non-negative integer user ID. Choose Anonymous (0) for anonymous. A stored ID without an account here shows User 42 (no account on this site), with the stored ID in place of 42. Name in the prompt adds [Caller: name] before the question. It changes only the text the model sees, not the account the question runs as. Choose train, validation, or test for the split. Empty fields are omitted on save, so editing a question without these optional values does not add them.

Images in questions¶

Add images in the question editor with Upload images, or enter them in Files as filename and image reference pairs. Uploads become managed files under private://ai_eval/question-files/ when private storage is configured, otherwise under public://ai_eval/question-files/. The question keeps the URI. Remove its Files entry to stop sending an image.

For a worked example from upload to result, see the tutorial Test an AI feature that reads images.

Supported images are png, jpeg, webp and gif, up to 5 MB each and 5 per question. The editor checks the extension, actual image type and size. References can be public:// or private:// URIs, or data:image/png;base64,... URLs with the matching image MIME type. FileSource YAML datasets can also use paths relative to their YAML file; resolved paths, including symlinks, must stay inside that folder. Remote URLs are refused. Other plain text file values are carried but never sent.

Chat and config agent targets attach images to the last user message. AI Prompt workflow targets send them with the rendered prompt. Other workflow runners and code-defined agents skip image questions. A model without image support also skips the question with a reason: it has no score and does not count as a zero. If an agent's model cannot be resolved, the images are sent and the question row shows a warning.

LLM judges receive the same images. A judge without image support returns no score and explains why. Validation from reviewed results reloads the source question's files only when its dataset still matches the run; otherwise the review is left out and the validation page explains the count. Trace labels carry no image files.

Expand a question on the results page to see its image thumbnails and any skip reason. The page says when it is showing a changed dataset. Run identity records each image's name, SHA-256, MIME type and byte count, without storing image contents in the results or exported envelopes.

Ground truth for the fact-checking judges¶

accuracy_grader reads expected_facts and presents it to the judge as ground truth. Rows without it are graded against the prose criteria alone, and fact_match_grader skips them entirely.

questions:
  - id: Q01
    input: "What do we know about project Acme? Team, status, recent activity."
    criteria: "Should mention project manager and active team."
    expected_facts:
      project_manager: "Jane Doe"
      active_team: ["Alex Chen", "Sam Park"]
      must_not_contain: ["Chris Lee", "Pat Rivera"]

The nested must_not_contain list is legacy. It remains valid for the life of the 1.1.x branch, but the architecturally clean home for the same constraint is a must_not_contain rubric check, which carries a case_sensitive flag and takes part in the rubric's combine rule. New content should use the rubric form.

Multi-turn input¶

input can be an array of chat messages instead of a string. Each message needs a role (system, user, assistant, or tool) and a content string.

questions:
  - id: q_multi_turn
    input:
      - role: user
        content: "Please review the module skeleton in the attached files."
      - role: assistant
        content: "I see a custom block plugin. What is the review goal?"
      - role: user
        content: "Check for service injection and dependency hygiene."

Chat mode and config agents send the conversation as messages, with each turn's role and content preserved. Single-turn agent questions also reach the model as written, including HTML, without a Task wrapper. Agent scores can change compared with earlier versions. Code-defined agents that read only the Task still receive a role: content transcript, as do workflow targets. The stored dataset is not modified. In chat mode and config agents, dataset system messages are dropped so the target's configured system prompt remains authoritative, including optimizer candidate prompts.

Asserting on tools¶

expected_tools asserts what an agent did while answering, which is a different claim from anything a grader can make about the text it produced. It is scored by tool_usage_grader against the agent's own record of the tools it executed. Sub-agent calls count: a tool a delegate called still ran during this question.

Each entry needs tool and should_run.

expected_tools:
  - tool: list_content_types
    should_run: true
  - tool: delete_content_type
    should_run: false

A tool may be named either way and both are accepted: the function name the model emits (list_content_types), or the implementation-side plugin id (ai_agent:list_content_types). A name containing a colon can only be a plugin id, since function names carry none. A bare name is checked against both, which leaves one ambiguity: if one tool's plugin id equals another tool's function name, a bare assertion cannot separate them.

`args`¶

args asserts what a call was given, and is the only thing that separates two calls to the same tool. It is a subset match, so name the arguments the assertion is about and any others the model passed are ignored.

expected_tools:
  - tool: get_content_type_info
    should_run: true
    args:
      node_type: kb_doc

That fails when the tool ran only for some other content type, which plain should_run would have passed. Three rules follow from how a record stores arguments:

Values compare as text, so 12 and "12" are the same assertion. Booleans and null compare strictly, because the usual coercions make false indistinguishable from an empty string.
Values must be scalar. A record stores the shape of a structured argument rather than its contents, so such an assertion could never match and is rejected.
An empty args object is rejected rather than read as an assertion about nothing.

`after`¶

after asserts order, naming a tool that must have run first. Read it as "by the time this ran, that had happened", compared on the first occurrence of each.

expected_tools:
  - tool: edit_content_type
    should_run: true
    after: get_content_type_info

args narrows which call of this tool the ordering is about; the named predecessor cannot be narrowed the same way, so any of its calls satisfies it. Two shapes are rejected rather than accepted and ignored: a blank after, which asserts nothing while reading like an ordering check, and after alongside should_run: false, which is incoherent because a tool that must not have run has no position to compare.

Warning

args and after are answerable only from a tool-call record. Chat-mode targets have none, and agents defined in code rather than as configuration expose none either. In that position the grader falls back to scanning the response text, which carries names at best, so an args or after expectation fails loudly and says why rather than being scored as met.

What no version of this carries is what a tool returned. Those are the agent's output, not a record of its behavior.

Splits¶

A labeled question belongs to that split only, which is how validation and test rows stay held out of optimizer training. Labeling every row by hand is avoidable: a dataset-level splits block hashes unlabeled questions and buckets them against cumulative ratios, deterministically.

splits:
  rule: hash
  field: id
  ratios:
    train: 0.7
    validation: 0.15
    test: 0.15
  seed: 1
questions:
  - id: q_rule_1
    input: "Summarize the release notes."
  - id: q_rule_3
    input: "List the enabled modules."
    split: test

An explicit label always wins over the rule. field defaults to id, the ratios must sum to 1.0, and changing seed reshuffles the assignment deterministically. A question covered by neither a label nor a rule lands in both train and validation, never test. drush ai-eval:run --split= selects one at run time; the default runs everything.

Where the schema is enforced¶

All three sources validate the whole document against schema/dataset.schema.json before returning questions. Invalid documents are rejected and logged. The file source also logs YAML parse failures. A malformed question rejects the document rather than silently dropping that row; valid questions marked disabled are excluded after validation.

Editor and CI validation can catch these errors before a run. Any Draft 2020-12 validator works; see the portable schemas for setups.

Authoring in the browser¶

Manage > Datasets (/admin/config/ai/ai-eval/datasets) holds entity-source datasets. Add dataset creates one; its question manager adds, edits, reorders, disables, and deletes questions without touching a file. The admin UI does not offer AI-generated questions: you write each question yourself, or promote one from a reviewed answer or trace.

Each question editor has its own Save, which stores that question at once. The dataset's Save at the bottom of the page saves the dataset's own fields, not an open question editor. While an editor holds changes you have not saved (typed text, a changed choice, an added or removed row), the dataset's Save stops and the editor says "Save or cancel this question first". So do Add question and the move and delete buttons of other questions (moving or deleting redraws the whole list), and leaving the page asks before it discards them. Save the question, or cancel it, then carry on.

Rubric reference is a select of the enabled rubric versions on this site, each shown as its label and ref, for example Tone (rubric/tone@1.0). A stored ref the site does not have (a rubric from another site, or one disabled since) stays selected and is marked "(not on this site)", so saving the question does not drop it. Expected tools suggests the AI module's function calls by function name and by plugin id, the two names the tool usage grader matches; a tool the site does not list can still be typed. Domain bundle suggests the bundles already used by questions and named by rubrics on this site, and Tags the tags the questions of this dataset already use; a new value can still be typed.

The Upload images field saves image files with the question. The Files repeater can also keep existing image URIs and other plain text entries.

The editor models scalar fields, so some questions are shown read-only rather than risking a silent flattening on save. That covers multi-turn input, any nested non-scalar value under expected, expected_facts, metadata or files, and any expected_tools entry carrying more than tool and should_run. The one exception is metadata.inputs: the editor shows it as JSON text and saves it back as a map, so a question with workflow inputs stays editable, and typing a JSON object such as {"language": "el"} as its value stores a map. An empty {} or [] is left out on save. Any other value that is not a JSON object (a list, a number, broken JSON), or an object holding an empty {} further down (stored questions cannot keep it apart from []), is kept as typed and the save is refused with a message saying why. A question using args or after therefore reads as read-only in the browser, and stays authored in YAML or in the entity's data blob.

Deleting a dataset¶

Delete on a dataset asks for confirmation. The page says how many questions go with the dataset and lists the tests that use it. Those tests cannot run until you choose another dataset for them (the Run button says so). Stored results, runs, annotations and review labels are kept. What goes with the questions is their authoring data (criteria, expected facts, files, rubric references), so a result page that reloads those fields shows less, and promoting a trace or annotation later falls back to the input the result recorded. Questions left by a dataset deleted before this change stay in the database; dropdown bundle suggestions and test case counts ignore them. Promoting the same trace into a new dataset uses its recorded input and does not recover criteria added only to a deleted question.

Working examples¶

schema/examples/valid/ holds a fixture corpus that CI validates on every run. The dataset fixtures are:

dataset-automator-field.yaml
dataset-caller-identity.yaml
dataset-expected-yaml.yaml
dataset-mcq.yaml
dataset-metadata.yaml
dataset-minimal.yaml
dataset-multi-turn.yaml
dataset-retrieval.yaml
dataset-split-explicit.yaml
dataset-split-rule.yaml
dataset-taxonomy-labels.yaml
dataset-tool-assertions.yaml
dataset-with-files.yaml
dataset-with-rubric-ref.yaml
dataset-workflow-inputs.yaml

schema/examples/invalid/ contains documents that must fail validation. These include the five rejected tool assertion shapes, a bare retrieval item ID in dataset-retrieval-items-bare-id.yaml, and JSON-string workflow inputs in dataset-workflow-inputs-string.yaml. The runner's internal input parser accepts a JSON object string, but the dataset schema requires a map.

Field inputs for automator targets¶

A question can supply source fields under metadata.inputs. These override fixed workflow_inputs for that question only; the question text goes on the configured source-field port. Scalars fill a field's main property, maps represent one item, and lists represent multiple items. Reference IDs must already exist on the evaluation site.

- id: article_summary
  input: 'The source article to summarize.'
  metadata:
    inputs:
      title: 'Library news'
      field_context:
        value: 'Background for this question.'
        format: plain_text
      field_topics:
        - target_id: 12
        - target_id: 18

For plain text on formatted fields, set text_format on the target or in metadata.inputs. Do not supply the target field, entity identity keys, or another value for the question port.

Retrieval expectations¶

For a ranked retrieval response, put expected page ids under expected.items:

questions:
  - id: open_day
    input: When is the fall open day?
    caller_uid: 0
    expected:
      items:
        - node/108
        - entity:node/41:en

node/108 is shorthand for entity:node/108; either accepts any translation. entity:node/108:en accepts only that translation. Other datasources use exact ids, such as custom_source/raw. Bare numbers are ambiguous. Use full page ids with language codes for string entity ids containing colons. Chunk expectations are not supported: a chunk suffix matches literally and will not match a page id.

Entity shorthand is node-only: use node/108 for nodes. For other entity types, use full datasource IDs such as entity:media/5 or entity:taxonomy_term/3. media/5 and taxonomy_term/3 are treated as exact datasource IDs, not entity shorthand, and will not match those entity results.

Exact duplicate expectations merge. Overlapping scopes, such as node/108 and entity:node/108:en together, are errors. Two different translations remain separate expectations. The list must contain at least one ID with a nonempty datasource and raw part, separated by /. The schema rejects 108 and node/. Without expected.items, retrieval graders skip the question.

Questions with lists in expected remain read-only in the question editor. Edit them in YAML. Choose expected pages the caller can view; inaccessible pages count as misses. These ids refer to content on one site.

Reference outputs for field automators¶

A target_match check can read any expected.<name> through reference_path. Use expected.reference for the final field text, for example. Taxonomy outputs contain label lines in order, so write the whole output as one YAML block string. A list of strings means alternative full responses. See dataset-automator-field.yaml, dataset-taxonomy-labels.yaml, and rubric-target-match-automator.yaml in schema/examples/valid/.