Datasets¶

A dataset is the list of questions a run asks. This page covers where datasets come from, the YAML format each question follows, and the keys that graders read. The authoritative contract is schema/dataset.schema.json and schema/README.md; nothing here contradicts them.

Where datasets come from¶

Datasets are loaded through the AiEvalDatasetSource plugin type. Three sources ship, and a target's Dataset select shows all three in one grouped list, one optgroup per source.

Source	Plugin id	A reference is	Good for
File	`file`	a `.yaml` filename in the configured dataset directory	version control, review in merge requests
Config	`config`	the suffix of an `ai_eval.dataset.<ref>` config object	datasets shipped inside a module
Entity	`entity`	the numeric id of an `ai_eval_dataset` content entity	teams authoring in the browser

The file source resolves names against the dataset_path setting, falling back to the module's own data/ directory. It takes a bare filename only: a reference containing a path separator loads nothing.

Only the entity source is writable. It is the one that accepts new questions, so promoting a trace or an annotated failure into a dataset writes there. See trace review.

flowchart LR
    F["YAML files<br/>dataset_path/*.yaml"]
    C["Config objects<br/>ai_eval.dataset.*"]
    E["Content entities<br/>ai_eval_dataset + ai_eval_question"]
    LD["Dataset loader"]
    RUN["Run engine"]
    PROMO["Promotion from traces<br/>and annotations"]

    F --> LD
    C --> LD
    E --> LD
    LD --> RUN
    PROMO --> E

Writing your own source plugin is covered in writing a dataset source.

The YAML format¶

A dataset document is an object with a questions array. The smallest valid dataset is two lines of content:

# yaml-language-server: $schema=https://git.drupalcode.org/project/ai_eval/-/raw/1.0.x/schema/dataset.schema.json
questions:
  - id: q1
    input: "What is Drupal?"

The comment on the first line is worth keeping. Editors that speak the yaml-language-server directive give you completion and inline errors against the published schema.

Required keys¶

Every question needs id and input. Nothing else is required.

Optional keys¶

Key	Type	What reads it
`criteria`	string	The prose standard the LLM judges grade against. A resolved `rubric_ref` appends to it.
`expected`	object	Grader-specific expectations. `expected.format` is one of `text`, `json`, `yaml`, `markdown` and is read by `format_grader`.
`expected_facts`	object	Ground truth handed to `accuracy_grader` and `fact_match_grader`.
`expected_tools`	array	Assertions about the tools an agent ran. See below.
`rubric_ref`	string	`rubric/<id>@<version>`. Attaches a shared rubric.
`choices`	array of strings	Multiple-choice options. Portable interchange shape; the runner does not score multiple choice natively.
`files`	object	Filename to content or path map. Input-side declaration only; the runner stages nothing into a sandbox.
`disabled`	boolean	`true` excludes the question from every run.
`split`	string	`train`, `validation`, or `test`. See below.
`bundle`	string	Snake_case eval domain this case belongs to. A question lives in exactly one.
`tags`	array of strings	Free-form labels.
`metadata`	object	Case-level carry-all. Judge templates may reference `{{ metadata.<key> }}`.

Unknown keys on a question are rejected by the schema, and the config and entity sources enforce that at load. The file source does not validate, so a typo such as expecteed_facts in a YAML file is not caught: the key is carried along and silently does nothing at run time. Validate file datasets yourself, either with the editor schema comment below or by loading them through a validating source.

Warning

The runner also reads user_name and caller_uid at the question level to run a question as a specific Drupal user, and neither is in the schema. Rows carrying them validate nowhere, so they load from the file source, which does not validate, and are rejected by the config and entity sources, which do.

Ground truth for the fact-checking judges¶

accuracy_grader reads expected_facts and presents it to the judge as ground truth. Rows without it are graded against the prose criteria alone, and fact_match_grader skips them entirely.

questions:
  - id: Q01
    input: "What do we know about project Acme? Team, status, recent activity."
    criteria: "Should mention project manager and active team."
    expected_facts:
      project_manager: "Jane Doe"
      active_team: ["Alex Chen", "Sam Park"]
      must_not_contain: ["Chris Lee", "Pat Rivera"]

The nested must_not_contain list is legacy. It remains valid for the life of the 1.0.x branch, but the architecturally clean home for the same constraint is a must_not_contain rubric check, which carries a case_sensitive flag and takes part in the rubric's combine rule. New content should use the rubric form.

Multi-turn input¶

input can be an array of chat messages instead of a string. Each message needs a role (system, user, assistant, or tool) and a content string.

questions:
  - id: q_multi_turn
    input:
      - role: user
        content: "Please review the module skeleton in the attached files."
      - role: assistant
        content: "I see a custom block plugin. What is the review goal?"
      - role: user
        content: "Check for service injection and dependency hygiene."

Chat mode sends the conversation to the provider as stored. Agent mode flattens it into a single role: content transcript at execution time, because the agent API accepts one string; the stored dataset is never modified. If a stored conversation carries its own system message, the target's configured system prompt wins, since that prompt is the thing under test.

Asserting on tools¶

expected_tools asserts what an agent did while answering, which is a different claim from anything a grader can make about the text it produced. It is scored by tool_usage_grader against the agent's own record of the tools it executed. Sub-agent calls count: a tool a delegate called still ran during this question.

Each entry needs tool and should_run.

expected_tools:
  - tool: list_content_types
    should_run: true
  - tool: delete_content_type
    should_run: false

A tool may be named either way and both are accepted: the function name the model emits (list_content_types), or the implementation-side plugin id (ai_agent:list_content_types). A name containing a colon can only be a plugin id, since function names carry none. A bare name is checked against both, which leaves one ambiguity: if one tool's plugin id equals another tool's function name, a bare assertion cannot separate them.

`args`¶

args asserts what a call was given, and is the only thing that separates two calls to the same tool. It is a subset match, so name the arguments the assertion is about and any others the model passed are ignored.

expected_tools:
  - tool: get_content_type_info
    should_run: true
    args:
      node_type: kb_doc

That fails when the tool ran only for some other content type, which plain should_run would have passed. Three rules follow from how a record stores arguments:

Values compare as text, so 12 and "12" are the same assertion. Booleans and null compare strictly, because the usual coercions make false indistinguishable from an empty string.
Values must be scalar. A record stores the shape of a structured argument rather than its contents, so such an assertion could never match and is rejected.
An empty args object is rejected rather than read as an assertion about nothing.

`after`¶

after asserts order, naming a tool that must have run first. Read it as "by the time this ran, that had happened", compared on the first occurrence of each.

expected_tools:
  - tool: edit_content_type
    should_run: true
    after: get_content_type_info

args narrows which call of this tool the ordering is about; the named predecessor cannot be narrowed the same way, so any of its calls satisfies it. Two shapes are rejected rather than accepted and ignored: a blank after, which asserts nothing while reading like an ordering check, and after alongside should_run: false, which is incoherent because a tool that must not have run has no position to compare.

Warning

args and after are answerable only from a tool-call record. Chat-mode targets have none, and agents defined in code rather than as configuration expose none either. In that position the grader falls back to scanning the response text, which carries names at best, so an args or after expectation fails loudly and says why rather than being scored as met.

What no version of this carries is what a tool returned. Those are the agent's output, not a record of its behavior.

Splits¶

A labeled question belongs to that split only, which is how validation and test rows stay held out of optimizer training. Labeling every row by hand is avoidable: a dataset-level splits block hashes unlabeled questions and buckets them against cumulative ratios, deterministically.

splits:
  rule: hash
  field: id
  ratios:
    train: 0.7
    validation: 0.15
    test: 0.15
  seed: 1
questions:
  - id: q_rule_1
    input: "Summarize the release notes."
  - id: q_rule_3
    input: "List the enabled modules."
    split: test

An explicit label always wins over the rule. field defaults to id, the ratios must sum to 1.0, and changing seed reshuffles the assignment deterministically. A question covered by neither a label nor a rule lands in both train and validation, never test. drush ai-eval:run --split= selects one at run time; the default runs everything.

Where the schema is enforced¶

The config and entity sources validate the whole document against schema/dataset.schema.json before returning anything, log the errors, and return an empty dataset on failure. The file source does not validate: it parses the YAML, drops rows that are not arrays, drops rows marked disabled, and hands the rest to the run. A YAML parse error in a file dataset reads as an empty dataset, which is the usual cause of "No questions loaded".

Validate file datasets yourself before a run. Any Draft 2020-12 validator works; see the portable schemas for editor, command line, and CI setups.

Authoring in the browser¶

The Datasets tab (/admin/config/ai/ai-eval/datasets) holds entity-source datasets. Add dataset creates one; its question manager adds, edits, reorders, disables, and deletes questions without touching a file. A generate button asks an LLM to draft synthetic questions from dimensions and seed examples you supply; drafts land in the dataset for you to review before they count.

The editor models scalar fields, so some questions are shown read-only rather than risking a silent flattening on save. That covers multi-turn input, any nested non-scalar value under expected, expected_facts, metadata or files, and any expected_tools entry carrying more than tool and should_run. A question using args or after therefore reads as read-only in the browser, and stays authored in YAML or in the entity's data blob.

Working examples¶

schema/examples/valid/ holds a fixture corpus that CI validates on every run. The fastest way to see a shape is to read the matching file: dataset-minimal, dataset-multi-turn, dataset-tool-assertions (presence, args, and after side by side), dataset-split-rule and dataset-split-explicit, dataset-with-rubric-ref, dataset-expected-yaml, dataset-mcq, dataset-metadata, and dataset-with-files.

schema/examples/invalid/ is the mirror image: files that must fail validation, including the five rejected expected_tools shapes described above.