Skip to content

The portable JSON Schemas

ai_eval ships five JSON Schemas for datasets, rubrics, judges, failure taxonomies, and decision rubric configuration. They are a public contract for Drupal Eval Commons, not private validation code, and this module is their reference implementation. For developers writing eval content, tooling that reads or writes it, or a runner other than ai_eval.

The files

Under schema/ in the module root, all Draft 2020-12.

File Describes
dataset.schema.json A bundle of eval questions with inputs, expected facts, tool assertions, and an optional rubric reference.
rubric.schema.json A reusable scoring rubric: ordered checks combined by a scoring rule.
judge.schema.json A reusable LLM-judge prompt with score type, optional level names, and calibration metadata.
failure_taxonomy.schema.json A failure-mode taxonomy document.
decision_rubric.schema.json Rules and scoring settings for the decision rubric grader.

schema/README.md is the contract. It carries the full check-kind table, the reference-string format, the case-level tool-assertion semantics, the Inspect AI field mapping, and the versioning policy. This page tells you where things are and what the rules mean for your code; when the two differ, the schema README wins.

Why they are public

The schemas describe a portable interchange format. Another tool can read a dataset written for ai_eval, and ai_eval can run a dataset written for another tool, as long as both validate against the same file. That is why changing one is an ecosystem-facing decision rather than a local refactor, and why the URLs are pinned rather than relative.

Schema-valid is not runner-executable

Two separate properties, deliberately kept apart.

  • Schema-valid: the artifact passes validation against the relevant schema.
  • Runner-executable: a given runtime can fetch each declared input, evaluate each declared check and write a score.

ai_eval executes a strict subset. In a schema-valid rubric, command and score_delta checks are skipped here, because they need a sandbox that ai_eval does not have; the other checks in the rubric still run. tool_usage as a rubric check is contract vocabulary that ai_eval never executes: the implemented form is the case-level expected_tools key. The full split is in writing a check executor and in the schema README.

This is a feature of the contract, not a gap in it. A portable format has to be able to express things a particular runner has not implemented yet, otherwise nobody can write the artifact that motivates the implementation.

Editor autocompletion

Add a yaml-language-server comment as the first line of any eval YAML. Editors that speak the directive, which includes VS Code with the Red Hat YAML extension and any LSP client using yaml-language-server, then give you completion, hover documentation and inline errors.

# yaml-language-server: $schema=https://git.drupalcode.org/project/ai_eval/-/raw/1.1.x/schema/dataset.schema.json
questions:
  - id: q1
    input: "What is Drupal?"

Use the matching URL for rubric and judge files:

# yaml-language-server: $schema=https://git.drupalcode.org/project/ai_eval/-/raw/1.1.x/schema/rubric.schema.json
id: code_gen_basic
version: "1.0.0"
checks:
  - kind: must_contain_any
    values: ["namespace Drupal\\"]
scoring:
  combine: all_pass

The comment is inert to every YAML parser, so it costs nothing at run time.

Validating outside Drupal

Any Draft 2020-12 validator works. With check-jsonschema:

pip install check-jsonschema
check-jsonschema \
  --schemafile https://git.drupalcode.org/project/ai_eval/-/raw/1.1.x/schema/dataset.schema.json \
  my-dataset.yaml

With Node's ajv-cli:

npx ajv-cli@latest validate \
  --spec=draft2020 \
  -s dataset.schema.json \
  -d my-dataset.yaml

Unknown fields on a check are rejected, via unevaluatedProperties: false, so a typo surfaces at validation rather than as a silently skipped check at run time.

Validating inside Drupal

Three services wrap the schemas, each built from the shipped file:

Service Validates
ai_eval.dataset_validator A dataset document
ai_eval.rubric_validator A rubric document, including check kinds contributed by tagged executors
ai_eval.failure_taxonomy_validator A failure taxonomy document

Each exposes validate() returning a bool and errors() returning the messages from the last call. The shipped file, config, and entity dataset sources call ai_eval.dataset_validator before returning any questions; do the same in a dataset source of your own.

ai_eval.rubric_validator is the one that changes shape at runtime: it splices in the definitions of every tagged check executor that implements CheckSchemaProviderInterface, so a rubric using a kind from your module validates on a site where your module is installed and is rejected as a typo where it is not.

The example corpus

schema/examples/ holds fixtures that the in-tree test suite asserts against, which makes them the most reliable reference for what a valid document looks like.

  • schema/examples/valid/ must all pass. Start here when authoring: there is a minimal dataset, a multi-turn one, both split forms, tool assertions, an MCQ case, a binary judge, and rubrics covering all_pass, weighted_avg, composite, llm_judge single and panel, target_match, command and score_delta.
  • schema/examples/invalid/ must all fail. Read these when you want to know what a rule actually forbids: dataset-tool-after-on-forbidden.yaml and rubric-composite-with-weight.yaml document their rules more precisely than prose can.

tests/src/Unit/SchemaExamplesTest.php validates every file in both directories on every CI run, so a fixture cannot drift from the schema.

Versioning

Two version numbers live in this contract and they are independent.

Schema-file version is the release tag named in $id. It identifies the release that ships the file and never changes once tagged. On the development branch it names the upcoming tag, currently 1.1.0-alpha6. When cutting a release, set every schema $id to the tag being cut.

The $id is an identity, not a download URL used by ai_eval. Validators read schema files from disk. The upcoming tag URL returns 404 until the tag exists; this is expected. Editor headers and documentation links use raw/1.1.x/ for the latest schema on that branch. These branch URLs can change; use a tagged release when you need an immutable schema.

Once a field ships in a tagged release, we will not rename or remove it, or tighten its pattern on that minor branch. We will not add required fields to a schema already shipped in a tag on that branch. Breaking changes ship on a new minor branch, with a migration paragraph in the changelog and a deprecation period of at least one minor release.

The expected.items and metadata.inputs fields shipped in 1.1.0-alpha1. The current expected.items pattern requires a datasource and a nonempty raw ID, such as entity:node/108; bare IDs do not validate.

Artifact version is the version field inside a rubric or judge YAML, also semver, and it versions that one artifact rather than the schema:

Bump Means
Patch, 1.0.0 to 1.0.1 Prose-only edits to description fields or non-scoring metadata.
Minor, 1.0.0 to 1.1.0 Fields added without changing how existing fields score.
Major, 1.0.0 to 2.0.0 Scoring semantics changed. Scores from the old version are not comparable to scores from the new one; re-baseline downstream consumers.

A field marked legacy stays schema-valid for the full life of its minor branch, the README names the replacement shape, the fixture corpus keeps at least one valid example using the legacy form so adapters can be tested against it, and the removal lands on the next minor branch. The case-level expected_facts.must_not_contain list is the current example: prefer the rubric-side must_not_contain check in new content.

The schema requires retrieval item IDs to have a nonempty datasource and raw part separated by /. Bare IDs such as 108 fail validation. The tool assertion keys args and after also have defined constraints. Move unrelated uses of those names into metadata.

Reference strings

Datasets reference rubrics, composite checks reference other rubrics, and llm_judge checks reference judge prompts. All three use one shape:

rubric/{snake_id}@{semver}
judge/{snake_id}@{semver}

{snake_id} matches ^[a-z][a-z0-9_]*$ and {semver} matches ^[0-9]+\.[0-9]+(\.[0-9]+)?$, with the patch segment optional.

The schema enforces the shape of the string only. It does not check that the artifact exists in any registry, so rubric_ref: rubric/totally_made_up@9.9.9 is schema-valid and resolution is the loader's problem.