Skip to content

The portable JSON Schemas

ai_eval ships four JSON Schemas that define what a dataset, a rubric, a judge and a failure taxonomy are allowed to look like. They are a public contract for Drupal Eval Commons, not private validation code, and this module is their reference implementation. For developers writing eval content, tooling that reads or writes it, or a runner other than ai_eval.

The files

Under schema/ in the module root, all Draft 2020-12.

File Describes
dataset.schema.json A bundle of eval questions with inputs, expected facts, tool assertions, and an optional rubric reference.
rubric.schema.json A reusable scoring rubric: ordered checks combined by a scoring rule.
judge.schema.json A reusable LLM-judge prompt with score type, optional level names, and calibration metadata.
failure_taxonomy.schema.json A failure-mode taxonomy document.

schema/README.md is the contract. It carries the full check-kind table, the reference-string format, the case-level tool-assertion semantics, the Inspect AI field mapping, and the versioning policy. This page tells you where things are and what the rules mean for your code; when the two differ, the schema README wins.

Why they are public

The schemas describe a portable interchange format. Another tool can read a dataset written for ai_eval, and ai_eval can run a dataset written for another tool, as long as both validate against the same file. That is why changing one is an ecosystem-facing decision rather than a local refactor, and why the URLs are pinned rather than relative.

Schema-valid is not runner-executable

Two separate properties, deliberately kept apart.

  • Schema-valid: the artifact passes validation against the relevant schema.
  • Runner-executable: a given runtime can fetch each declared input, evaluate each declared check and write a score.

ai_eval executes a strict subset. A schema-valid rubric using command or score_delta needs a pending executor before it runs here, and tool_usage as a rubric check is contract vocabulary that ai_eval never executes: the implemented form is the case-level expected_tools key. The full split is in writing a check executor and in the schema README.

This is a feature of the contract, not a gap in it. A portable format has to be able to express things a particular runner has not implemented yet, otherwise nobody can write the artifact that motivates the implementation.

Editor autocompletion

Add a yaml-language-server comment as the first line of any eval YAML. Editors that speak the directive, which includes VS Code with the Red Hat YAML extension and any LSP client using yaml-language-server, then give you completion, hover documentation and inline errors.

# yaml-language-server: $schema=https://git.drupalcode.org/project/ai_eval/-/raw/1.0.x/schema/dataset.schema.json
questions:
  - id: q1
    input: "What is Drupal?"

Use the matching URL for rubric and judge files:

# yaml-language-server: $schema=https://git.drupalcode.org/project/ai_eval/-/raw/1.0.x/schema/rubric.schema.json
id: code_gen_basic
version: "1.0.0"
checks:
  - kind: must_contain_any
    values: ["namespace Drupal\\"]
scoring:
  combine: all_pass

The comment is inert to every YAML parser, so it costs nothing at run time.

Validating outside Drupal

Any Draft 2020-12 validator works. With check-jsonschema:

pip install check-jsonschema
check-jsonschema \
  --schemafile https://git.drupalcode.org/project/ai_eval/-/raw/1.0.x/schema/dataset.schema.json \
  my-dataset.yaml

With Node's ajv-cli:

npx ajv-cli@latest validate \
  --spec=draft2020 \
  -s dataset.schema.json \
  -d my-dataset.yaml

Unknown fields on a check are rejected, via unevaluatedProperties: false, so a typo surfaces at validation rather than as a silently skipped check at run time.

Validating inside Drupal

Three services wrap the schemas, each built from the shipped file:

Service Validates
ai_eval.dataset_validator A dataset document
ai_eval.rubric_validator A rubric document, including check kinds contributed by tagged executors
ai_eval.failure_taxonomy_validator A failure taxonomy document

Each exposes validate() returning a bool and errors() returning the messages from the last call. The shipped config and entity dataset sources call ai_eval.dataset_validator before returning any questions; do the same in a dataset source of your own.

ai_eval.rubric_validator is the one that changes shape at runtime: it splices in the definitions of every tagged check executor that implements CheckSchemaProviderInterface, so a rubric using a kind from your module validates on a site where your module is installed and is rejected as a typo where it is not.

The example corpus

schema/examples/ holds fixtures that the in-tree test suite asserts against, which makes them the most reliable reference for what a valid document looks like.

  • schema/examples/valid/ must all pass. Start here when authoring: there is a minimal dataset, a multi-turn one, both split forms, tool assertions, an MCQ case, a binary judge, and rubrics covering all_pass, weighted_avg, composite, llm_judge single and panel, target_match, command and score_delta.
  • schema/examples/invalid/ must all fail. Read these when you want to know what a rule actually forbids: dataset-tool-after-on-forbidden.yaml and rubric-composite-with-weight.yaml document their rules more precisely than prose can.

tests/src/Unit/SchemaExamplesTest.php validates every file in both directories on every CI run, so a fixture cannot drift from the schema.

Versioning

Two version numbers live in this contract and they are independent.

Schema-file version is the branch pinned in the $id and in the yaml-language-server URL, currently 1.0.x. That branch is stable: no field is renamed or removed, no pattern is tightened, and no required field is added on a branch that has shipped. Additive changes, meaning new optional fields, new check kinds and new enum values on permissive enums, may land on a stable branch. Breaking changes ship on a new minor branch such as 1.1.x, with a migration paragraph in the changelog and a deprecation period of at least one minor release for the old shape.

The git.drupalcode.org/project/ai_eval/-/raw/1.0.x/schema/*.json URLs are the canonical home for the v1 schemas and will not move while 1.0.x is supported. Pin to the branch, not to a commit.

Artifact version is the version field inside a rubric or judge YAML, also semver, and it versions that one artifact rather than the schema:

Bump Means
Patch, 1.0.0 to 1.0.1 Prose-only edits to description fields or non-scoring metadata.
Minor, 1.0.0 to 1.1.0 Fields added without changing how existing fields score.
Major, 1.0.0 to 2.0.0 Scoring semantics changed. Scores from the old version are not comparable to scores from the new one; re-baseline downstream consumers.

A field marked legacy stays schema-valid for the full life of its minor branch, the README names the replacement shape, the fixture corpus keeps at least one valid example using the legacy form so adapters can be tested against it, and the removal lands on the next minor branch. The case-level expected_facts.must_not_contain list is the current example: prefer the rubric-side must_not_contain check in new content.

One narrowing has landed on 1.0.x and the README names it rather than hiding it. expected_tools entries permit unknown keys, so before args and after were defined those two names validated with any content at all. Defining them constrains what they may hold. A document that used either name for its own purposes before they were defined needs the value moved into metadata.

Reference strings

Datasets reference rubrics, composite checks reference other rubrics, and llm_judge checks reference judge prompts. All three use one shape:

rubric/{snake_id}@{semver}
judge/{snake_id}@{semver}

{snake_id} matches ^[a-z][a-z0-9_]*$ and {semver} matches ^[0-9]+\.[0-9]+(\.[0-9]+)?$, with the patch segment optional.

The schema enforces the shape of the string only. It does not check that the artifact exists in any registry, so rubric_ref: rubric/totally_made_up@9.9.9 is schema-valid and resolution is the loader's problem.