Failure modes

The vocabulary you use to say why an AI output failed, and the rates the module computes from it. For operators doing error analysis on eval results.

A score tells you that a question failed. It does not tell you what went wrong. A failure mode is a named code you attach to a failing question so that the "what went wrong" becomes countable: hallucinated_fact on nine of forty questions is a finding you can act on, forty scattered free-text notes are not.

Failure modes never affect a score, a question verdict, or the quality gate. They are a reporting layer over human annotations.

The module ships no failure modes

There is no built-in taxonomy. The vocabulary is entirely yours, and the failure-modes screen is empty on a fresh install. A taxonomy earns its keep by coming out of your own traces, so the module leaves it to you.

The intended sequence is open coding then axial coding: annotate real failing questions with free-text notes first, then cluster those notes into a small set of named modes, then tag with the named modes from that point on. The workflow hub walks through exactly those steps.

flowchart LR
    A[Run an eval] --> B[Annotate failing questions with notes]
    B --> C[Cluster the notes into named failure modes]
    C --> D[Tag annotations with the modes]
    D --> E[Read the failure rates]
    E --> F[Promote failing questions into a dataset]

Defining a mode

Failure modes are config entities, so they export and deploy like any other Drupal configuration. Manage them at /admin/config/ai/ai-eval/failure-modes, reachable from Manage > Failure modes. The list shows the mode label, its taxonomy, its status and the operations links, ordered by weight. Use the Add failure mode local action to create one.

Field	Meaning
Label	The human name, shown in the tagging select and in the rates table
Machine name	The code itself, snake_case, for example hallucinated_fact
Description	Free text saying when to use this mode and when not to
Taxonomy	Groups modes for portable export and import. Defaults to default; the taxonomies already in use are suggested
Weight	Sort order within the taxonomy. Lower sorts first
Status	Enabled or disabled. A disabled mode is retired: see below

Disabling a mode retires it

A disabled mode is not offered in the annotation form or in the trace review verdict bar, and a request that picks one is refused with a "not saved" message. A label that already uses the mode keeps it and shows it as "(disabled)"; saving that label again keeps the mode. Failure rates still count it and mark the row "(disabled)". Typing a new mode name that matches a disabled mode is refused too. A taxonomy export leaves disabled modes out.

Prefer disabling to deleting. A deleted mode leaves labels pointing at a mode that no longer exists, and the rates then show its machine name.

Viewing the list needs operate ai eval or administer ai eval. Adding, editing and deleting need administer ai eval.

Keep the vocabulary small. A taxonomy with thirty modes gets used inconsistently by two annotators and produces rates nobody trusts.

Tagging an annotation

Modes attach to annotations, and an annotation attaches to one question of one eval result. Open a result, expand the question in the Question Results table, and the annotation form offers a verdict of pass, fail or defer, a free-text note, and a failure-mode select. The select only appears once at least one failure mode exists.

Saving with an empty select clears any tag already on the annotation. There is one annotation per user per question, so a second person annotating the same question creates their own rather than overwriting yours.

Annotating requires annotate ai eval results or administer ai eval, and so does seeing annotations. A user with only operate ai eval gets no Annotation column on a result, and the workflow's Annotate step names the permission it needs.

Failure rates

Rates appear as a Failure modes table in two places: on a single eval result, scoped to the annotations on that result, and on the per-target dashboard, scoped to the annotations on every result for that target.

Column	Contents
Failure mode	The mode label, “Passed”, “Failed, no failure mode” or “Deferred”, or the label with (disabled)
Count	Questions whose counted annotation carries that mode in the scope
Rate	That count as a percentage of every annotated question in the scope

Each question of a run counts once; the same question in another run counts again. When several reviewers annotate one question, the earliest annotation counts, as in judge validation.

The denominator is every annotated question in scope, not just the tagged ones. Questions without a mode appear as Passed, Failed, no failure mode, or Deferred according to their verdict. These rows keep the rates summing to 100%. A low rate on the top mode can mean many questions passed or were deferred; it does not by itself show that failures are rare. With no annotations at all the table reads "No annotations yet."

Rates come from a single grouped query over the annotation table, so they stay cheap on a target with thousands of annotations.

Portable taxonomies

A taxonomy exports to the versioned shape that schema/failure_taxonomy.schema.json describes, so a taxonomy is portable across sites. Encoded as JSON it looks like this:

{
  "id": "drupal_agent_failures",
  "version": "1.2.0",
  "modes": [
    {
      "id": "hallucinated_fact",
      "label": "Hallucinated fact",
      "description": "States a fact about the site that is not true.",
      "weight": 0
    }
  ]
}

The schema requires the taxonomy id and every mode id to be snake_case starting with a letter, version to be a two or three part number, and each mode to carry id, label and description. weight is optional and no other keys are allowed inside a mode.

Validation is a separate step

The serializer and the schema are two different things. export() builds the shape by construction and import() reads it defensively; neither one validates. Validation lives in its own service, ai_eval.failure_taxonomy_validator, and only runs if a caller invokes it.

That matters on import. A payload that breaks the schema is not rejected: missing keys fall back to defaults, a label defaults to the mode id, and an unknown extra key is ignored rather than refused. The one guard in the import path is that a mode whose id resolves to an empty string is skipped instead of creating an entity with an invalid machine name.

Validate before you import anything you did not produce yourself. FailureTaxonomyValidatorTest::testExportRoundTripsThroughValidator() shows the intended pairing: export, validate, then import.

Import is additive. Every mode in the payload is created or updated in place, and modes that exist locally but are absent from the payload are never deleted. A shared taxonomy therefore cannot clobber the codes a site added for itself.

Export leaves out disabled modes, and the schema has no status key: whether a mode is enabled is a site setting. Import never changes the status of a mode that exists, and a new mode is created enabled.

No UI or Drush command yet

Both services are wired and usable, but nothing in the admin UI or the Drush command set calls them in this release. Moving a taxonomy between sites today means either custom code against those services or ordinary Drupal config export and import of the ai_eval.failure_mode.* config objects.

What comes next

Annotated questions are the raw material for two other things.

An annotation can be promoted into a dataset from the result screen, so the failure becomes a regression test. The promoted question carries the verdict, the note and the failure mode in its metadata, which keeps the reason the case exists attached to the case. Promotion is idempotent and only the annotation's own author can do it. See datasets for what the promoted question looks like and trace review for the same move starting from a production trace.

The annotations themselves are also what the workflow hub counts when it decides whether your error analysis has enough coverage to be worth reading.

If what you actually want is for the module to catch a class of failure automatically rather than have a human name it, that is a grader, not a failure mode. See graders.