Failure modes

The vocabulary you use to say why an AI output failed, and the rates the module computes from it. For operators doing error analysis on eval results.

A score tells you that a question failed. It does not tell you what went wrong. A failure mode is a named code you attach to a failing question so that the "what went wrong" becomes countable: hallucinated_fact on nine of forty questions is a finding you can act on, forty scattered free-text notes are not.

Failure modes never affect a score, a question verdict, or the quality gate. They are a reporting layer over human annotations.

The module ships no failure modes

There is no built-in taxonomy. The vocabulary is entirely yours, and the failure-modes screen is empty on a fresh install. A taxonomy earns its keep by coming out of your own traces, so the module leaves it to you.

The intended sequence is open coding then axial coding: annotate real failing questions with free-text notes first, then cluster those notes into a small set of named modes, then tag with the named modes from that point on. The workflow hub walks through exactly those steps.

flowchart LR
    A[Run an eval] --> B[Annotate failing questions with notes]
    B --> C[Cluster the notes into named failure modes]
    C --> D[Tag annotations with the modes]
    D --> E[Read the failure rates]
    E --> F[Promote failing questions into a dataset]

Defining a mode

Failure modes are config entities, so they export and deploy like any other Drupal configuration. Manage them at /admin/config/ai/ai-eval/failure-modes, reachable as the Failure modes tab under Traces. The list shows the mode label, its taxonomy, its status and the operations links, ordered by weight. Use the Add failure mode local action to create one.

Field	Meaning
Label	The human name, shown in the tagging select and in the rates table
Machine name	The code itself, snake_case, for example hallucinated_fact
Description	Free text saying when to use this mode and when not to
Taxonomy	Groups modes for portable export and import. Defaults to default
Weight	Sort order within the taxonomy. Lower sorts first
Status	Enabled or disabled. See the warning below: nothing acts on this yet

Disabling a mode does not retire it

Nothing filters failure modes on status. The tagging select is populated with loadMultiple() and no status condition, so a disabled mode is still offered to annotators, can still be saved onto an annotation, still counts in the rates table, and is still included in a taxonomy export. To retire a mode so it stops being used, delete it rather than disabling it.

Viewing the list needs operate ai eval or administer ai eval. Adding, editing and deleting need administer ai eval.

Keep the vocabulary small. A taxonomy with thirty modes gets used inconsistently by two annotators and produces rates nobody trusts.

Tagging an annotation

Modes attach to annotations, and an annotation attaches to one question of one eval result. Open a result, expand the question in the Question Results table, and the annotation form offers a verdict of pass, fail or defer, a free-text note, and a failure-mode select. The select only appears once at least one failure mode exists.

Saving with an empty select clears any tag already on the annotation. There is one annotation per user per question, so a second person annotating the same question creates their own rather than overwriting yours.

Annotating requires both annotate ai eval results and administer ai eval.

Failure rates

Rates appear as a Failure modes table in two places: on a single eval result, scoped to the annotations on that result, and on the per-target dashboard, scoped to the annotations on every result for that target.

Column	Contents
Failure mode	The mode label, or (untagged)
Count	Annotations carrying that mode in the scope
Rate	That count as a percentage of every annotation in the scope

The denominator is all annotations in scope, not just the tagged ones, and untagged annotations are always shown as their own bucket. That is what makes the rates sum to 100%, and it means a low rate on your top mode may just mean most annotations are untagged rather than that the failure is rare. With no annotations at all the table reads "No annotations yet."

Rates come from a single grouped query over the annotation table, so they stay cheap on a target with thousands of annotations.

Portable taxonomies

A taxonomy exports to the versioned shape that schema/failure_taxonomy.schema.json describes, so a taxonomy is portable across sites. Encoded as JSON it looks like this:

{
  "id": "drupal_agent_failures",
  "version": "1.2.0",
  "modes": [
    {
      "id": "hallucinated_fact",
      "label": "Hallucinated fact",
      "description": "States a fact about the site that is not true.",
      "weight": 0
    }
  ]
}

The schema requires the taxonomy id and every mode id to be snake_case starting with a letter, version to be a two or three part number, and each mode to carry id, label and description. weight is optional and no other keys are allowed inside a mode.

Validation is a separate step

The serializer and the schema are two different things. export() builds the shape by construction and import() reads it defensively; neither one validates. Validation lives in its own service, ai_eval.failure_taxonomy_validator, and only runs if a caller invokes it.

That matters on import. A payload that breaks the schema is not rejected: missing keys fall back to defaults, a label defaults to the mode id, and an unknown extra key is ignored rather than refused. The one guard in the import path is that a mode whose id resolves to an empty string is skipped instead of creating an entity with an invalid machine name.

Validate before you import anything you did not produce yourself. FailureTaxonomyValidatorTest::testExportRoundTripsThroughValidator() shows the intended pairing: export, validate, then import.

Import is additive. Every mode in the payload is created or updated in place, and modes that exist locally but are absent from the payload are never deleted. A shared taxonomy therefore cannot clobber the codes a site added for itself.

No UI or Drush command yet

Both services are wired and usable, but nothing in the admin UI or the Drush command set calls them in this release. Moving a taxonomy between sites today means either custom code against those services or ordinary Drupal config export and import of the ai_eval.failure_mode.* config objects.

What comes next

Annotated questions are the raw material for two other things.

An annotation can be promoted into a dataset from the result screen, so the failure becomes a regression test. The promoted question carries the verdict, the note and the failure mode in its metadata, which keeps the reason the case exists attached to the case. Promotion is idempotent and only the annotation's own author can do it. See datasets for what the promoted question looks like and trace review for the same move starting from a production trace.

The annotations themselves are also what the workflow hub counts when it decides whether your error analysis has enough coverage to be worth reading.

If what you actually want is for the module to catch a class of failure automatically rather than have a human name it, that is a grader, not a failure mode. See graders.