Glossary¶
The words AI Eval uses on its screens and in these pages. The first sentence of each entry is the short definition the tutorials show when you point at or tap a term; the rest adds detail and links.
Test¶
A saved setup that says what to ask, which AI feature answers, how answers are scored and what counts as good enough.
The code and some screens call a test a target. Each test has a dataset, a list of graders and a quality gate. See Evaluation targets.
Agent mode¶
A test mode that runs an AI agent end to end, with its tools, the way a visitor would use it.
The agent searches, calls tools and writes its own prompt, so the test measures
the whole feature. Most graders read only the final answer text. The Tool Usage
Grader checks which tools ran; Accuracy and Groundedness can also read the
tool-call record. The optional droost_state grader checks live Drupal state.
See Evaluation targets.
Chat mode¶
A test mode that sends each question straight to one provider and model, with an optional system prompt.
Use it to test a prompt or compare models without an agent around them.
Vision model¶
A model that can read images as well as text.
A question with images needs one, both for the model under test and for any LLM judge that scores it. AI Eval asks the provider whether the model supports image input. If the provider does not list the model under test, the question is skipped with a reason and gets no score. If it does not list a judge's model, that judge gives no score and the other judges still score the question. AI Eval trusts the provider's list, so check that your model really reads images. See Images in questions.
Caller¶
The Drupal user a question runs as, which decides what content the AI feature is allowed to see.
A question can name a caller user ID (0 is an anonymous visitor). Without one, the test runs as the person who started it. An editor can see unpublished or restricted pages that visitors cannot, so set the caller when you want to test what visitors get.
"Caller" has a second sense in calibration. A calibration caller is not a user. It is the feature, such as a field automator, that asks the model a question. The two never refer to the same thing: a test's caller is a Drupal user, a calibration caller is a feature.
Dataset¶
The list of questions a test asks, together with what a good answer to each looks like.
Several tests can share one dataset. See Datasets.
Example¶
One question in a dataset, with its ID, its input and its criteria.
Screens say example or question for the same thing.
Criteria¶
Plain-language text that says what a good answer must contain.
LLM judges read the criteria next to the answer and score how well the answer meets them. Criteria that are vague or wrong give scores that are vague or wrong.
Grader¶
Anything that scores an answer: an LLM judge or a deterministic check.
Each grader gives every answer a score on its own scale, which AI Eval converts to 0 to 5. See Graders.
LLM judge¶
A grader that asks a language model to score an answer against the criteria and give a one-line reason.
A judge is fast and reads meaning, but it can be wrong in a consistent way, so check it against your own labels before you rely on it (see judge validation). AI Eval sets the judge's temperature to 0 where the provider allows, to keep its scores steady.
Deterministic check¶
A grader that runs code instead of a model, so it gives the same score every time.
Examples: the answer is valid JSON, the agent called a given tool, the answer does not contain a phrase.
Rubric¶
A shared, versioned set of checks that several questions can point to, so the same rule is written once.
A question uses a rubric through its Rubric reference, for example
rubric/tester_says_no_info@1.0. The Rubric Checks Grader runs the checks of
that rubric; questions without a rubric reference are left out of that grader.
See Rubrics and checks.
Rubric check¶
One rule inside a rubric, such as "the answer must contain any of these phrases".
Text checks are deterministic: they look for the phrases in the answer, ignoring upper and lower case unless Case sensitive is on. They are cheap and exact, but they miss an answer that says the same thing in other words.
Review rule¶
A set of editorial criteria that the AI Content Review module checks pages against, such as tone and voice, inclusive language, or grammar.
Editors run a rule on a page to get a score and suggestions per criterion.
Content review judge¶
An LLM judge that scores an answer with one criterion of a review rule, the same check editors use on real pages.
Each answer becomes an unsaved page of the content type you choose, reviewed and then thrown away. Scores run from 0 to 100 and are converted to 0 to 5. Every criterion of every rule is a judge of its own.
Site judge¶
The provider and model every LLM judge uses unless the judge has its own model set.
It is set on the AI Eval settings page. A judge's own model is set on its Edit prompt tab.
Question score¶
The average of an answer's grader scores, on a scale of 0 to 5.
Graders that do not apply to a question are left out of the average, not counted as zero. See Scoring and quality gates.
Question pass threshold¶
The question score an answer needs to pass, from 0 to 5.
It comes from the test, then the site setting, then 3.5. A question passes when its score is at least the threshold and no must-pass grader scored below it.
Must-pass grader¶
A grader whose own score must reach the question pass threshold, whatever the other graders say.
Use it for a rule that cannot be traded off, such as "never invents a price".
Pass rate¶
The share of measured questions that passed, from 0 to 1 (shown as a percentage).
A question is not measured, and is left out of the count, when a grader errored on it or no grader could score it.
Average score¶
The mean of the measured questions' scores, from 0 to 5.
Quality gate¶
The pass-or-fail rule for a whole run: a metric (pass rate or average score) and the threshold it must reach.
If fewer than half of the questions could be measured, the gate fails whatever the others scored. See Scoring and quality gates.
Hard and soft gate¶
A hard gate makes drush ai-eval:run exit with an error when the run fails it;
a soft gate only reports the failure.
The admin screens show the verdict the same way for both. The difference matters when a CI pipeline runs your tests.
Run¶
One pass through a test's dataset: every question is asked once and every answer is scored.
A run started in the browser moves one question at a time while its page is open. See Running evaluations.
Result¶
The stored outcome of a finished run: every score, the gate verdict and the run details.
Annotation¶
Your own verdict on one answer in a result (pass, fail or defer), with a note that says why.
Annotations never change a result's scores or its gate. They record what a person thinks, next to what the graders said, and they are kept per user.
Open coding¶
Reading answers one by one and writing a short, free-form note on what went wrong, before deciding on any categories.
The term comes from qualitative research. The notes you collect are later grouped into failure modes.
Failure mode¶
A named, recurring way the AI feature gets things wrong, such as "misses a time that is on the page".
You create failure modes by grouping similar notes. Counting them shows which problems happen most. See Failure modes.
Promote¶
Copy a question you reviewed into a dataset, so tests that use that dataset check it again.
Promoting from an annotation copies the question, its input, criteria and other expectations into the dataset you choose, or into a new one. A new dataset is unused until a test points to it.
Promoting from an agent run in trace review uses the first message the visitor wrote as the question's input, keeps the agent's final answer with it, and keeps the failure mode and note from your run label.
Trace¶
The record of one call your site made to an AI model: what it sent, what came back, which model answered and how many tokens it used.
AI Eval imports traces from the files an OpenTelemetry collector writes, and groups the calls of one question together for review. See Production traces.
Span¶
The OpenTelemetry word for one recorded step, such as one call to a model.
A trace file holds many spans. AI Eval imports the spans that are AI calls and skips the rest, such as page requests.
OpenTelemetry¶
An open standard for recording what software does, step by step, and sending those records to a collector.
In Drupal, the AI Observability submodule of the AI module sends one span per AI call. A collector writes them to a JSON file that AI Eval can import.
Agent run¶
All the calls one agent made to answer one question, reviewed together as one group.
An agent run usually holds a search step (embeddings) and one or more chat calls; a chat call can carry the tool calls and tool results of that step.
Import batch¶
The set of traces added by one import, with its own ID so the import can be undone as a whole.
Failure rate¶
How often a failure mode occurs: its count divided by all the reviewed conversations, passes included.
Reviews by every reviewer count, each conversation once, within the tags you filter on.
Conversations without a failure mode get their own rows, Passed, Failed, no failure mode and Deferred, so the rates add up to 100%.
Draft¶
Unsaved work in the guided workflow, kept for your user for seven days.
A draft never changes a saved test until you save it. See Creating a test step by step.
Judge validation¶
Checking an LLM judge against answers you labeled pass or fail yourself.
AI Eval reports two rates: of the answers you passed, how many the judge passed (TPR), and of the answers you failed, how many it failed (TNR). A judge is trusted when both reach 0.90, every selected answer was scored, and at least 10 distinct pass and 10 distinct fail answers were scored. See Trusting your judges.
Gold label¶
Your pass or fail on one answer, given on a judge's labeling screen and judged on the one thing that judge scores.
A Completeness judge's gold labels say whether each answer is complete, not whether it is good overall. Validation compares the judge with these labels.
A deferred answer is not a gold label: it is set aside and does not count toward coverage.
Domain tags¶
The trace tags that choose which imported answers a judge is labeled and validated on.
A judge for a support assistant should be checked on support answers, not on everything the site ever imported. Domain tags are set on the judge's Edit prompt tab.
TPR and TNR¶
Two agreement rates: of the answers you passed, how many the judge passed (TPR), and of the answers you failed, how many it failed (TNR).
A low TPR means the judge fails good work; a low TNR means it lets bad work through. One overall agreement figure would hide that: a judge that passes everything agrees with every pass you gave.
Confusion matrix¶
A two-by-two table of your verdicts against the judge's: both passed (TP), both failed (TN), the judge passed what you failed (FP), and the judge failed what you passed (FN).
Scoring guidance¶
Extra instructions added to a judge's prompt, on its Edit prompt tab, to correct how it scores a pattern it gets wrong.
Changing the guidance and saving it makes a new prompt version, so earlier validations no longer count for trust.
Self-agreement¶
How often you give the same verdict to the same answer when you label it a second time without seeing your first verdict.
It tells you how reliable your labels are, and so how much a judge's agreement with them can mean.
Cohen's kappa¶
Agreement corrected for chance: 1 is perfect, 0 is what chance gives, below 0 is worse than chance.
It cannot be computed when chance alone would already give full agreement, for example when every label is the same verdict.
Raw agreement can look high when most labels are the same verdict; kappa discounts that.
Judge against judge¶
A comparison of two validation runs on the same gold labels, to tell whether one judge or model really agrees with people more often than the other.
Only the answers the two judges disagree on can tell them apart, so with few disagreements it often reports no measurable difference: not that the judges are equal, but that the labels cannot rank them.
Pass-rate range¶
The range the true pass rate probably lies in, given how few questions the run asked.
AI Eval computes it as a 95% Wilson interval. With five questions that all pass, the range is 57% to 100%: the run cannot tell a feature that is right every time from one that is right three times in five. More questions narrow it.
Judge independence¶
Whether the judge comes from a different model family than the model it scores.
Models tend to score their own family's answers higher. AI Eval warns when the families match, or when it cannot tell, which is common in agent mode.
Decision model¶
A model that answers a yes-or-no question with a probability, such as TypeSafe Jev, through the AI module's Decision operation.
It does not write text. A probability of 0.80 means the model thinks "yes" is likely, but how often such answers are really right is a separate question: that is what a calibration checks.
Decision rubric judge¶
A judge that asks a decision model each rule of a house-style rubric, counts the errors an answer makes (up to two per rule) and scores it from 100 down.
Besides the score it reports a pass probability. Its settings are
ai_eval.decision_rubric; see Calibrating a decision judge.
Pass probability¶
The decision rubric judge's estimate of how likely an answer is to pass: the chance that the errors it finds stay within what the pass score allows.
It is the model's own number, so it can be off. A calibration corrects it; the score is never changed.
Calibration¶
A correction curve, learned from answers people reviewed, that aims to bring a decision model's probabilities closer to how often such answers are really right.
A fit can also make them worse, which is why each fit is checked on held-back answers before it applies.
A calibration belongs to one provider, model and question wording. It applies only while it is active and the wording still matches. See Calibrating a decision judge.
Held-back answers¶
The reviewed answers a fit does not learn from, about half, kept aside to check whether the curve makes the numbers better.
Answers to the same question, or about the same entity, stay on the same side, so the check is not flattered by near-copies of what the curve learned from.
Overall error¶
How far a set of probabilities is from what people decided, from 0 (perfect) to 1; lower is better.
It is the Brier score: the mean of the squared gaps between each probability and the verdict (1 for yes, 0 for no). A fit applies on its own only when it does not raise the overall error on the held-back answers; someone can still activate a worse fit by hand.
Calibration caller¶
A feature that asks a decision model a question, such as a field automator, and can be selected to use a calibration.
Not the same as a test's caller. A test's caller is the Drupal user a question runs as. A calibration caller is the feature that asks the model.
Tokens¶
The units of text a model reads and writes, which providers bill by.
The run summary shows tokens in (read) and out (written) for the whole run.