Skip to content

Calibrating a decision judge

The Decision rubric judge asks a decision model (the AI module's Decision operation, for example TypeSafe Jev) each rule of a house-style rubric and scores the answer by the rubric's arithmetic. Besides the score it reports a pass probability: the chance that the errors it found stay within what the rubric's pass score allows. With 5 points per error and a pass score of 90, a page passes with at most 2 errors.

By default the judge sees only the answer, sent under the rubric's state_key. A rule such as "do not add facts that answer a different question" needs the question too: set input_key in ai_eval.decision_rubric (for example question) and the judge also gets the test question under that name. Word the rubric's question to say so ("This is a visitor's question and the assistant's reply...").

A model's probabilities are often off. If the answers it calls 80% likely to pass are right only half the time, a calibration learned from answers people have reviewed corrects the number. The judge then reports both: "Pass probability 0.982, calibrated 0.810". Calibration corrects the pass probability, never the score.

Calibrations live under Manage > Calibrations (/admin/config/ai/ai-eval/calibrations) and need the Validate AI Eval judges permission.

Fit a calibration has one tab per kind of fit: Judge (the decision rubric judge, from reviewed test results), Answer file (any question and model, from a file of labeled answers) and, with the submodule, Field automators (from recorded decisions). A line under the tabs says the same.

Fitting from results reviewed on this site

Every run with the decision rubric judge keeps each answer's raw pass probability, and the provider, model and question it came from. When you mark those answers pass or fail on the results page, each reviewed answer pairs the model's number with what it should have been.

Open Fit a calibration. The Judge tab lists the reviewed answers grouped by provider, model and question, with how many passed, failed or were deferred, whether the group is for the judge's current rubric, and whether it can be fitted:

  • Only the first review of an answer counts; a second reviewer's verdict on the same answer is ignored, as in judge validation. Deferred answers are counted but not fitted on.
  • A group needs at least 30 pass or fail answers.
  • Answers to the same dataset question stay together: they are all fitted on or all held back to check the fit. A group whose answers are for too few questions cannot hold any back yet, and one whose fitted answers all passed or all failed cannot be fitted either. The table says which.

Choose a group that can be fitted, optionally name it, and Fit and save. Counts are checked again when you submit, in case reviews changed.

Fitting from answers reviewed elsewhere

The Answer file tab fits answers you bring with you. Fill in:

  • Model: the provider and model that gave the answers, chosen from the models your site offers for the Decision operation. It starts on the decision rubric judge's own model. On a site whose AI module lists no decision models, type it as provider__model, for example typesafeai__jev-latest. A calibration belongs to one provider and model: the same model name under another provider is a different model.
  • Question: choose Decision rubric judge (current) to calibrate the judge as it is configured now. The option shows the rubric's question, its number of rules and how many errors still pass. It is missing when the judge could not grade with the rubric either: no rules, points per error below 1, or two rules with the same id. Choose Other wording (outside data) only for answers from another module that asks its own question; then paste that wording and its criteria exactly as the model receives them.
  • Answers: one reviewed answer per line, id,probability,verdict, where the verdict is 1 for yes, 0 for no, or the share of reviewers who said yes. For the decision rubric judge the probability is its raw pass probability, before any calibration, which a run stores with each grade as grader_scores.decision_rubric.decision.p_pass; not one rule's probability and not the calibrated number. At least 30 answers with both verdicts among them. Paste them, or upload the same lines as a CSV file, which is used instead of the pasted text.

On both tabs, half the answers fit the curve and the other half check it. The detail page shows how far off the model was on the checking half before and after the correction.

If someone edits the rubric while you are on the form, saving is refused with "The rubric changed while you were on this page." Reload and check the question before fitting again.

A fit that makes things worse

A curve fitted on a few answers can make the numbers worse on the answers held back. AI Eval checks the overall error (the Brier score, 0 to 1, lower is better) on those answers before and after the curve:

  • Better: neither the overall error nor the off-by figure rose. The calibration is saved and applies, replacing any earlier one for the same model and question.
  • Mixed: the overall error did not rise but the off-by figure did. It still applies, and the list says the two measures disagree.
  • Worse: the overall error rose, by any amount. The calibration is saved but not active. The earlier calibration, if there is one, keeps applying; if there is none, the answers stay uncorrected.

The list shows each calibration's name, status, what it corrects (the selected callers, and the decision rubric judge when it asks that question with that model) and result. A calibration you did not name shows the question on the Judge and Field automators tabs, and the model id on the Answer file tab. To apply a worse fit anyway, open it and choose Activate it anyway, or use Activate in the list. Activating deletes the other calibrations for that model and question. Only one inactive fit is kept per model and question: a newer worse fit replaces it.

If two calibrations for the same model and question are ever active at once (for example after importing configuration), neither is applied and the detail page says so. Delete all but one.

When a calibration stops applying

A calibration applies only to the exact question it was fitted on. For the decision rubric judge that is the question, the note, every rule's text in order, the answer levels, the names the page and the question are sent under, and the error allowance that the pass score and points per error give. Change any of them and the judge stops applying the old curve; its grades read "uncalibrated" until you fit a new one. Settings that leave the error allowance the same (for example a pass score of 91 with 4 points per error) keep the curve.

Each calibration's detail page says whether the judge applies it now:

  • "asks this question now with this model": the judge corrects its pass probability with this curve.
  • "asks this question now, but not with this model": the judge answers with a different model, so the curve is not used.
  • "does not ask this question now": the rubric or its pass setting has changed since the fit, or the curve was fitted on other wording. Another module may still read it.

Calibrations fitted before AI Eval included the levels and pass setting in this check (#3594764), or before it recorded the provider (#3594777), no longer apply. Their detail page says so. Fit them again from the current rubric.

Use a calibration for other callers

Enable AI Eval Decision Calibration. Open a calibration from Manage > Calibrations, then open its Callers tab. Select the field automators that must use it. Select Save callers. The field's AI Automator Settings list the calibrations that select its automator, for people who may view calibrations.

A curve fitted on the Answer file tab can match only Decision: Boolean automators. That tab cannot enter array instructions or true/false criteria. A curve fitted from recorded decisions (see below) uses the wording that the caller sent, so it can match any caller.

A caller is a feature that asks an AI model a question. The active calibration applies only when the question wording, provider, model that answered, and a selected caller all match. Criteria are part of the question wording. A model alias in the request does not override the model that answered. If the model that answered is empty, no calibration applies. With no selected callers, no caller uses the curve.

The correction applies to true/false probabilities. Score and choice answers stay the same. The decision rubric judge continues to correct its own pass probability. The raw probability stays in the answer's raw data under ai_eval_calibration, with the calibration ID and the matching caller tag.

An eval run of a scoped automator records the raw probability. The field that the automator fills gets the calibrated probability.

A new active fit for the same provider, model and question keeps the callers of the curve it replaces, unless the new curve already has its own selection.

Saved tags for callers not listed on the form are checked by default. This includes deleted automators and tags saved when AI Automators is not enabled. Uncheck a saved tag to remove that caller.

Record and review callers' decisions

A user with administer ai eval permission chooses what the site records. On Calibrations, open Record decisions. Select the field automators whose true/false answers you want to review. Choose a sample rate greater than 0 and at most 1. The default, 1, records every answer. Saved callers stay listed when an automator is removed.

Stop after limits how many answers of a caller are stored. Leave it empty for no limit. When a caller reaches it, recording for that caller stops and Record decisions shows Recording stopped. If the stored count is already above the limit, it shows Recording stopped: 42 recorded, limit 3. Calibrations still apply to its answers. Requests running at the same time can exceed the limit by at most the answers in flight at that moment. Deleting entities removes their recorded answers; recording resumes when the caller is below its limit. When a request has more than one recorded tag, each answer is stored once, under the first open tag whose sample draw succeeds.

A record keeps the entity reference, field name and question wording. It also keeps the provider, model that answered, raw probability and calibration applied, if any. With Keep the text sent to the model, for review, it also keeps the state as text or pretty JSON, up to 64 KB, only when the request names a known content entity type and an entity ID (for a new entity, once it is saved). This option is off by default. A separate flag shows when text was shortened; the text itself has no added marker. Files are never kept. Only their names, MIME types and sizes in bytes are stored with the text, for example "1 file(s) sent: photo.png (image/png, 1024 bytes), not kept". A missing name is shown as "unnamed file".

Stored answers, text and file summaries are kept until their entity is deleted. Unchecking the option stops storing new text and file summaries and keeps what is stored. Kept text and file summaries are visible only to reviewers who can view the entity; otherwise the page says "Text hidden: you cannot view this item."

An answer is not recorded when:

  • the response does not report which model answered;
  • the question wording contains text from the state.

Keep content out of the question wording and send it in the state. If a prompt puts content into a question (for example with a {{ context }} token), its answers are not recorded. A calibration also cannot apply to them, because the wording is different for every entity.

An entity not saved yet has no ID when the answer is recorded. When the AI module reports the entity's UUID, the answer is linked to the entity, and its text is kept if you chose that, as soon as the entity is saved. It then works like any other answer, and is deleted with the entity. Without the UUID, or when the entity is never saved, the answer stays without a link or text and the review page says the entity was not saved yet.

Open Review decisions and select a group: one caller, provider, model and question. The page shows the question once and up to 50 answers not reviewed yet, oldest first. Follow the entity link to read the content, if you have access to it. When text was kept, open Text the model judged under the entity. Choose True, False or Skip, then select Save reviews. The page does not show the model's probability, so it does not influence your review. Skipped answers stay in the list. If another reviewer labeled an answer first, your label is not saved and the page says so.

The group page always shows how many answers the caller recorded and how many of this question and model are reviewed; fitting needs 30. When recording stopped with fewer than 30 reviewed, a user who can change Record decisions can select Record 50 more. This raises the limit by 50 from the higher of its previous limit and the stored count. The Reviewed tab lists reviewed answers with your label, the model's probability and, when a calibration corrected it, the corrected probability and calibration name.

Fit from recorded decisions

On Fit a calibration, open the Field automators tab. A group needs at least 30 reviewed answers, both true and false on the fitted side, and answers on the held-out side. Answers about the same entity stay on the same side of the split. If a group is not ready, review answers about more entities.

Fit and save checks readiness again. The new curve applies only to the callers in its scope that ask the question with the same wording. Its scope is the group's caller. Callers that ask the same question of the same model share one curve: if a curve for that question is active, the new curve replaces it and also applies to its callers. The Also applies to column lists them before you fit. A fit that makes the held-out Brier score worse is saved inactive, as on the other fitting tabs.

Check newer answers

The Callers tab of an active calibration has a section Since this calibration was fitted. It uses the answers recorded after the fit and then reviewed, for the same provider, model, question and selected callers. It counts each entity once, using the entity's most recently recorded reviewed answer; an answer that names no entity counts on its own. Answers recorded before the caller was selected still count, because the check applies the curve to the raw probabilities. The calibration stores when it was fitted, not when it was activated, so for a curve activated later this includes answers from before activation. The section compares the Brier score of the raw probabilities with the Brier score after the curve. With fewer than 30 entities it gives no verdict. When the curve makes the Brier score higher, the Callers tab shows a warning and suggests fitting again. The calibration stays active until you replace or delete it.