Known gaps

Tracks known limitations that are not yet fixed. Unlike decisions/, these are not accepted architectural trade-offs. They're TODO items.

Retrieval has no langcode filter

Where: ai_search module (contrib), RagTool::execute() (web/modules/contrib/ai_search/src/Plugin/AiFunctionCall/RagTool.php).

Problem: the RAG query is built from search_string, amount, and min_score only: nothing constrains results to the visitor's current interface language. On any multilingual site where translated nodes share the same underlying content, a query whose top-scoring match happens to be a translation in a different language will have that translation loaded (AnswerService::loadTranslated()) and fed to the LLM as a source, regardless of what language the visitor asked in.

Current mitigation (config only, per agent): - Raise min_score (Tools → RAG/Vector Search → Property setup → min_score, on the agent's own edit form) to cut down on weak keyword-only matches; see Getting started for how to pick a value. - Add an explicit language/relevance instruction to "Extra generation guidance" (AiAnswersAgentForm) telling the model to answer in the question's language and to decline weakly-related sources.

This reduces how often a wrong-language or off-topic chunk gets cited, but doesn't prevent one from being retrieved in the first place. A sufficiently ambiguous query can still score a wrong-language chunk above the threshold and get handed to the model. The config change only asks the model to compensate after the fact.

Real fix (not implemented): add a langcode constraint to the Search API query in RagTool::execute(), e.g. restrict to (or boost) the current interface language, so mismatched-language chunks aren't retrieved as candidate sources at all.

Status: open, not tested against an adversarial query. The one manual check performed happened to score the correct-language source highest, before this gap was identified.

Feedback scoring never reaches Langfuse

Where: drupal/langfuse (contrib), LangFuseClient (web/modules/contrib/langfuse/src/LangFuseClient.php); consumed from FeedbackLogger::scoreTrace().

Problem: FeedbackLogger::scoreTrace() calls $this->langfuseClient->createScore($traceId, 'user_feedback', $value, $comment). That method does not exist anywhere: not on LangFuseClient, not on LangFuseClientInterface, not on the underlying dropsolid/langfuse-php-sdk. The call throws Error: Call to undefined method Drupal\langfuse\LangFuseClient::createScore(), caught by the existing catch (\Throwable $e) and logged as a warning. The thumbs up/down buttons appear to work (no error reaches the visitor), but no score is ever submitted to Langfuse.

Even a naive fix ($client->getTrace($traceId)->score(...)) wouldn't hold up: feedback normally arrives in a separate, later request than the one that generated the answer, and LangfuseSyncSubscriber::onKernelTerminate() wipes the trace's local State cache at the end of every request. getTrace() returns NULL by the time feedback is submitted.

Current mitigation: none. Feedback still writes to ai_log (when ai_logging is installed). Only the Langfuse score is missing.

Real fix (not implemented, filed upstream): add createScore(string $traceId, string $name, float $value, ?string $comment = null): void to LangFuseClient/LangFuseClientInterface, built on Client::sendEvent() with a hand-built score-create event keyed only by trace ID; this was verified to work even against a trace already wiped from local state. Filed as langfuse #3548465. No ai_answers change is needed once it lands. FeedbackLogger already calls the right signature.

Status: open, upstream, not yet fixed or released.

In-module rerank shows a different order than the LLM saw

Where: AnswerService::maybeRerank() (src/Service/AnswerService.php).

Problem: the in-module rerank pool used to be a 4x over-fetch against the tool's result count, giving the reranker room to actually reorder. It now equals the tool's amount exactly, so maybeRerank() only reorders the final set, and it does so after the LLM has already generated its answer from the raw, unreranked tool order. That means two different orderings of the same chunks exist within a single context window: the one the LLM read, and the one presented in the reference list.

Current mitigation: none; the inconsistency between what the model read and what's shown is accepted as-is for now.

Real fix (not implemented): adopt the ai_reranker Search API index processor once drupal/ai ^1.5 is available. That processor reranks the query at the point the tool itself issues it, producing one consistent ordering everywhere instead of two. Tracked in ai_answers issue 3610866, which also adds a "maximum references" setting so amount can go back to being a wide over-fetch once rerank happens upstream of the tool. The ai_reranker processor itself needs an upstream fix too: its extractItemText() should prefer extraData('content') over its current source.

Status: open, blocked on drupal/ai ^1.5 and an ai_reranker upstream fix.